MongoDB takes “Queryable Encryption” wizardry GA -- say it has improved latency, useability

MongoDB has taken its “queryable encryption” (QE) capabilities to general availability (GA) – allowing users to query fully encrypted data.

MongoDB Queryable Encryption is a set of client-side libraries and server-side code that let applications encrypt sensitive fields in documents, so they remain encrypted while the server processes them. (Clever.)

The company has kept the underlying encryption technology open source with Go, Node/Javascript, Python and other libraries published under Apache 2.0 licences “so developers can freely examine our cryptographic techniques and code” per Kenn White, Security Principal, MongoDB.

White noted to The Stack today that “this transparency also helps our customers meet security and compliance requirements.”

MongoDB Queryable Encryption: An example?

The New York-based company gave the example of an “authorized application end-user at a financial services company [who] may need to query records using a customer’s savings account number.

"When configured with MongoDB Queryable Encryption, the content of the query and the data in the savings account field will remain encrypted when traveling over the network, while it is stored in the database, and while the query processes the data to retrieve relevant information.

“After data is retrieved, it becomes visible only to an authorized application end user with a customer-controlled decryption key to help prevent inadvertent data exposure or exfiltration by malicious actors.”

MongoDB said its QE design goals included protection against insider threats as well as the risk of stolen credentials, misconfigurations, phishing or system errors: “QE needs to protect the database against an adversary that accesses the database through legitimate means and, in particular, using commands that are authorized for its role” a technical whitepaper published today (August 15, 2023) emphasises.

See also: The world’s first fully specified, end-to-end encryption standard just landed. That's big

GA comes 14 months after a preview release explored by a range of blue chip customers including Renault, and the company says it has been focused on improving user experience and latency over the past year.

Queryable Encryption can be used with AWS, Azure, and GCP key management services as well as other services compliant with the key management interoperability protocol (KMIP).

Another goal has been making it easier for developers to create queryable encrypted apps without needing cryptographic expertise.

MongoDB Queryable Encryption: Get your math on...

The queryable encryption capabilities build on the pioneering work of Brown University cryptographer Seny Kamara and Tarik Moataz.

The two co-founded Aroki Systems, a queryable encryption database company bought by MongoDB in 2021 – and backed by a strong MongoDB team have refined the capabilities to improve usability.

Kenn White, Security Principal, MongoDB, told The Stack that the company had been “focused on performance and optimising for typical operational workload patterns, as we talked with customers during the beta/preview phase, and it paid off – we managed to make 5-100X improvements in latency and throughput over the initial release.”

He added: “Our early adopters were curious about the levels at which data can be encrypted. Understanding that the needs for encryption may vary, we quickly made sure that flexibility was in-built to Queryable Encryption. Users can selectively encrypt individual fields within a document, a subdocument, or the entire document. Each field is secured with its own key and is decrypted seamlessly on the client.”

QE clients were designed to be stateless so that they can be used by short-lived and lightweight clients (e.g., running in containers).

White noted to The Stack in an emailed comment that “one thing that may be surprising to hear is that only a portion of the team's work was dedicated to cryptography per se - a major amount of our engineering went into the developer experience: reducing installation and setup friction, carefully evaluating default settings and eliminating unnecessary choices in lieu of opinionated safe & performant defaults.

“This technology is integrated into virtually every major programming language, so making simple tutorials & best practices in all our developer docs was top of mind, be they C#.NET, Python, Java, Go or any of the dozen+ frameworks we support,” he added.

“Our in-house Cryptography Research Group spent hundreds of hours working side by side with the core Engineering and Product Security teams during the design and development phases…

"We've briefed and assisted front-line developers working on introducing Queryable Encryption into applications spanning capital markets, investment and retail banking, health insurers, retail card payments, and consumer electronics. We learned so much about how real customers struggle to shore up their defensive posture and countless interesting use cases, most of which we're obligated to not share. It's early days, but by every indication there is strong interest in nearly every vertical, and we're thrilled to see this technology reach GA,” he concluded.

nb. The Stack typically aims to break down some of the technology world’s innovations into digestible language as well as explaining how it might be deployed but with structured encryption, as with homomorphic encryption, our reach exceeds our grasp; whilst the deployability of the technology may have had some of the complexity stripped out of it, its fundamental machinery is understandably formidably complex. We can, however, point more educated readers to this interesting technical whitepaper and this somewhat more esoteric paper by Seny Kamara and Tarik Moataz published this month. Happy reading.