AWS turns from open source villain to hero (at least, for some Elasticsearch users)...

In early 2021 Elastic -- creator of the widely used open source search engine ElasticSearch and its data visualisation companion Kibana -- decided to tighten up the licensing on them from the permissive Apache 2.0 to two more restrictive licenses (the Elastic License and Server Side Public License (SSPL).

(Elasticsearch can be used to search textual, numerical, geospatial, structured, and unstructured data. It has a large financial services user base: Elastic names Rabobank as one client -- the Dutch multinational has built an architecture that lets it pull transactional information from mainframes into both a data store and Elasticsearch, allowing customers to search multiple accounts going back eight years. Rabobank now sees upwards of 200 events per second across its 80TB of data: "Each query can span thousands of accounts, with corporate customers having over 5,000 accounts that they can now query at once", Elastic notes, without the bank having to add any extra operations to their mainframes: handy, and cost-effective.)

Elastic's licensing decision -- as with the open source licence changes made by several other companies in recent years -- was aimed at "restricting cloud service providers from offering our products as a service without sharing their modifications and the source code of their service management layers," Elastic said at the time. (AWS had launched an Elasticsearch service in 2015; as with its subsequent 2018 launch of managed Apache Kafka and 2019 launch of a managed Apache Cassandra service the move drew a decidely mixed reaction from the open source community, some of whom saw it as "strip mining" open source.)

Enter OpenSearch

Elastic's move in early 2021 drew a reaction from AWS within 12 weeks, with the cloud hyperscaler pledging on April 12 to turn from poacher to gamekeeper and fork Elasticsearch and Kibana, then license them under the permissive Apache 2.0. That move would "ensure users continue to have a secure, high-quality, fully open source search and analytics suite with a rich roadmap of new and innovative functionality", AWS said, pointing to support from Capital One, Red Hat, and SAP. (AWS added it would build the project on Elasticsearch 7.10.2 and Kibana 7.10.2, add new features, and rebrand its managed Amazon Elasticsearch service to Amazon OpenSearch.)

This week the project spread its wings after a flurry of last-minute bug fixes, with OpenSearch becoming production-ready with the launch of OpenSearch 1.0. AWS said it had added "multiple new enhancements... to the project: data streams, trace analytics span filtering, report scheduling, and more. As the OpenSearch GitHub repo's welcome page puts it: "We're looking to sustain (and evolve!) a search and analytics suite for the multitude of businesses who are dependent on the rights granted by the original, Apache v2.0 License."

See also: DataStax on evolving enterprise data strategies, a new CEO, and its renewed commitment to open source

"Since introducing OpenSearch, the team has worked in the open to bring the project to this point", OpenSearch's developers said in a July 12 blog. "We thoroughly tested the software, added test automation, build infrastructure, and made other updates to ensure that the code is suitable for production use. We also invested in backwards compatibility and upgrade functionality to make the transition to OpenSearch as simple as possible for users of open source Elasticsearch. With this work, the project will be able to move quickly with a regular release cadence that anyone can follow on the project’s public roadmap.

They added: "As we have mentioned before, developers embrace open source software for many reasons, perhaps the most important being the freedom to use that software where and how they want. We encourage anyone to use, modify, extend, embed, monetize, resell, and offer OpenSearch as part of their products and services. Broad adoption benefits the entire community."

Elastic, meanwhile, is deepening a partnership with Microsoft. The two recently announced Elastic on Azure, a service that bakes Elastic into the Azure portal, letting Microsoft customers consolidate billing through the Azure marketplace, provision an Elasticsearch deployment in an Azure region, ingest logs, view and manage Elasticsearch deployments within the Azure portal... A June 2021 presentation by the company for investors  meanwhile shows it continues to grow fast, with revenues of $608.5 million for the year (up 42% year-on-year), 93% of which is subscription revenue.

We'd be very keen to hear from early OpenSearch users, as well as Elasticsearch contributors about their experiences and this evolution. Get in touch.

Follow The Stack on LinkedIn