Introducing Apache Airavata: "Guardian of the Gateway to the Clouds"
Apache Airavata is a middleware framework designed to manage and monitor large scale scientific applications and workflows on distributed computing resources, including local clusters, supercomputers, and computational grids. It bundles a server package with an API, client SDKs, and a general purpose reference user interface to the Airavata API, built on Django.
Ultimately, it sets out to provide an interface for scientific applications and data for end users who are unfamiliar with/or need more than a CLI interface can provide.
The project (which has built to help tackle the tedious workloads many scientists face after conducting their experiments -- these typically involving collecting the data then running it through multiple applications. Airavata was born out of the idea that this flow sequence could be automated; with legacy command line scientific applications wrapped with web services, and jobs run on a heterogeneous set of computational resources.
The Apache Airavata Journey
Airavata’s genesis began way back in 2003, when the gargantuan weather analysis project, Linked Environments for Atmospheric Discovery (LEAD), then led by Dr. Dennis Gannon, commenced at the Extreme Computing Lab at Indiana University.
(‘Airavata’ in the Indian classical language Sanskrit, refers to a mythical three-headed elephant regarded as the guardian of the gateway to the clouds. The name was chosen because the original Airavata Project’s focus was to mine huge amounts of weather data to accurately predict severe weather scenarios.)
Born at China’s WeBank, now incubating in the ASF: Introducing Apache EventMesh
A year and half after entering the Apache Incubator, the Apache Software Foundation (ASF) announced in October 2012, that Apache Airavata had graduated from the Apache Incubator to become a Top-Level Project (TLP).
“For us, the ASF represents an important open community model as well as open source model for diversely-developed, sustainable scientific software. We hope to convince more in our community to follow suit,” Marlon Pierce said -- speaking as theScience Gateway Group Lead at Indiana University. Pierce had initiated a science gateway software-based Apache project in late 2008, in collaboration with Suresh Marru, who is now Vice President, Apache Airavata.
See Also: What is Apache AGE? (#1 in our emerging open source series).
Currently, Apache Airavata is aimed at the science gateway community wherein it helps scientists make optimum use of scientific software and data that are spread across distributed computing resources. These are handled by multiple providers who include universities and commercial entities. A multi-tenanted service is thus ensured which backs up multiple client gateways simultaneously.
Scientific gateway developers use Apache Airavata as their middleware layer between job submissions and grid systems. They can view elaborate information and manage jobs executed, resources accessed, and applications employed by their users. The latter can incorporate Apache Airavata with desktop and web interface components to oversee applications, workflows and generated data using Apache Thrift-based API. Its most valuable attribute is its capability to utilise components separately or as an integrated solution.
Airavata focuses on the following features:
- Desktop tools and browser-based web interface components for managing applications, workflows and generated data.
- Sophisticated server-side tools for registering and managing scientific applications on computational resources.
- Graphical user interfaces to construct, execute, control, manage and reuse scientific workflows.
- Interfacing and interoperability with various third party data, workflow and provenance management tools.
The project has seen over 9,000 commits since it joined the ASF and is regularly updated by an engaged community. Among some recent chunky projects, are this 2018 effort to make sure its JSON outputs can be swapped for something more fluid, using the Thrift API. Apache Airavata source code, documentation, mailing lists, and related resources are available at http://airavata.apache.org/.
