Taking generative AI to production: What CTOs need to consider

The terms "AI" or "artificial intelligence" were used 827 times on just 76 earnings calls over the summer. Mounting levels of hype sometimes drown out the true shape of progress at the enterprise level however – where Chief Technology Officers and others are scrambling to deliver on AI’s compelling promise and deliver real return on investment.

As DataStax’s Chief Product Officer (CPO) Ed Anuff notes to The Stack: “With this constant sugar high of experimentation, we don’t typically get as much visibility into those moving beyond the prototypes to production. What I'm seeing though, is that there's multiple projects underway at all the established enterprises and startups we work with.”

Getting generative AI into production

He says: “They are working to get AI support assistants – anything that supports customer interactions – into production. I think we’re going to see widespread generative AI production use cases, conservatively, by Q2 or Q3 of next year. One major national retailer is targeting November of this year. That’s ambitious, but they're moving fast…”

The potential here to improve customer interactions whilst also sharply trimming onerous service overheads is powerful; as is the potential to gain hugely useful real-time insights with natural language searches into mission-critical data. Yes, there’s a lot of work to do to get data cleaned and tagged for training in most organisations, but it is emphatically happening – because deploying generative AI meaningfully, for most CTOs, means going beyond spinning up a shiny user interface hiding a plugin, or API calls to someone else’s proprietary large language model.

The real opportunity at the enterprise level comes in deploying proprietary data to underpin and train generative AI applications, with Bloomberg and McKinsey among the growing number of organisations to have showcased and explored LLMs trained on their own datasets. (DataStax meanwhile has supported online travel giant Priceline and SkyPoint, a SaaS company that works with senior citizens and care-givers, to build out native generative AI capabilities that are already now in production.)

Generative AI and first-party data

Production use of generative AI applications trained on first-party enterprise data in short is coming soon, at scale, says Anuff: “The difference between playing with ChatGPT, and using generative AI for your business is your own data. If you’re going to build a sales agent, you want it to have access to your proprietary data, like all the products you have in inventory – and have that in real time. You want to know what’s in stock, the order history, previous things customers bought.”

That, he notes, is very demanding on your database at enterprise scale.

The best vector database? DataStax CPO says think deeply

DataStax’s Astra DB – a heavily optimised, zero-friction and cloud-native drop-in for Apache Cassandra – is battle-tested in production at truly global enterprise scale. New vector search capabilities meanwhile make Astra DB AI-ready; by enabling complex, context-sensitive searches across diverse data formats for use in generative AI applications.

(A vector represents the semantic context of a topic or a piece of text or any unstructured data. The ability to query that vector is critical to building generative AI out in a performant way. Most database firms are scrambling to add native vector search capabilities to their offerings.)

CTOs looking for the best vector database should be mindful, says Anuff, that whilst there are various new vendors on the market promising databases exclusively and “purpose-built” for AI applications, ultimately a “vector database is a database: it needs to support vector data types natively, deeply in the code and it needs to have the facilities for querying efficiently. But it also needs everything else…”

What does he mean?

“There's non-vector data that goes into the database; you need the ability to scale that database; backup and restore; bulk import data; connect it to other systems; you may need HIPAA and PCI compliance. There’s a whole set of concerns that don't go away, just because you’re suddenly building AI;  in fact, often, they become more critical.”

DataStax is betting that as generative AI deployments using first-party data gain steam at the enterprise level, its DBaaS, AstraDB, can be the vital one-stop-shop for high-scale, generative AI applications.

See also: Apache Cassandra: Good for gay clusters and rarely on fire, but everything needs a little TLC

That may be the bread-and-butter of data-heavy workloads like LLMs. But DataStax is also, Anuff says, using the company’s data nous to support customers in building custom chatbots powered by their own data by introducing “Retrieval Augmented Generation” (RAG) capabilities into their platform.

That’s a way to retrieve up-to-date data from outside a foundation model and augment prompts by adding the relevant retrieved data in context; if you’re offering holiday advice via an LLM trained on slightly stale data that is unable to account for the fact that the customers’ chosen destination is literally on fire, for example, all that heavy data-wrangling is moot. That’s why RAG is critical and where DataStax is putting a lot of work into supporting its customers to ensure that their models are delivering timely responses.

Change is clearly happening fast in the space. By 2025, more than half of software engineering leader role descriptions will explicitly require oversight of generative AI, according to Gartner on August 30. And speaking with The Stack, Anuff drives home the point: The AI hype may sometimes be infuriating, but this is not blockchain 2.0, where genuine production use cases with a compelling ROI were nigh-invisible.

Rather, he emphasises, “every business I speak with is racing to provide human-level interaction for their customers via generative AI.

Up first is service support: “A lot of the automated agents that people interact with right now are just annoying and incredibly stupid.”

“Those are going to be replaced very quickly with these systems.”

That means data platforms must evolve by serving data in real time, being available 24/7 and able to ingest massive amounts of information. If you want to build out massively scalable AI agents with your proprietary datasets, you need the right partners, tested technology, and the depth of experience DataStax brings to its partners, he suggests. Because believe the hype. The world is changing fast.

Produced with DataStax.