The Big Interview: Bloomberg CTO Shawn Edwards on democratising alt data and educating the cloud
"Anybody can throw some data at a neural net and go 'gee whizz!'"
“If you've talked to enough technologists that work at banks, no matter how great you are, and how well you do, you’ll know you're always a cost centre” says Bloomberg CTO Shawn Edwards, sitting down to talk with The Stack.
It’s perhaps not a view he had 19 years ago as a technologist at Bear Stearns (“way back when it was a good company” he says drily, as a Bloomberg communications professional sitting nearby winces gently.)
His point isn't to dunk on banks though, but to emphasise the importance that folks across the IT function are afforded at Bloomberg: “At banks traders and portfolio managers are always the rock stars” he adds: “Here at Bloomberg technologists are the rock stars; I came reluctantly to interview at Bloomberg. I was blown away.”
In his two decades at the company, Bloomberg CTO Shawn Edwards has built his own supergroup of around 350 engineers, architects, developers -- and created a unique and expansive role for the CTO’s office.
His team is the sharp end of the spear of a technology function that spans over 6,500 people. Part ideas incubator, part research house, part product factory, part wrenching-on-infrastructure, the CTO’s office responsibilities span market data and compute infrastructure; AI research, and information security.
See also: US Army CIO Raj Iyer on rolling out Virtual Mobile Infrastructure
Edwards also oversees Bloomberg’s open source office and runs engagement with the academic community, for example in the data science space, and more. It’s a sweeping remit. (Bloomberg was a prominent early adopter of tools like Kubernetes – starting to work with the container orchestration toolkit in 2016 when it was still in alpha – and contributed heavily to its development. The company has been dubbed by the Eclipse Foundation a “leading example of how the financial industry can support and contribute to open source communities”)
As he puts it: “[In the CTO’s office] we’re researchers, product managers, and head up the innovation lab. We have started a number of products within areas like advanced analytics and event platforms. We get salespeople and product managers and technologists working with a larger product team to incubate ideas.“
"It’s a really interesting mix of responsibilities. I tell people I have the best damn job in the company.”
Bloomberg: A bigger tech influencer than many realise
Bloomberg, for those not au fait with the company – or only with part of its proposition – was founded in 1982 as a news, data, analytics, and communication services for the global business and financial world. A world-class global IT infrastructure builder, when the cloud was just a twinkle in Jeff Bezos’ eye, Bloomberg runs arguably one of the world’s largest private networks: an expansive network of its own 140+ sites in over 100 countries linked by a mix of its own fiber infrastructure and those of other carriers.
Edwards cut his teeth on this infrastructure side, he recalls, telling The Stack: “I came as an individual contributor; they didn't have a CTO office. I started working on infrastructure and various customer-facing products… you know, getting woken up at 4am [and then] created the first UX design team at Bloomberg.”
The company, which sits on over 100 petabytes of data, now provides access to 575+ exchange products from more than 365 trading venues around the globe, with 24/7 customer support in 17 different languages.
Its infrastructure, which includes some 70,000 servers across three primary data centres, furnishes capital markets customers with a consolidated feed of normalised data from 35 million+ instruments – helping drive enterprise clients’ investment workflows – while its financial products team provides execution and order management systems, financial data management services, integration and distribution technologies and more.
Innovating around these products is a big part of Bloomberg CTO Shawn Edwards’ job.
Recent launches born in his office include BQuant Enterprise, a cloud-based platform for quantitative analysts and data scientists in the financial markets which the company boasts is the “first off-the-shelf data science solution that is designed specifically for financial markets.” BQuant lets users write a Python function to access Bloomberg’s data sets; its own Quant research team used BQuant and an alternative data set of weather feeds to assess snowfall impact on US retailers’ performance or cyclones’ impact on manufacturers.
That launch came as customers are increasingly wanting to consume wildly heterogeneous datasets in a wide range of ways, including pulling data streams into their own environments, the cloud or indeed data exchanges on third party platforms: "We're not there yet, but I want to make every one of our data elements accessible through an API so our customers [can] build the best of breed workflows and best of breed analytics" he says.
"No! That's not how markets work..."
Despite rolling so much of its own infrastructure, Edwards is quick to point to early cloud efforts however: "We evolved to make [data provision] cloud-native before the financial industry was using public cloud,” he says, adding that what has changed in recent years is how customers want data delivered: “They want it in Parquet, to have it ready in S3, or Snowflake; we embrace all of that... One of the first products that went to public cloud – a little counter-intuitively – was our market data feed B-PIPE, which is now in all three major public clouds.”
(To give a sense of the scale at which B-PIPE operates, consider that the price of a single traded asset can update hundreds of thousands of times in a day. A 2018 presentation by one Bloomberg technologists on getting B-PIPE to the cloud noted that it handles upwards of 80 billion "ticks" a day -- all of which needs to get to customers ranging from hedge funds to multinational banks that heavily prioritise low-latency.)
Those early cloud efforts were an education for the hyperscalers he notes -- which were not initially primed for the low-latency setup that market participants needed -- saying their early proposals for Bloomberg were more set up with a mindset geared towards hosting websites than needing to handle streams of data at pace.
"“One of the problems was ‘how do you take a high volume, low latency stream of data, and go from one tenancy to another?’ The original thing that they wanted us to do was, ‘oh, you just go out to the internet, there's the egress; then that person just reads it from the internet, and that's an ingress’ – and of course, they ring the toll bill both ways. No! That’s not how markets work… That's going to increase jitter, latency and all sorts of madness. So we actually worked with them [AWS] on their PrivateLink and on their load balancer to help design it.
"We're seeing good results" he adds, noting that "we [also] built BQuant with the intention to be in the cloud... having that full SciPy, Jupyter Notebook tech stack with the ability to build applications and distribute then to other users, but it also has a full [underlying] distributed compute stack -- Apache Spark, TensorFlow, PyTorch – that allows people to do massive model building and testing with back testers and optimisers”.
"Anybody can throw a neural net some data..."
Another focus of his team in recent years has been a “very large effort to really rethink how we store and model our data. Linked data is something that's incredibly important for us because we are sitting on rich time series data. We recently built a query capability called BQL that provides the ability to query and run massive time series calculations across all of our asset classes; across all of our data. With this query language, we understand currency conversions. We understand calendar alignments, we understand how to join an issue to a supply chain to all sorts of other things. The dream is that you should be able to query anything you want. We're not there yet," the Bloomberg CTO notes, "this is a journey; we're still adding that capability."
That will let his team help develop fresh products and insights around alternative data, he suggests, hinting at some projects in gestation: "Alternative data is one of those terms that people talk about a lot, but there's very few people who can actually have the capability or the compute power or the specialties to really take advantage of alternative data. We want to take the ‘alt’ out of alternative data. We think we can bring it to the masses..."
"Alternative data is large and messy, but we think it will become table stakes and [if] we can democratise it; that’s really exciting" he says. "This requires a lot of leveraging human expertise, and I think that's where Bloomberg has the advantage. Anybody can take some neural net and throw it some data and say, ‘gee whizz’ – but, to me, you need the machine learning expert, you need the computational power and you need the third piece: the domain expertise. Our data team has deep expertise in various data domains. So we work with them closely not only building their tools and their processes and their pipelines, but also on tackling specific data domains."
Throughout the conversation, one thing shines through: Edwards loves his job. In a merry-go-round market of CIOs and CTOs, someone doing a 19-year-stint in the same company and still seemingly full of enthusiasm is a rarety. That early jab at banks was no doubt a thinly veiled attempt to attract more data science talent in a hot job market, and Edwards certainly talks a good game: "There's something incredibly powerful about working for a company where instead of being buried down below working on improving ad placement by 1% or whatever, you're actually having a direct connection to solving problems. I love our collaborative environment -- love to get on the whiteboard with this diverse set of interesting people! Bloomberg's a big believer in creating teams that are cross-departments and cross-functional. Great ideas come out of that: you're on the whiteboard, you're challenging each other's ideas, you have this really collaborative, collegial group. It's so empowering."
