From roughneck to data dynamo: John Lewis’ Head of Data Products on perspective, Big Query vs Snowflake, and data ambitions
“We have staggering amounts of data. And we have the same problem as everyone else. It's all siloed: we need it in one place…”
Henrietta Sorel-Cameron, Head of Data Products and Platforms at retailer the John Lewis Partnership, has worked in some of the world’s more far-flung and challenging corners in a previous incarnation in the energy sector: Basra, Siberia, remote drill rigs on the North Sea...
It’s a far cry from the offices of the John Lewis Partnership, the UK’s largest employee-owned business; an omnichannel retailer with 34 stores and the Waitrose chain of supermarkets (which also have their own farm) – where she is responsible for a team that owns four data platforms and oversees eight product teams across marketing, trading and beyond.
An earlier career is not what she has sat down with The Stack to talk about, but we’re fascinated by her time in the oil and gas sector, which started as a graduate at the well level; working as a “roughneck” on jobs like mixing mud – fluid used to bring rock cuttings to the surface. (It is now data insights that she is looking to surface, although we’ll eschew the “data is the new oil” cliches to which the interview so readily lends itself).
But she admits after some gentle nudging that one thing this previous life has given her is “excellent perspective” on the inevitable hiccups that emerge in the data world: a career that often involved nerve-wracking journeys in armoured cars to get to drill sites reminds one that an ETL project going a little south is not, typically, going to result in grievous physical injury; however ferocious your CDO or chief executive is.
John Lewis data transformation: "We have staggering amounts of data..."
The John Lewis Partnership “JLP” is now six months into an ambitious 20-month (“quite aggressive”) data transformation programme. It is working with partners at Deloitte to consolidate a sweeping array of datasets from across JLP into the Snowflake data platform.
“It’s no secret that retail is tough. It has been for a while. So we need to be able to make data-driven decisions” Sorel-Cameron tells The Stack.
“We have staggering amounts of data. And we have the same problem as everyone else. It's all siloed and we need it all in one place…”
It is the company’s second attempt at a consolidated data warehouse she says: “Our first iteration was on Google’s BigQuery” she says. (GCP is still John Lewis’s infrastructure provider; it is running Snowflake on GCP.)
“We had built an overly complicated platform. I think that was one of the pitfalls of falling into a technology programme that didn't have end-users in mind and the business in mind. That was one of the big drivers to move. And after significant issues with ingesting data and incredibly complicated access management, our enterprise architects suggested Snowflake” she says, speaking at the SaaS company’s summit in Las Vegas.
Apart from the platform itself, what has changed this time? “Initially, it was very much a technology project. We've done a lot of work to change that mindframe and say, ‘no, this is cultural change; this is behavioural change… this isn't another tech project.’ I think that has gone a huge way to help build support. You have to build that trust with the business.”
And data requirements are “very varied across the business” she adds.
“We deal with supply chain, customers, partners, trading websites… there's so many different teams that are already using data. So for us the challenge has been ‘when is the right time to do this?’ Because this is fundamentally changing how the partnership is going to access data.
“It’s moving everything onto Snowflake – the front end of that is Tableau – whereas at the moment, that's a mix of all sorts of different systems. We haven’t turned anything off yet, but we have been in a situation where people get very frustrated with what they can't do, or the amount of manual workarounds that they have to do [ to get access to data.]”
So what kind of data is JLP sitting on?
“We've got data coming from our till systems. We've got data coming from suppliers; we've got weather data; footfall data; geospatial data and route planning data; customer feedback surveys; numbers from what's in the branches to what's in the lorries on the way to the branches, to what's in the warehouses; we've got picking numbers, we've got orders; we've got ethical scores from everyone; we’ve got third-party data on sourcing, fabrics, timber… we've got a plethora of data,” she says.
“It’s an exciting time. As part of our rollout we engaged each director, one at a time, to gauge what and business questions they want to answer. As part of the move on to Tableau, we are getting them set up with the training they need [so they have the] ability to [self-serve].”
Proving business value for the transformation started with some very concrete use-cases, she says: “A lot of our [data] use-cases were reporting-focused analytics, but we're increasingly seeing operational use-cases. For example, we partner with Deliveroo, so you can get your Waitrose order on your online shop. Availability all comes through Snowflakes; we pull it [stock data] from the warehouse, we do some magic with it, and then we send your order out to delivery. Availability is available immediately on the app, so that you can buy milk without being disappointed to be told it's out of stock. Those are the types of real-world use cases that we then prove our value into the business” she says.
That helped win approval earlier this year for what she describes as a “huge digital transformation programme… [because we were] quite far along on our journey and could prove that we can do this; you can trust us with the data; we do have good use cases; we do enable value.”
The opportunity to unearth new insights when all of those siloes converge and everyone can start drilling is substantial. It has been a team effort, she emphasises. As data journeys go, there is still some distance to go, but the wheels are rolling well and no armoured guards have been needed.
