Searching for search: UK Gov wants to dump GOV.UK Elasticsearch
"We would also like to reduce the long term maintenance work..."
The UK government wants to dump its in-house Elasticsearch GOV.UK service in favour of a “measurably better” solution, preferably a managed UK gov search service, according to a new tender.
With a budget of £898,000, the Government Digital Service has already done its preliminary research, specifically naming Google Programmable Search Engine, Bing Custom Search, AWS CloudSearch, Algolia and Miso as being of interest. The budget will cover evaluation and trials, implementation, and finally support and optimisation, over nine months.
“Our goal is to significantly improve the quality of the GOV.UK site search experience for users by improving the relevancy of results. We would also like to reduce the long term maintenance work required of the GOV.UK team,” said the tender.
See also: UK Gov digital strategy — sensible but stale, and with a glaring gap
The UK gov search move will mark the end of a decade of Elasticsearch, which replaced Solr in the summer of 2012 as the search engine behind GOV.UK. But subsequent posts about the website’s search function have shown the GDS has struggled with keeping Elasticsearch up-to-date.
In 2017 “what should have been a simple code-change” to a make a search tool compatible with Elasticsearch 2.4 caused certain results to stop showing for users, forcing GDS to roll back the change and review its code. And in 2020 the move to Elasticsearch 5 saw GDS duplicate its search API to run the two systems in parallel, then migrate one service at a time.
While that UK gov search migration went relatively smoothly, the move from v5 to v6 caused some headaches, according to the blog post: “Not only did we see Elasticsearch 6 performing significantly worse in the metrics, we even received a couple of support tickets about strange search results! For example, a search for ‘theresa may’ gave Tax your vehicle as the top result, as it's popular and has the word ‘may’ in the page text.”
This was due to significant changes in “relevancy” in Elasticsearch 6, which the GDS team had not expected to have such a major impact. After reworking, the team managed to get the newer version to deliver results “on par” with Elasticsearch 5.
“We decided that we could live with things being a little worse, as after the upgrade we planned to start work on improving search performance anyway,” noted the post.
Given these issues, it’s understandable why GDS would prefer to move to a managed option for UK gov search.
