Recovering from ransomware: Are your backups enough?
Some victims can't start restoring because their backup solution requires AD to log in
In the past year there have been more that 130,000 news stories about ransomware alone, writes Alex Papadopoulos, Director, Incident Response Consulting, Secureworks. In the first three months of 2023 Royal Mail, JD Sports, WH Smith and Ferrari are amongst those to have hit the headlines after suffering attacks. Gangs like Lockbit, Cl0p and Conti are now as famous, in tech circles at least, as The Beatles and Queen. Well.... sort of.
Ransomware is a serious and ongoing threat to businesses, accounting for a quarter of all cyber-attacks. While businesses today are more cognisant of this threat than ever before, it isn’t stopping the onslaught. So, imagine the worst happens and you’ve been hit. What happens next? Whether or not you decide to pay the ransom, you need to get your business back up and running. Recovery is crucial But it’s not as simple as some might think.
Picture the scene, we’re undertaking a tabletop incident response exercise with a customer and the conversation goes like this: “What would you do after you’ve been hit with ransomware?”
“We’ll just restore from backup.”
“Ok, how long do you think that will take?”
“Two days?”
Creating backups has been part of good business practice for decades. A common misconception is that in the wake of an attack, you can just flip a switch and backups will get you up and running right away.
The harsh reality is that the whole process can take huge amounts of time, if it can be done at all.
Recovering from ransomware: 2 Questions...
When you think about building for cyber resilience and recovering from ransomware there are two questions that you need to ask about your backup strategy:
- Are your backups protected against someone with full administrative access?
Imagine, a threat actor has gained access to your network. The likely scenario is that they have full administrative access to your Active Directory. With domain admin level access, they have the keys to everything, including your back up. We see adversaries head straight for the backups, which they destroy, before they go ahead and do the rest of the damage. They don’t want you to have a quick recovery plan and they know backups are core to that.
- How long will it take to restore if whole system is bricked?
Best case scenario, you have your backups intact. Have you actually tested how long it will take to restore everything? If you have a fully bricked system you will need to start by reinstalling the operating system first, then applications and services, before being ready to restore any data from backups – what we call a bare metal restore.
The reality is that restoration can take significant time to achieve.
Join peers following The Stack on LinkedIn
A bare metal restore on my laptop, for example, might take four hours. If you have 1,000 laptops to restore, that’s a lot of work over a lot of half days, if you have one person you can dedicate to it. Of course, if you have the resources, you can hire in labour to cut down this time. But what about the servers?
Restoring database servers or virtualisation servers requires specialised expertise that few people have. It might take a week to fully restore, configure and bring one core system back to a fully operational state.
And how many such core systems do you have?
Recovering from ransomware: Recovery in the real world
Let me put that into context in one of our real incident response engagements.
Our customer suffered a serious ransomware incident. In this case, they felt the ransom payment demand was something they could and should pay. So they paid it out on day one and got the decryption key.
But the ransomware had impacted most servers in the business. More than 5,500 servers were affected. In this case resource was not an issue. They hired staff where they needed them and supported the efforts financially as necessary, with the single goal of getting the business fully operational as quickly as possible.
Even with unlimited resources, it took three months to get them 95% recovered – and that was the most they could achieve, as some systems had been corrupted and could not be recovered.
"Functionally, they couldn’t even start the process of restoring because their backup solution required Active Directory to log in"
In another incident response engagement, the impacted organisation had backups which survived the intrusion and ransomware. What didn’t survive was the victim’s Active Directory.
Functionally, they couldn’t even start the process of restoring because their backup solution required Active Directory to log in. They needed to authenticate, and nothing worked. The customer had to fix this basic building block before they could do anything else. It’s a basic building block, but one that requires specialist expertise which they did not have inhouse. So even though they had backups, there was no way of using them.
It’s easy to have a simple, romantic view of restoring from backups. When we delete a file by mistake, we are used to being able to ask the backup admin to restore it. 20 minutes later, all is right with the world.
It's very, very different when you're looking at your entire network and everything is in flames, and someone says “How quickly can we restore?”
What about the cloud?
It’s not just the on-premises systems we need to consider in cyber resilience. Mission critical workloads are shifting to the cloud and must be part of the backup and restore strategy. The cloud is not a magic land that no one can touch. If your Active Directory gets compromised and cloud assets are within the same authentication domain, then there's no reason why they will not get impacted by the attack.
Restoring cloud assets carries its own complexity. Depending which cloud platform you're on and exactly which capability you use, there will be different types of backups. And this is something completely different to the traditional on-site backup that most people have in their minds when they speak of backups.
For example, if you work on AWS, they published a blog which shows the different types of backups that exist in AWS and how to use the right tool for the restoration functionality that you need.
There are a number of options there. It's not one size fits all. And all too often, organisations have not put enough focus on understanding how cloud storage aligns to their business and therefore business risk.
Prevention is better than cure
While you absolutely need to have backups, the reality is, that you really, really don’t want to have to use them to recover from a cyber attack. It is infinitely cheaper and more cost effective to invest in prevention, early detection and containment rather than having to contend with restoring everything from scratch.
To build resilience and be capable of recovering from ransomware, businesses must plan for failure, which means you need good backups but you should also ask the following questions:
- Have you taken the time to consider, test and document how you can recover under all the scenarios we’ve just looked at?
- How many systems do you have to potentially recover? What are the complexities? How quickly can you do it?
- Do you have an alternative infrastructure in place, or the means to secure such infrastructure quickly, to keep the business running while you do the full restoration?
- Have you tested and do you have the capability to restore your Active Directory or whatever identity and access management (IAM) technology you use?
Ransomware remains a very real and potentially very costly threat to businesses. It’s crucial to have solid cyber hygiene. Prevention will always better than cure, so make sure you have best in class detection and prevention in place. But attacks happen, so invest in, understand and test your entire recovery strategy. Backups are a crucial part of that process, and should be tested prior to the business needing to rely on them – but do remember that backups are just one part of the resilience puzzle.
