We got hit by ransomware. This is how it played out.
"We ran around and physically cut the cords connecting the servers to make sure the virus couldn’t spread any further... There was an eery silence."
On May 7th, 2020, data storage specialist Spectra Logic fell victim to a Netwalker ransomware attack. Senior IT director Tony Mendoza explains how it happened -- and the lessons the company learned. [Editor's note: this exercise in transparency is wholeheartedly the right approach to sharing an all-too-common experience and we applaud it. Everyone benefits from understanding threat vectors, resilience play books, etc.]
When did you first suspect you had been hit?
The morning of the incident two members of my team reported that a lot of routine processes were not working properly -- none of which were related. This set off my alarm bells. It was when a third staff member came in to tell me that he was experiencing the same issues that the penny dropped, and we rushed to the data centre to see what was going on.
We found a ransom note hidden in the files on one of the servers. It said we’d been hit by the “Netwalker” virus. The ransom was approximately $3.6 million, and it had to be paid in Bitcoin within five days.
We then ran around and physically cut the cords connecting the servers to make sure the virus couldn’t spread any further. When we brought them all down, there was an eery silence. Our data centre hadn’t been silent since 2012 when we moved in to our new buildings!
What was the vector through which the infection had happened?
One of our employees, who was connected to our network by VPN from a private laptop, opened a piece of malware that normally would have been blocked by our virus protection software, Sophos. Yet, due to the rapid switch to remote working, Sophos wasn’t installed on that system. This is something that never would have been allowed in normal circumstances.
Had you considered any practice sessions around a ransomware scenario?
We have a fairly comprehensive business continuity plan that includes disaster recovery. While we continually practice and improve this plan, we never considered ransomware as one of our disasters. The plan itself was extremely helpful in recovering our systems but we had to make some game-time decisions that we now include in our plan.
Did your company Red Team or pen test before, if so, how often?
Yes, we do pen test annually and keep an extremely secure infrastructure. We also understand that email phishing/malware is a weakness for organisations, and although we have systems in place to provide threat protection, end user behaviour continues to be where we focus our security efforts.
How robust and rehearsed was your backup/resilience strategy?
We had prepared for a situation like this for years, and we already had a strong Disaster Recovery plan in place. We rely on CommVault for our daily backups to both Spectra tape and Spectra BlackPearl NAS, and we augment those backups with VM snapshots and StorCycle software for data migration.
Our first steps were to check the backups to see what we had for a Disaster Recovery scenario, and to make sure that our email server was not compromised. It wasn’t, thankfully. We reached out to the FBI and also to our cybersecurity insurance company, who set us up with a security firm that deals with these types of scenarios on a daily basis. They advised us on the exact steps to take in order to “stop the bleeding.”
Once we confirmed that we had a backup on tape, this gave us the confidence to refuse to pay or negotiate the ransom.
We then carried out complete wipes and rebuilds of every server. It was estimated that it would take four to six weeks before everything was back up and running. The company was back up after five days, and it took another week to get all of our systems back online. After another two weeks, we had worked out any lingering issues. Ultimately, we overcame the attack with virtually no data loss and absolutely no data stolen, confirmed by a third-party security audit.
Looking back, what are the things you'd do differently and urge everyone to do?
First, I would urge everyone to abide by the golden rule of keeping multiple copies of data on multiple mediums in multiple locations.
If every copy of your data is compromised, even the best IT experts will not be able to help you restore that data. We would have had to negotiate the ransom had we not kept a copy on tape behind an air-gap. We are aware that encryption-by-ransomware is not clean or simple, and even if you pay the ransom, decryption tools are not guaranteed to work.
We had quite recently invested in cybersecurity insurance, and I would highly recommend this for all companies that aren’t large enough to warrant an in-house cybersecurity team. Having experts close at hand was critical. There is a tough balance to find when it comes to IT security. You can always add more security, but at some point, it will have an effect on the user experience, and could even hinder company goals that are achieved through IT. We would recommend consulting with security experts to create a strategy that balances risk and IT policy.
The unfortunate truth is that organisations have to presume that they will be hit. I read that ransomware attacks have increased by an incredible 900 percent in 2020 (VMware Carbon Black report), and that Netwalker criminals have raked in over $25 million in ransom payments since March (McAfee report). We simply have to follow the Scout motto and be prepared.
What have you learned, what are you changing, how are you tightening security at your company?
In addition to continuously assessing our Disaster Recovery plan, we are currently exploring different methods to replicate our disk snapshots to a dark site. This decentralisation of data can create management challenges, so we’re looking into ways to manage it centrally at the same time.
Given phishing (often a vector) is getting so sophisticated, how do you hope to tackle that?
Again, we have systems in place to provide some level of threat prevention, but this is still the weak point. Luckily our wounds are still fresh and our end users have become hyper-aware of phishing threats.
We do offer voluntary training to help identify threats, but we feel that as users become more complacent again, we may have to require training and send out targeted phishing tests to our employees from our IT department. At this time, we rely on upgraded cybersecurity systems to help limit the blast radius of an attack.
Any last-minute advice to other IT professionals regarding Ransomware mitigation?
Unfortunately, a ransomware attack is not a matter of “if”, but of “when”. Be prepared to recover! Make sure you are confident in your data backups/replication. Protect those data end points with air gap if possible. And of course, invest in as much cyber-threat protection as your budget and your culture can tolerate.
As Senior Director of Enterprise Business Solutions, Tony is responsible for the entire IT infrastructure at Spectra Logic. He oversees hardware, DevOps, software and cloud system integration at the company.
