Cloudflare says hackers stole source code, breached Atlassian and AWS environments
Incident forces huge hygiene effort, as Cloudflare rotates 5,000 credentials, triages 4,893 systems, reimages and reboots every machine after compromise.
Cloudflare says an attacker breached a production Atlassian server, stole 76 sensitive source code repositories, and compromised one of its AWS environments – in an attack that started after it failed to rotate three credentials stolen during an attack on supplier Okta in October 2023.
The attackers had access to its Atlassian environment and associated source code management system (Atlassian Bitbucket) for nine days.
They used this access to steal 76 repositories “almost all related to how backups work, how the global network is configured and managed, how identity works at Cloudflare, remote access, and our use of Terraform and Kubernetes" Cloudflare said on February 1, as well as breach an "AWS environment that is used to power the Cloudflare Apps marketplace."
What credentials did it use to gain access?
Cloudflare said the attackers hit it after getting access to "one service token and three service accounts... leaked during the Okta compromise."
What were they?
"One was a Moveworks service token that granted remote access into our Atlassian system. The second credential was a service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance, the third account was a Bitbucket service account which was used to access our source code management system, and the fourth was an AWS environment that had no access to the global network and no customer or sensitive data. The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This was incorrect... " Cloudflare admitted.
The incident in November 2023 pushed the cybersecurity and CDN company to rotate 5,000 individual credentials, launch “forensic triages” on 4,893 systems, and reimage then reboot every machine in its global network, as well as call in CrowdStrike for incident response, it said.
Cloudflare breach: “No services were implicated”
Cloudflare blamed the attack on a nation state that had “the goal of obtaining persistent and widespread access to [our] global network” – and said that its lateral movement had been contained: “No services were implicated, no changes were made to our global network systems or configuration,” the company insisted in a blog published February 1.
Despite the Cloudflare breach, the company said it had been able to limit the attacker's lateral movement. It added with regard to the stole repositories: "We’ve open sourced a large amount of our source code... our focus was not on someone having access to the source code, but whether that source code contained embedded secrets" – the AWS environment "was segmented with no access to global network or customer data.")
In a post authored by its CEO, CTO, and CISO, Cloudflare said (potentially embarrassingly) that the breach began after the attacker used “one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023.”
This is despite Cloudflare having boasted on October 20 that it had been able to “validate the scope of [that] incident and contain it before the attacker could gain access to customer data, customer systems, or our production network" and that "we were able to quickly mitigate the incident before the threat-actors were able to establish persistence…”
In that incident, Identity and Access Management provider Okta was alerted to a compromise of its systems by customer BeyondTrust on October 2. Troublingly, Okta did not confirm a breach to the company until October 19 after “we persisted with escalations” said BeyondTrust.
(The attackers had gained access to a Okta support environment that let it grab customer support files including API requests and session cookies shared with Okta for debugging purposes. Cloudflare, BeyondTrust, 1Password, and several others were hit. Okta belatedly confirmed that the “threat actor ran and downloaded a report that contained the names and email addresses of all Okta customer support system users.”)
Okta pointed out to The Stack that the attacker began targeting Cloudflare’s systems"almost a month after Okta shared guidance to customers to rotate credentials on October 19 when "we notified customers, shared guidance to rotate credentials, and provided indicators of compromise (IoCs) related to the October security incident. We can't comment on our customers' security remediations."
It was not immediately clear why Cloudflare had "mistakenly believed" the tokens in question were unused, as it admitted.
See also: Not HAR HAR as Okta breach escalates
Cloudflare said it embarked on a huge security hygiene overhaul after the incident, even returning hardware to manufacturers when the attackers tried (unsuccessfully) to “access a console server in our new, and not yet in production, data center in São Paulo… The immediate “Code Red” effort ended on January 5, but work continues across the company around credential management, software hardening, vulnerability management, additional alerting, and more” its leadership concluded.
"A large part of our “Code Red” effort was understanding what the threat actor got access to and what they tried to access. By looking at logging across systems we were able to track attempted access to our internal metrics, network configuration, build system, alerting systems, and release management system. Based on our review, none of their attempts to access these systems were successful. Independently, CrowdStrike performed an assessment of the scope and extent of the threat actor’s activity, which did not bring to light activities that we had missed and concluded that the last evidence of threat activity was on November 24 at 10:44."
Its blog in full is here. No customer action is reported needed.
