CircleCI warns users to "immediately rotate all secrets" amid credential abuse evidence

Updated January 6, 11:00 BST: CircleCI has updated its advisory which deserves revisiting.

CircleCI is calling on customers to “immediately rotate any and all secrets” after a security incident. The breach appears to have occurred around December 21 and to have gone unnoticed over the Christmas period.

Credentials stolen in the attack are already being abused, with CircleCI customers saying on various channels that they have spotted (via the use of Canary Tokens) improper access of AWS credentials, for example.

CircleCI emailed customers January 4 recommending that they “review internal logs for their systems for any unauthorised access starting from December 21, 2022, through today, January 4, 2023.”

CircleCI CTO Rob Zuber then published an update on January 5.

The company has shared minimal details and no indicators of compromise with furious customers, saying it is actively investigating the incident and is “committed to sharing more details with customers in coming days.”

Zuber later added more guidance on how to rotate any and all secrets stored in CircleCI.

One security researcher, Daniel Hückmann, shared an attacker IP address for AWS credentials that were stored in CircleCI: 54.145.167.181 and suggested that defenders search cloudtrail logs for events from this IP. Those looking for audit logs from CircleCI,can request them from settings (documentation is here: Audit logs).

CircleCI security incident: Potential for downstream impact

The company provides hosted Continuous Integration and Continuous Delivery (CI/CD) services to millions of developers and lists customers that include Condé Nast, HashiCorp, PyTorch, and Snyk, among others.

With a large technology supplier customer base including providers of widely used open source tools, the potential for CircleCI breach to have notable downstream downstream impact is significant.

CircleCI CTO Rob Zuber said in a terse update on January 5: “At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well.”

He did not specify how many customers may be affected or what systems were exposed.

Developer tool providers have become a hot target for attackers as they often offer up tokens that facilitate further access to corporate accounts, for example credentials for cloud services.

The CircleCI security breach comes four months after GitHub warned of a major phishing campaign that started on September 16 and which targeted its users with emails that impersonate CircleCI to steal GitHub account credentials and two-factor authentication (2FA) codes via Attacker in the Middle (AiTM) attacks.

The incident also comes after Travis CI, another continuous integration provider (used to build and test software projects hosted on Bitbucket, GitHub, GitLab et al), exposed tens of thousands of user tokens via its Travis CI API in 2022 and was castigated for what was seen by customers as a deeply inadequate response.

As security company Aqua noted at the time, the incident allowed “anyone to access historical clear-text logs. More than 770 million logs of free tier users are available, from which you can easily extract tokens, secrets, and other credentials associated with popular cloud service providers such as GitHub, AWS, and Docker Hub.”

Are you a concerned CircleCI customer? Have views. Get in touch.