Securing CI/CD pipelines: Protecting your assets
"The CI/CD pipeline is the ultimate delivery vehicle..."
A Continuous Integration and Continuous Delivery (CI/CD) pipeline is an incredibly powerful tool, but it can also be a liability if it's not protected, writes Daniel Riedel, Copado. Think of it as a castle -- it contains the most valuable stuff in the kingdom. As a result, it's also a target, so it needs to be a stronghold that's protected from interlopers. In any kingdom, having an insecure castle would be worse than having none at all and the same is true for CI/CD pipelines. Training and culture, automation, and auditability are all critical tools that protect CI/CD pipelines and prevent breaches. This is why DevSECOps is so important.
Why a Bad CI/CD pipeline Is Worse than None at All
CI/CD is exceptionally powerful at maintaining software at scale. However, it requires a significant amount of responsibility. Companies must implement strong security controls to protect it. Otherwise, a bad actor can use that CI/CD to ship their malicious code to thousands of users. That's exactly what happened to SolarWinds.
In this case, the bad actors targeted the third-party software, the SolarWinds Orion Platform. They started their attack with a successful test code injection in October of 2019. However, it wouldn't be until four months later when the real attack began. They injected a code dubbed Sunburst and waited. One month later, SolarWinds shipped out the updates and they were installed by more than 18,000 customers. It was finally discovered in December of 2020, but by then, much of the damage was done.
SolarWinds shows us that CI/CD pipelines can be weak, even for recognized industry leaders. However, just like a castle is an asset and also a security tool for protecting royalty, a CI/CD pipeline can be an asset and a tool that protects a company.
The "Keys" to Keeping Your CI/CD Pipeline From Being a Liability
If the CI/CD pipeline is the castle, then credentials are the keys to get in. One wouldn't just leave them under the front mat or one of those fake rocks for anyone to find. They'd protect them and make sure they were only accessible for those who absolutely needed them. In CI/CD pipelines, credentials can be the single biggest vulnerability. Addressing that weakness means looking at training and culture, automation, and auditability.
Training and Culture
Training and culture go hand-in-hand. Training teaches people the standard; culture ensures they follow it. Everyone needs to take accountability for security in an organization and that needs to be a clear part of training. When addressing the importance of complex passwords, it's not enough to direct individuals to establish strong ones -- it's vital to communicate why it's necessary. Companies need to explain what's at risk when setting insecure passwords to truly ingrain the importance. They also need to make it easy for workers to establish, store and remember secure passwords. A few things to consider include
- Password entropy systems: These programs set minimum standards for password creation and will reject any attempt to use an insecure option. Someone attempting to use "password123," for example, would be immediately blocked from doing so, and the minimum standards would be reiterated.
- Random password generators: These tools will generate completely random passwords. They have the added bonus of overcoming issues with social engineering, as the password in no way will relate to anything personal about the employee.
- Password managers: Password managers store and retrieve highly complex credentials so workers can enjoy the security of a strong password without the need to memorize it.
“It is borderline criminal” — US Air Force CSO quits with a bang over DevSecOps failures
Leaders set an example. A manager who writes their passwords down on a piece of paper or uses common, easy to guess ones makes it seem like such behaviors are acceptable. After all, why are workers going to hold themselves to standards that even managers don't follow?
When something does go wrong, it's essential to avoid the "blame game." That is a toxic part of an organizational culture that keeps individuals from admitting to breaches. Instead, they try to hide them to avoid becoming the scapegoat. Breaches are the responsibility of everyone, and that makes everyone accountable for mitigating them.
Securing CI/CD Pipelines: Automation Matters
Automation eliminates the risks of insecurity created through human error. On top of that, the CI/CD pipeline is the ultimate delivery vehicle. Companies can add these tools to their system and use their pipeline to secure their pipeline. The National Cyber Security Centre has a few suggestions for facilitating this:
- Key management: This is the enterprise level of password management in that it allows organizations to control access and store credentials.
- Configuration checks: Hackers constantly scan repositories for credentials that are accidentally stored in configuration files. If a developer makes a mistake like that, the company has about 43 seconds to remove it and change it. Otherwise, a breach is all but guaranteed. Automated configuration checkers will alert developers to these issues.
- Least privilege access model: If they don't need access, they shouldn't have it. If they only need it for a single task, it should be revoked immediately after. Single-use passwords or keys are ideal tools when someone is only entering a system for a single project.
- Automated key rotation: Secret keys can control the functions necessary to build and deploy new code. A program that automates key rotation will ensure that access credentials don't go stale.
- Pipeline enforced rules: The pipeline itself can govern what code is acceptable pre-deployment and what will need to be rewritten. This mandates proper adherence to policies and limits vulnerabilities from misconfigurations.
- Hard breaks: While this section is all about automation, sometimes, it's best to go in the other direction. A hard break establishes manual approval requirements for code deployments. This can build in some level of oversight by triggering an approval process in the event of a sensitive deployment.
Auditability
Log management was a crucial component in discovering the SolarWinds breach. It was discovered by the aptly named cybersecurity firm FireEye. While completing an audit of their system, they found an issue. According to Charles Carmakal, the leader of the company's incident response team, they searched through 50,000 lines of source code to discover the backdoor in the SolarWinds system. That review helped them uncover the primary issue and mitigate its damage.
Auditability connects directly to proper logging. Immutable, automatic logging of all actions within a system helps companies get a real-time snapshot where anomalies can be detected and traced. Logs should be held in a separate, secure place to keep bad actors from changing them to escape detection.
CI/CD pipelines are a valuable tool -- right up until they're not. Then, they become a liability for a company and all of its customers. However, the CI/CD pipeline can become a self-governing entity with the proper implementation of automated tools. Much like a castle, it's a thing that must be protected but is also an instrument of protection in and of itself.
