Novel cloud attack pivoted from K8s to Lambda, pulled IAM keys from Terraform
A recent attack tracked by researchers at cloud security company Sysdig saw the attackers use an exposed Kubernetes container to move laterally to the victim’s AWS account – gaining initial access by exploiting a public-facing service in a self-managed Kubernetes cluster hosted inside an AWS cloud account.
The company's write-up is a striking example of how attacks on cloud environments can escalate as well as how some poor hygiene practices can result in friction-free access to privileged data. (One small example: The attackers' target here had left clear-text IAM keys in the terraform.tfstate file inside of an S3 bucket...)
Michael Clark, Director of Threat Research, Sysdig, told The Stack: "I can't give details about the specific service, but it was exposed to the Internet. A mistake was made in the deployment and the image had a known existing vulnerability, which was not prevented by any policies/processes. Usually, the admission controller would prevent such an occurrence. This is one of the things which makes them so valuable...")
Research by fellow security firm Civo recently found that 53% of companies are concerned about the security of Kubernetes -- and the incident seen by Sysdig is another sharp reminder that cloud and container security are, for many companies, still greenfield areas that need significant work to avoid such data breaches. Sophos meanwhile estimates that cyberattacks in the cloud have increased by 56% over the past year.
Kubernetes container compromise to IMDSv1 creds
Dubbing the campaign SCARLETEEL, Sysdig’s researchers said that once the attacker gained access to the pod, they dropped malware (a cryptominer, possibly as a distraction) whilst gaining credential access through a worker’s temporary credentials in Instance Metadata Service (IMDS) v1 (used to configure or manage a running instance) to enumerate and collect information on its behalf using the cluster role permissions.
The team said: “Retrieving AWS temporary security credentials bound to the EC2 instance role from IMDS v1 is a very well-known practice that we’ve covered in previous blog posts. The attacker may discover the IAM role bound to the worker instance running: role_name=$(curl http://169.254.169.254/latest/meta-data/iam/security-credentials/) and later obtain the AccessKeyId, SecretAccessKey, and temporary token.
Join peers following 👉 The Stack on LinkedIn 👈
Sysdig said: “Due to excessive granted permissions, the attacker was able to: Enumerate AWS resources [and] find credentials of other identity and access management (IAM) users both set as Lambda environment variables and pushed in plain text to Amazon S3 buckets. [They then] used the credentials found in the previous step to move laterally. They directly contacted the AWS API, further enumerated the account, and proceeded with information gathering and data exfiltration. During this step, they were able to:
- "Disable CloudTrail logs to evade detection.
- "Steal proprietary software.
- "Find the credentials of an IAM user related to a different AWS account by discovering Terraform state files in S3 buckets" Sysdig said, with the attackers ultimately retrieving and reading “more than 1 TB of information, including customer scripts, troubleshooting tools, and logging files…”
The incident report by the cloud and container security specialist highlights a number of configuration and hygiene issues that an intelligent attacker can exploit to do far more than drop the cryptominer that may have been the simple initial intent. For example the AWS cluster role was misconfigured with excessive read permissions: "The original intent was to allow the reading of a specific S3 bucket, but the permissions allowed the role to read everything in the account, which enabled the attacker to gain further knowledge of the AWS account, including Lambda" Sysdig said, adding that "the Lambda function held proprietary software and the keys needed to execute it...Using curl or wget commands, the attacker successfully exfiltrated the Lambda code and stole proprietary code and software from the Lambda functions. There was also evidence that the attacker executed the stolen software.."
The attacker also succeeded in disabling some of the CloudTrail logs configured in the account because of extra permissions assigned to one of the users compromised in the previous steps. Another key failure (pun intended) of the infrastructure owners was how they had configured there use of Infrastructure-as-Code tool Terraform: "When Terraform is integrated and automated in continuous integration/continuous delivery (CI/CD) pipelines, the state file needs to be accessible with proper permissions. In particular, the service principal running the pipeline needs to be able to access the storage account container that holds the state file" as Sysdig notes.
"This makes shared storage like Amazon S3 buckets a perfect candidate to hold the state file. However, Terraform state files contain all data in plain text, which may contain secrets. Storing secrets anywhere other than a secure location is never a good idea, and definitely should not be put into source control! The attacker was able to list the bucket available and retrieve all of the data. Examining the data with different tools such as Pacu and TruffleHog during the incident investigation, it was possible to find both a clear-text IAM user access key and secret key in the terraform.tfstate file inside of an S3 bucket..." Sysdig's full write-up is here.
