235.7 billion downloads; 2 staff on call. PyPI briefly shuts up shop amid burnout, malice
An uptick in malicious new projects being created on the Python Package Index (PyPI) repository forced it to shut down new user registrations over the weekend in a worrying sign for the open source project.
PyPI is a repository of open-source Python packages supplied by the worldwide community of Python developers. Over the past year it served a massive 235.7 billion downloads for its 448,941 projects.
“New user and new project name registration on PyPI is temporarily suspended,” admins said on May 20.
“The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave.”
PyPI malicious packages: “I was burnt out after two weeks of doing it”
It lifted the prohibition just a day later, but the incident is a reminder of just how much critical open source infrastructure is maintained by threadbare teams – the Python Software Foundation’s Director of Infrastructure told The Register that over the weekend two of the three people who respond to incidents were on leave, leaving them and one other to field every security report, as reports of malicious packages climbed to around 40 per day: “It was just getting to the point where I didn’t feel confident that I as an individual was going to be sitting here all weekend watching that inbox. Effectively I was burnt out after two weeks of doing it.”
Follow The Stack on LinkedIn
PyPI has been hit harder before. In February security firm Sonatype said that it had spotted malicious actors flooding PyPI with over 5,000 malicious packages that drop a Windows trojan via Dropbox…”
Checkmarx said: “In the past few months, we have witnessed actors publishing overwhelming amounts of malicious packages in several open-source registries; among these is the NPM incident early last month that flooded the JavaScript package manager and caused a sporadic denial of service. Late last year, we saw a similar incident on the Nuget package manager, which included over 140,000 malicious packages.”
Python Software Foundation to start charging for organisation accounts
Funding for two new roles from the OpenSSF and the Linux Foundation and AWS respectively should bolster the team soon. Strikingly, given the sheer volumes of downloads PyPI sees, it remains heavily manual work and Ee Durbin, the infrastructure director told The Reg that “one of the projects they’re going to be working on is building us out to the point where we have automation-friendly ways of responding to these [incidents].”
Last month the Python Software Foundation announced that in a bid to build “financial support and long-term sustainability” of PyPI it would start charging for corporate members for a new feature: organisation accounts, or self-managed teams, with exclusive branded web addresses: “Increased revenue for PyPI allows it to become a staffed platform that can respond to support requests and attend to issues in a timeframe that is significantly faster than what our excellent (but thinly spread) largely volunteer team could reasonably handle” it said.
ASF also gets hammered with security spam, sees heavy workload
Earlier this year The Stack reported on similar workload challenges at the Apache Software Foundation, where all vulnerability reports that are emailed in are handled by volunteers; with the organisation in 2022 hiring a sole administrator to help. This group received over 22,600 emails to its security email address in 2022, with manual spam filtering and thread grouping stripping that back to around 1402 non-spam threads.
Each email had to be manually reviewed however to avoid missing real issues. As the ASF noted: “Unfortunately security reports do sometimes look like spam, especially if they include lots of attachments or large videos, and so we are careful to review all messages to ensure real reports are not missed.”
Actually excluding spam, many were “the unfiltered output of some publicly available scanning tool, and often come along with a request for some sort of monetary reward” leaving 599 reports of new vulnerabilities in 2022, which spanned 122 of the top-level projects. These 599 reports include both external reports, as well as issues found internally by projects and their communities, said the ASF’s VP, Security, Mark Cox in January.
Major open source software consumers would do well to consider Microsoft’s S2C2F: A consumption-focused framework that uses a threat-based, risk-reduction approach to mitigate threats across the open source software supply chain. S2C2F is applicable to all open source dependencies consumed into a developer’s workflow, e.g. source code, language packages, modules, components, containers, libraries, or binaries.
As its authors noted on its release last year: “Securing the OSS supply chain in any organization is going to be near impossible if developer teams don’t follow a uniform process for consuming OSS. Enforcing an effective secure OSS supply chain strategy necessitates standardizing your OSS consumption process across the various developer teams throughout your organization, so all developers consume OSS using governed workflows.”
See Azure CTO’s writeup on its release here and view the S2C2F requirements or download the guide.
