22,600+ emails = 599 vulnerabilities. Security disclosure triage is HARD

22,600+ emails = 599 vulnerabilities. Security disclosure triage is HARD

Security researchers regularly chafe at the deafening silence when they report a critical vulnerability in software: White Hats simply wanting to help organisations fix their cybersecurity all too often still find themselves being ignored – or worse, threatened with legal action when trying to help publicly exposed organisations that have not provided a clear route to reporting bugs; whether with a security@ email address or a bounty programme.

Yet the latest annual security report from the Apache Software Foundation (ASF), which is home to over 350 high-profile open source software projects, paints the flipside of that picture. Vulnerability reports that are emailed in are handled by volunteers, it notes, with the organisation in 2022 hiring a sole administrator to help.

This team received over 22,600 emails to its security email address in 2022 – yet after spam filtering and thread grouping there were 1402 non-spam threads. Each had to be reviewed however to avoid missing real issues.

As the ASF notes: “Unfortunately security reports do sometimes look like spam, especially if they include lots of attachments or large videos, and so we are careful to review all messages to ensure real reports are not missed.”

ASF security report: 22,600+ emails = 599 actual new vulnerabilities

Actually excluding spam, many were “the unfiltered output of some publicly available scanning tool, and often come along with a request for some sort of monetary reward” leaving 599 reports of new vulnerabilities in 2022, which spanned 122 of the top-level projects. These 599 reports include both external reports, as well as issues found internally by projects and their communities, said the ASF’s VP, Security, Mark Cox on January 31.

The four projects with the most reports in 2022 were Airflow with 49 reports (and 19 CVE released in 2022), Commons with 37 reports (4 CVE), HTTP Server with 36 reports (13 CVE), and Tomcat with 26 reports (6 CVE).

Airflow and HTTP Server are part of the HackerOne Internet Bug Bounty program,Cox noted.

22k emails; 599 vulns: ASF security report highlights triage challenge
Breakdown of ASF security email threads for calendar year 2022 (after removal of ‘spam’ messages). Credit: ASF

The report comes amid heightened focus on open source software security, with the US’s Cybersecurity and Infrastructure Security Agency (CISA) is building out a new supply chain risk management office (C-SCRM) that is being led by Shon Lyublanovits. A rise in malicious open source packages in particular has left many security professionals. Security firm Sonatype for example detected close to zero in 2019,  1,200 in 2020, 12,000 in 2021, and now 88,000 in 2022 – suggesting awareness has grown rapidly among criminal groups of the potential attack vector. (A well-known demonstration of its efficacy was a campaign by security researcher Alex Birsan that saw him hit over 35 major companies including Apple, Google and Microsoft via this kind of “substitution attack” which exploits a design flaw in the way native installation tools and DevOps workflows pull dependencies into a software supply chain; as well as the way many developers freely consume snippets of code for their builds.)

See also: Global CISOs, White House, agree 10-point open source security plan