Synopsis: This report explores the state of security across all Apache Software Foundation projects for the calendar year 2019. We review key metrics, specific vulnerabilities, and the most common ways users of ASF projects were affected by security issues.

Released: 31 January 2020

Author: Mark Cox, Vice President Security, The Apache Software Foundation

Anyone finding security issues in any Apache project can report them to security@apache.org where they are recorded and passed on to the relevant dedicated security teams or project management committees (PMC) to handle.  The security committee see all the issues reported across all the addresses and keep track of the issues throughout the vulnerability lifecycle.  

The security committee is responsible for ensuring that issues are dealt with properly and will actively remind projects of their outstanding issues and responsibilities.  As a board committee, we have the ability to take action including blocking their future releases or, worst case, archiving a project if such projects are unresponsive to handling their security issues.  This, along with the Apache Software License, are key parts of the ASF’s general oversight function around official releases, allowing the ASF to protect individual developers and giving users confidence to deploy and rely on ASF software.  

The oversight into all security reports, along with tools we have developed, gives us the ability to easily create statistics on the issues. 

Statistics for 2019
In 2019 our security addresses received in total over 18,000 emails. After spam filtering and thread grouping this comes to 620 non-spam threads.  Unfortunately many security reports do look like spam and so the security team are careful to review all messages to ensure real reports are not missed for long.

Diagram 1: Breakdown of ASF security email threads for calendar year 2019*

Diagram 1 gives the breakdown of those 620 threads.  138 threads (22%) were people confused by the Apache License.  As many projects use the Apache License, not just those under the ASF umbrella, users can get confused when they see the Apache License and they don’t understand what it is.  This is most common for example on mobile phones where the licenses are displayed in the settings menu, usually due to the inclusion of software by Google released under the Apache License.

The next 162 of the 620 (26%) are email threads that are not spam but are also not reports of new vulnerabilities.  These are generally people asking support-type questions or how old vulnerabilities were dealt with.

That left 320 reports of new vulnerabilities in 2019, which spanned across 84 of the top level projects.  These 320 reports are a mix of both external reporters and internal; for example where a project has found an issue themselves and followed the ASF process to assign it a CVE name and address it.  Note that we don’t track the reporter affiliation, and ASF reporters often use non-ASF email addresses for reporting, so we can’t give a break down of internal vs external reports .

The next step is that the appropriate project triages the report to see if it’s really an issue or not.  At this stage invalid reports, or things that are not actually vulnerabilities at all, get rejected back to the reporter.  Of the remaining issues that are accepted they are are assigned appropriate CVE names and eventually fixes are released.

As of January 1st 2020, 19 of those 320 reports were still under triage (i.e. the project had not yet determined if the report is accepted or rejected).  The process of triage and investigation varies in time depending on the project, availability of resources, and number of issues to be assessed.  As a general guideline we try to ensure projects have triaged issues within 90 days of the report.  The timeline for the fixing of issues depends on the schedules of the projects themselves and issues of lower severity are most often held to future pre-planned releases.  

The remaining closed 301 reports led to us assigning 122 CVE names.  Some vulnerability reports may include multiple issues, some reports are across multiple projects, and some reports are duplicates where the same issue is found by different reporters, so there isn’t an exact one-to-one mapping of accepted reports to CVE names.  The Apache Security committee handle CVE name allocation and are a Mitre Candidate Naming Authority (CNA), so all requests for CVE names in any ASF project are routed through us, even if the reporter is unaware and contacts Mitre directly or goes public with an issue before contacting us. 

Noteworthy events
During 2019 there were a few events worth discussion; either because they were severe and high risk, they had readily available exploits, or otherwise due to media attention.   These included:

• January 2019: Securonix published a report outlining an increase of attacks of Apache Hadoop instances that have not been configured with authentication.  Public exploits and a Metasploit module exist to perform remote code execution on unprotected Hadoop YARN systems.
• April 2019: A flaw in Apache HTTP Server 2.4 (CVE-2019-0211).  A user who has access to write scripts on a web server could elevate those privileges to root.  A public exploit is available for this issue.
• April 2019: A flaw in older versions of Apache Axis that parsed a file retrieved insecurely from an expired domain, allowing remote code execution (CVE-2019-0227).
• June 2019: Jonathan Leitschuh contacted us after finding a number of Java build dependencies were being downloaded over insecure paths (i.e. HTTP rather than HTTPS).  We did not classify these as security vulnerabilities in themselves as exploiting them would require MITM attacks at build time.  We worked with ASF projects including those identified by the reporter to ensure that we use secure URLs.  Now, in 2020, a number of repositories are requiring secure URLs.
• August 2019: The Black Duck Synopsys team reviewed older Struts releases and advisories and found some discrepancies in the reported affected versions.   The Struts team worked through their findings and issued corrections where needed.  This can be important if users are running older versions that they don’t think are affected by an issue based on the advisories, but they actually are.  However, those same users are likely vulnerable to the other issues that have since been fixed and so we’d always recommend users upgrade to the latest version of Struts to ensure they have a version that contains fixes for all the published security issues.
• August 2019: Netflix found a number of denial of service vulnerabilities affecting various HTTP/2 implementations. ASF projects containing HTTP/2 implementations were investigated and analysed the issues reported. Both Apache HTTP Server and Apache TrafficServer released updates to address denial of service issues that affected them.  Apache Tomcat also made performance improvements to HTTP/2 handling but the issues were not classed as denial of service.
• September 2019: A RiskSense report highlighted vulnerabilities known to be used by Ransomware which included four in ASF projects.  The four vulnerabilities were all fixed in earlier years and all had updates and mitigations available before any ransomware took advantage of them.  Users should always ensure they pay attention to security updates in any ASF projects they use and prioritise updating for any remote or critical vulnerabilities. The four vulnerabilities were:

   — CVE-2016-3088 in Apache ActiveMQ.  Targeted by XBash, this issue was trivial to exploit.  It was fixed in Active MQ 5.14.0 and mitigation was also available.

   — CVE-2017-12615 in Apache Tomcat.  It is surprising to see this issue on the list as it affects a non-default and quite unlikely flaw.  However, it’s an issue that is probed by Lucky (a variant of "Satan"), so if there is a server configured in this way it will get exposed. This issue only affected Windows platforms on non-default config, it was fixed in Tomcat 7.0.81, and mitigation is also available.  Note that Lucky will also do brute force attacks targeting weak passwords on  accessible Tomcat Web Admin consoles.

   — CVE-2017-5638 in Apache Struts.  This issue is known to be exploited in the wild, however the first exploitation was discovered after the advisory and fix was published.  Used by Lucky (a variant of Satan).  It was fixed in Struts 2.3.32 and 2.5.10.1, and a mitigation is also available.

   — CVE-2018-11776 in Apache Struts.  This issue is also used by Lucky.  It was fixed in Struts 2.3.35, 2.5.17, a possible mitigation is available but upgrading is advised.

• Dec 2019: A flaw in Apache Olingo allowing XML External Entity (XXE) attacks (CVE-2019-17554).  This issue could be used, for example, to retrieve arbitrary files from a server.  A public exploit example exists for this issue.
• A number of flaws in Apache Solr through the year that could allow remote code execution.  Public exploits exist for some of the issues as well as a Metasploit module.
• The European Commission EU-FOSSA 2 project sponsored bug bounty programs for users finding security issues in both Apache Kafka and Apache Tomcat.  No issues were fixed in Apache Kafka.  Two issues were fixed in Apache Tomcat: CVE-2019-0232 (Important severity, affecting Windows platforms, public exploits including a Metasploit module are available) and CVE-2019-0221 (Low severity).   As well as running the bug bounties, EU-FOSSA 2 also sponsored a successful hackathon in June 2019.

Apache Software Foundation projects are highly diverse and independent.  They have different languages, communities, management, and security models.  However one of the things every project has in common is a consistent process for how reported security issues are handled.

The ASF Security Committee work closely with the project teams, communities, and reporters to ensure that issues get handled quickly and correctly.  This responsible oversight is a principle of The Apache Way and helps ensure Apache software is stable and can be trusted.

This report gave metrics for calendar year 2019 showing from the 18,000 emails received we triaged over 300 vulnerability reports leading to fixing just over 100 (CVE) issues.  If you have vulnerability information you would like to share with or comments on this report please contact us.