Last week, I attended the White House Open Source Software Security Summit, along with VMware’s Chief Security Officer, Alex Tosheff, and Michael Kennedy, our VP of Global Government Relations and Public Policy. Led by Deputy National Security Advisor for Cyber and Emerging Technology Anne Neuberger, the summit was attended by both private industry leaders and government agencies.
This important gathering was precipitated by the Log4j vulnerability, but the real issue is — how can we ensure source code, build, and distribution integrity in all open source software (OSS)?
Building open source software and contributing to its many communities is a big part of VMware’s engineering and innovation spirit, and we believe the way forward hinges on continued collaboration. As such, we brought some practical recommendations to the table for accelerating both industry and public/private partnerships, as detailed in the following letter we submitted in advance of the summit.
January 9, 2022
Anne Neuberger
Deputy National Security Advisor for Cyber and Emerging Technology
The White House
1600 Pennsylvania Avenue, Northwest
Washington, D.C. 20500
Ms. Neuberger,
Thank you again for inviting VMware to the Open Source Software Security Summit at the White House. As you may be aware, VMware is both a heavy user of and contributor to open source software. As such, VMware takes open source, and in particular open source security, very seriously. To that end, we participate in and contribute to many open source-related industry consortia, such as the Linux Foundation, Apache Software Foundation, Cloud Native Computing Foundation and the Open Source Security Foundation (OpenSSF), to name just a few.
VMware believes that open source software has been and will continue to be an important source of innovation, industry collaboration, and industry standardization. It has enabled both new and existing businesses to rapidly build out new technologies on openly available, proven software capabilities. And because of the above-mentioned consortia, open source enables industry participants to collaborate around common standards and interoperability, which benefit all participants as well as consumers. Much of this benefit is due to the fundamental “openness” of open source.
Unfortunately, the increasing use of open source in commercial software has also meant that it is increasingly targeted by cyber-attackers. This increased threat of cyber-attacks is not unique to open source software, as evidenced by SolarWinds, Colonial Pipeline, and other recent high-profile cyber attacks. However, while companies have taken steps to address security issues in their proprietary software, the same degree of security hardening has not happened at scale across the thousands of open source projects regularly used by businesses. Security challenges with open source software stems neither from its openness nor because the projects are maintained by “hobbyists”[1]. Rather, data shows that historically the main cause of open source vulnerabilities are a lack of security education and awareness, discipline in open-source sourcing, secure build-and-release pipeline, and best practices and reference architectures for properly applying tooling. It is exactly these areas we must address collectively as an industry in order to enhance the aggregate security of open source projects.
We recognize the many security issues that we and our industry peers face with open source, and we are firmly committed to addressing them. While we as an industry strive to address open source security issues, we must ensure that we do not unintentionally eliminate what makes open source a critical enabler of innovation – its inherent accessibility and openness. We believe the common guiding principle should be to provide education and apply safeguards to dramatically improve open source security while continuing to enable innovation and collaboration. On balance, we believe open source is good for the industry and consumers, and that we should be doing more of it, but doing it the right way!
VMware recommends that the open source community and industry participants take the following actions. As the responsibility is a shared one, we recommend coordinating and implementing these recommendations across the community in two ways:
- Establish a presidentially appointed, public/private working group consisting of White House, National Security Council, and Administration representatives with senior industry participants providing a regular cadence of oversight and strategic direction. This working group could be a basis for international collaboration as open source does not begin or end at U.S. borders.
- Leverage OpenSSF to align industry participants and implement recommendations across open source projects. Many, if not most, of the industry participants on the OpenSSF governing board have been invited to the Summit. Thus, we should use this foundation as the vehicle to align on priorities and enact the actions we identify in the Summit and in the working group.
VMware is committed to continuing to improve open source security, and we look forward to addressing these issues live in our meeting on Tuesday and at the Summit on Thursday. Thank you for your leadership and the opportunity to engage on these important issues.
Thank you,
Kit Colbert, Chief Technology Officer
Alex Tosheff, Chief Security Officer
Michael Kennedy, Vice President, Global Government Relations and Public Policy
[1] The majority of open source contributors are employed by technology companies, as shown in many studies, e.g., the Linux Foundation OSS Contributor Survey: https://www.linuxfoundation.org/wp-content/uploads/2020FOSSContributorSurveyReport_121020.pdf.
Recommended Actions
Open Source Community and Projects
- Open Source Artifact Creation
While S-BOMs are important, attackers can infiltrate the software binary artifact creation process, meaning that while the specific versions of source code used may be safe, the actual binary artifacts have been compromised. As an industry, we must be able to validate and audit the entire process of conversion from open source repository source to compiled binary artifacts included in the S-BOM. VMware recommends the community implement a “defense in depth” approach that leverages multiple layers of defense to prevent build system compromise:- Artifacts: The servers used to build software binary artifacts should be ephemeral, using infrastructure-as-code techniques to dynamically instantiate and destroy those servers such that they exist only for the time needed to build the binary artifacts. This reduces the window of time an attacker will have to compromise the build server.
- Application Control: The build servers should use Application Control practices to ensure least privilege principles are adhered to. Enforcing that only expected build-related applications are able to execute dramatically reduces the risk of an attacker compromising the build process.
- EDR: The build servers should use Endpoint Detection and Response for forensics and detection of suspicious activity. EDR helps determine if or when attackers gain access to build servers so that the binary artifacts can be considered compromised and discarded.
- Micro-segmentation: Build servers often need to communicate with other servers on the network or out to the public Internet. To mitigate the risk of that connectivity, build systems should leverage micro-segmentation to prevent other compromised systems in the rest of the network from impacting the build system.
- Third Parties: For best security, organizations should build all open source software from source code and not rely on third party binaries. Taking this approach gives organizations more confidence in the security of their open source binary artifacts. Building from source code may impose an undue burden on some organizations, so we as an industry must provide repositories of open source binary artifacts where roots-of-trust can attest to the above build principles.
- Code Quality Checks and Balances
VMware supports efforts like the OpenSSF Best Practice Badge to highlight projects that focus on good code quality and other critical practices. Code quality is measured across 6 dimensions:- Change control: Publicly readable version control and version management.
- Bug and vulnerability management: Issue submission, tracking, responsiveness to issues, archival, and reporting.
- Secure coding principles training: Maintainers demonstrate an understanding of key competencies as laid out in broadly available trainings, e.g., through OpenSSF.
- Quality Controls: Aspects of quality control include an automated build system, automated regression test suite, new functionality test coverage, and warning flags and assertions.
- Security: Secure development practices, including proper use of cryptography.
- Credential management: Ensure proper handling of private credentials (prevent leak).
Businesses Incorporating OSS Into Their Products and Services
- Community Engagement Incentives
All users of open source software, particularly businesses, should be incentivized to actively engage with the corresponding communities through bug reports and code contributions. Passive “consumption” of software without engagement should be flagged as dangerous and not a best practice. VMware recommends the community adopt automated metrics to measure reviewers, testers, and contributors in proportion to S-BOM documented open source use. - Software Supply Chain Agility Incentives
We as an industry need to maintain the ability to rapidly patch systems exposed to open source vulnerabilities after vulnerability disclosure. This includes building new software artifacts and deploying those artifacts into production systems. Speed is critical to eliminate risk of attack. - Public-Private Collaboration and Threat Information Sharing
VMware recommends building on the strong momentum of public-private partnerships around cybersecurity by accelerating our coordination on open source software security activities:- Planning: Conduct joint cyber planning, including the development of crisis action plans, through public-private sector collaboration.
- Coordination: Establish common situational awareness, threat information sharing and analysis that equips public and private partners to take risk-informed, coordinated action.
- Collaboration: Ensure multi-channel, multi-source delivery of information and guidance to enable quick and comprehensive communication of cyber risk reduction actions.
VMware firmly believes that working with industry and government is key to building more innovative, interoperable, scalable and secure solutions so that innovation can continue to drive benefits for all. To participate in this dialog, visit my Strength in Numbers blog and share your thoughts with the OSS community.