How CISA Mitigated a Contractor-Caused Security Incident

CISA's rapid response to a contractor's exposure of sensitive AWS GovCloud keys highlights the importance of robust cybersecurity measures and continuous improvement.

The Cybersecurity and Infrastructure Security Agency (CISA) recently faced a significant security challenge when a contractor inadvertently exposed sensitive AWS GovCloud keys in a public GitHub repository. This incident underscored the critical need for vigilant cybersecurity practices and effective incident response strategies.

The agency’s swift and comprehensive actions not only mitigated the immediate risk but also provided valuable insights for enhancing future security measures.

On May 15 CISA initiated an internal incident response after being alerted by a security researcher, through an investigative reporter, about the exposure of sensitive information.

The researcher had discovered the keys while scanning public code repositories. This timely notification allowed CISA to take immediate action to secure its cloud resources and code repositories.

Immediate Actions and Mitigation

Upon receiving the alert, CISA’s Office of the Chief Information Officer (OCIO) swiftly took steps to mitigate the exposure.

The first priority was to stop the bleeding by eliminating public exposure and preventing further harm. The reported public repository was taken offline, and a copy was saved for later analysis. This repository was not part of CISA’s official GitHub but rather a personal repository owned by a contractor.

CISA’s development environment was taken offline, and associated credentials were reset. The individual responsible for the exposure had their system access revoked. By analyzing the copy of the public repository and associated cybersecurity telemetry, CISA determined the scope of the incident. The individual had uploaded copies of a CISA build and deployment repository to their personal GitHub account, which included CISA’s Infrastructure As Code and build code. Additionally, both admin and build credentials were copied into the public repository.

Assessing the Impact

During the forensic investigation, CISA analyzed log files to assess the impact of the incident. The analysis revealed that the leaked credentials were not used outside of CISA’s environments, and no customer or mission data was exposed. This finding was crucial in understanding the extent of the breach and ensuring that no further damage had occurred.

To address the incident, CISA implemented several corrective actions. All credentials across all environments where the individual was an administrator were rotated, not just the exposed credentials. CISA also tuned the allow and deny lists for its code repositories and limited user ability to upload to public code repositories. After completing these actions, CISA’s development environment was brought back online.

Lessons Learned and Future Improvements

Following the incident, CISA conducted a thorough review to identify areas for improvement. One key lesson was the importance of taking cybersecurity tips and external reporting seriously. The collaboration with the security researcher and the investigative reporter was instrumental in mitigating the risk. CISA also recognized the effectiveness of applying Zero Trust principles to both production and development environments, emphasizing the need for strong visibility and alerting across all environments.

Enhancing logging capabilities was another critical takeaway. CISA’s Security Operations Center (SOC) had the necessary logs to investigate the incident successfully. Continuous improvement of logging capabilities remains a key element of a strong security program. CISA identified further logging opportunities during the incident response and has since implemented those additions to enhance visibility.

CISA also highlighted the need to tighten controls on public repositories. Users had the ability to upload to public repositories, which posed a risk. To mitigate this, CISA leveraged its endpoint detection and response (EDR) solution to monitor and manage uploads, enabling developers to pull code from public repositories while reducing the risk of uploading sensitive content.

Monitoring for secrets in repositories is another area that requires attention. No repository should contain secrets, yet secrets made it into CISA private repositories. CISA has since rotated all secrets and created an action plan to improve management of developer secrets and to better monitor for exposed secrets going forward.

Building comprehensive playbooks is essential for rapid response in case of an incident. CISA had missed creating a GitHub/Cloud playbook and had to spend time building one during the early stages of the incident. The agency encourages organizations to fine-tune playbooks following any response, which CISA is practicing in this case.

Simplifying incident reporting channels is crucial to ensure that incidents affecting the organization itself are handled differently from those involving its products or customers. In CISA’s case, these channels were not well defined, leading the security researcher to try multiple avenues. To reduce ambiguity, CISA is refining its reporting channels to make them easier and faster for researchers to use.

Bolstering development environment guardrails is a best practice for organizations. Consolidating developer environments ensures consistent security controls, streamlines oversight, and reduces the risk of unmanaged tooling. CISA had been advancing consolidation efforts when the incident happened, affirming the need for such guardrails for strong development environment security.

Ensuring cryptographic key readiness is an often-overlooked yet vital element of cybersecurity. The complexity of CISA’s systems and interconnections with federal and industry partners caused CISA’s key rotation to take longer than anticipated. Drawing on this experience, CISA encourages others to maintain mature and well-tested key-management capabilities.

It is not a matter of “if,” but “when” a cybersecurity incident will happen to your organization. It is important to the broader cybersecurity community that we address these matters openly to strengthen trust and foster transparency. Such transparency unlocks opportunities for learning that will enhance not only CISA’s security posture but that of other organizations as well.