mythsthe platformsCodecov / Bash Uploader2021developer tooling / ci pipelines

everyone assumed:

a secret in our CI is low risk

what was actually true:

A secret in your pipeline is a live credential to something real, and the pipeline is not a vault.

how it happened

  1. whoattackercredential from a Docker image
  2. the gapBash Uploaderran in thousands of pipelines
  3. thenCI secretsAWS keys, tokens, passwords
  4. the result~2 months unseenfound by a hash mismatch
the gap that let it happen the result

what everyone assumed

Secrets live in CI because that is where builds need them: the API key to deploy, the token to publish, the credential to reach the database. It feels like a controlled place. The pipeline is ephemeral, the environment tears down when the job ends, and the secrets are injected just for the run. Storing them there is how continuous integration is supposed to work. The exposure feels bounded to the seconds a job is alive.

what happened

Beginning around January 31, 2021, an attacker who had extracted a credential from a Codecov Docker image began periodically modifying the Codecov Bash Uploader script, a tool many teams ran inside their CI to send coverage reports. The modified script used a simple command to send the pipeline’s environment variables, along with git information, to a server the attacker controlled. Every environment variable in the job went with it: AWS keys, deploy keys, API tokens, service-account passwords. The altered script served correctly, produced correct results, and ran across thousands of pipelines for roughly two months while nothing looked wrong.

step by step

  1. a credential is extracted from a Docker imageIt allows changes to the Bash Uploader script.
  2. 31 January 2021the uploader is quietly modifiedIt now sends every environment variable in the pipeline to the attacker.the assumption breaks here
  3. it runs in thousands of pipelines for two monthsResults stay correct, so nothing looks wrong.
  4. 1 April 2021a customer's hash check does not matchCodecov discloses on 15 April; secrets across more than 23,000 customers need rotating.

how it surfaced

On April 1, 2021, a customer noticed that the hash of the Bash Uploader on Codecov’s site did not match the one published on GitHub, and reported it. Codecov investigated and disclosed the compromise on April 15, notifying affected users. The uploader had reached more than 23,000 customers, and any secret that had passed through those pipelines during the window had to be treated as exposed and rotated. The assumption retired itself in the rotation: every credential a team thought was safely scoped to a build turned out to be a credential that had left the building. What is stored in CI does not stay in CI when CI itself is the thing that was tampered with.

A secret in your pipeline is a live credential to something real, and the pipeline is not a vault.

It is a program that runs untrusted-adjacent code from the internet, and any of that code can read what you put in the environment. The blast radius of a CI secret is not the build; it is everything that secret unlocks.

picture it at a company like yours

The company here is made up. The other two paths are not: they happened, and each links to the story.

your company
a normal dayYour developers run a small, popular helper script every time they build the product.
the gap nobody seesThe script can see every password and key used in the build, and nobody checks it.trusting what a supplier ships
  1. in this story
    the day it mattersThe script is quietly changed at the source to copy those secrets out.
    what it costs youEvery key your build touched for months has to be treated as stolen and replaced.
  2. or it goes like this
    howA genuine, signed update from a certified vendor carries a backdoor.
    what leavesa quiet way into your networkwho ends up with ita state intelligence service
    it happenedSolarWinds / Sunburst, 2020
  3. or it goes like this
    howA build helper everyone uses is hijacked and its versions repointed.
    what leavessecret keys, printed into build logswho ends up with itanyone reading public logs
    it happenedtj-actions/changed-files, 2025

sources

  1. Codecov, "Bash Uploader Security Update"
  2. Codecov, "Post-Mortem / Root Cause Analysis (April 2021)"
  3. Rapid7, "Analysis of the Codecov Supply Chain Compromise"
  4. CISA, "Codecov Releases New Detections for Supply Chain Compromise"

← all myths

your turn

what you can do about it

  1. this week

    List the credentials your CI holds and mark the ones that reach production data or cloud infrastructure.

    CI secrets feel low-risk because they are out of sight, tucked in a settings panel you configured once. The quiet-months question is direct: if every secret in your pipeline leaked today, what would each one open, and how fast could you rotate it. The teams that recovered from Codecov in hours were the ones who already knew that list and could rotate without guessing.

  2. go deeper

  3. if you're not sure where to start

    Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.

    open the checklist →

the letter · every two weeks

one access gap in your inbox, with the fix.

Every issue takes one way access creeps in, shows a real incident where it went wrong, and ends with one thing to check that week. Written for the people who run the tools, without a security team behind them.

  • 1one mechanism, explained plainly
  • 2one real incident, taken apart
  • 3one step you can take on Monday

get the next issue

One email every two weeks. Your address stays here and is never passed to vendors. One click to unsubscribe. Privacy.