Reproducible Environments, Configuration, and Secrets

Someone checks out yesterday’s commit and reruns training. The candidate comes back different. A base image moved, a package resolved differently, the feature query now reads corrected history, and a default threshold changed in a remote configuration service.

Reproducibility means recovering the conditions that mattered when the run executed. Source code is one input among many. Some inputs must be pinned; others are meant to move.

That the same commit can produce a different system is not a thought experiment offered here for the sake of argument. The rest of this lesson is built on incidents whose own operators, vendors and regulators wrote down where the difference came from: a trading firm whose deployment reached seven of its eight order-routing servers, a security vendor whose crashing update was not code at all, and a coverage tool whose upload script exported whatever credentials passed through its customers’ pipelines.

A run inherits more than the repository contents. Five layers sit underneath it, and each one has a documented failure attached to it later in this lesson.

The application layer is the part a commit hash actually identifies: source revision, build scripts, feature logic, model code, tests. The dependency layer holds language packages, system libraries, drivers, runtime and accelerator stack. In December 2021 that layer was carrying CVE-2021-44228 into every application that had pinned Apache Log4j2 anywhere in the range 2.0-beta9 through 2.15.0. The execution layer holds the image digest, hardware class, kernel behavior, locale and resource limits. It is the layer where Knight Capital Americas ran one logical deployment on eight SMARS servers and got two different systems. The configuration layer holds hyperparameters, thresholds, endpoints, feature flags and policy versions. CrowdStrike shipped Rapid Response Content through it, outside the code release path. The protected inputs are credentials, encryption material, tokens and access grants. That layer leaked out of customers’ CI runners through Codecov’s Bash Uploader for two months in 2021.

None of those five failures was a source-code failure. Every one of them changed what the system did.

Exact numerical replay is sometimes impossible across hardware, kernels and nondeterministic operations. That is not a cautious hedge by this lesson. It is what the two dominant vendors in the accelerator stack print in their own documentation.

PyTorch’s “Reproducibility” note opens with the sentence: “Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms.” The same page adds that CPU and GPU runs need not agree even when the seeds are identical.

NVIDIA is more specific about where the nondeterminism lives. Its cuDNN Backend documentation, under “Odds and Ends”, carries a section called “Reproducibility (Determinism)”. Across different architectures, it states, no cuDNN routines guarantee bitwise reproducibility. Even on the same architecture, these do not guarantee reproducibility across runs, because they use atomic operations: cudnnConvolutionBackwardFilter when CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0 or ALGO_3 is used, cudnnConvolutionBackwardData when CUDNN_CONVOLUTION_BWD_DATA_ALGO_0 is used, cudnnPoolingBackward when CUDNN_POOLING_MAX is used, cudnnSpatialTfSamplerBackward, and cudnnCTCLoss and cudnnCTCLoss_v8 when CUDNN_CTC_LOSS_ALGO_NON_DETERMINSTIC is used. A team chasing bit-identical training output on that stack is chasing something its own suppliers decline to promise.

Operational reproducibility asks a different question. Can the team reconstruct the assets, environment, parameters and evidence well enough to explain and defend the result? The required strength depends on consequence. Someone exploring an idea may tolerate numbers that differ slightly. A regulated release may require preserved artifacts, immutable identifiers and a documented equivalence test.

The build world settled on the strict version of the idea and wrote it down. The Reproducible Builds project’s definitions page states: “A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts”, and reproducibility is verified by bit-by-bit comparison of those artifacts. The GNU Mes Reference Manual, an independent host, reproduces that sentence word for word. That is the top rung of the ladder. PyTorch’s opening disclaimer is why most training runs stand several rungs below it, and the useful skill is knowing which rung you are claiming.

On 1 August 2012 Knight Capital Americas lost over $460 million in roughly 45 minutes. The repository was not at fault. Neither was the new code.

A staged deployment had begun on 27 July 2012. The SEC found what went wrong with it: “During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers.”

Seven servers ran the new RLP code. The eighth read the same repurposed flag and activated dormant “Power Peg” code, a path Knight had stopped using in 2003 and had never retested after moving its cumulative-quantity function in 2005. The SEC’s settled order of 16 October 2013 sets out the sequence and imposed a $12,000,000 civil money penalty. The same event, its date and its software cause were described independently of the regulator by Carol Clark, in a Chicago Fed Letter for the Federal Reserve Bank of Chicago.

Every element a reproducibility manifest exists to record is visible in that story: which artifact reached which machine, what a reused identifier now meant, and what was still executable on a host that no one had rebuilt from a clean state in nine years.

Lifecycle and sensitivity determine how a value should be managed, and the three classes fail in different ways.

Build-time configuration defines the artifact and should travel with its identity: dependency versions, compile flags. A change requires a new artifact. The values live in provenance, and rollback means selecting an earlier immutable build — an earlier digest, in the sense the Open Container Initiative gives the word, not an earlier tag.

Release policy is mutable by design: thresholds, feature flags, routes. It needs its own versioning, approval and audit, because a change here may not require retraining and a rollback here need not touch the model. The danger is treating a separate lifecycle as a lighter one. On 19 July 2024 a CrowdStrike Rapid Response Content update crashed Windows sensors. The new IPC Template Type defined 21 input parameter fields; the integration code invoking the Content Interpreter supplied only 20. Reading the 21st value produced an out-of-bounds memory read. The vendor’s own root cause analysis of 6 August 2024, “External Technical Root Cause Analysis — Channel File 291”, states the design decision underneath: “Rapid Response Content is configuration data; it is not code or a kernel driver.” That sentence is the whole decision and the whole failure at once. The content shipped outside the code release path, Microsoft estimated that 8.5 million Windows devices were affected, and the Cybersecurity and Infrastructure Security Agency issued its alert, “Widespread IT Outage Due to CrowdStrike Update”, the same day. Configuration that can halt a kernel needs a release process shaped like the code one, whatever it is called.

Secrets are protected values that must rotate without appearing in artifacts or logs: credentials, signing keys, delivered through a secure control plane under least privilege, with rotation that does not alter the recorded artifact identity. The rule is broken constantly, and two organisations counted the breakage independently for the same year. GitHub reported that more than 39 million secrets were leaked across GitHub in 2024. GitGuardian’s “The State of Secrets Sprawl 2025” opens with a narrower scope: “23.8 million secrets leaked on public GitHub repositories in 2024, marking a 25% year-over-year increase”. The two numbers are not a contradiction. One is a platform operator scanning everything it hosts; the other is an outside vendor counting public repositories. The same report found that 70% of secrets leaked in 2022 were still active. A leaked secret does not expire on its own. Only rotation ends it.

Capture the inputs needed to reconstruct and explain a run. This is not the author’s housekeeping preference. A federal standard already requires most of it.

NIST Special Publication 800-218, the Secure Software Development Framework version 1.1 of February 2022, names practice PS.3 “Archive and Protect Each Software Release”. Task PS.3.1 requires that the necessary files and supporting data for each software release — integrity verification information and provenance data, for example — be securely archived. Task PS.3.2 reads: “PS.3.2: Collect, safeguard, maintain, and share provenance data for all components of each software release (e.g., in a software bill of materials [SBOM]).” Those tasks are not NIST’s own invention. They trace to Executive Order 14028 of 12 May 2021, “Improving the Nation’s Cybersecurity”, whose section 4(e) directs that the guidance include criteria for “(vi) maintaining accurate and up-to-date data, provenance (i.e., origin) of software code or components” and “(vii) providing a purchaser a Software Bill of Materials (SBOM) for each product directly or by publishing it on a public website”.

1. Identify immutable artifacts. Record source, data snapshot, model, image and dependency digests. That is PS.3.1’s integrity verification information and PS.3.2’s provenance data, expressed as identifiers that cannot be repointed the way a tag can.

2. Record runtime conditions. Capture hardware class, driver stack, resource limits and determinism settings. The cuDNN routine list in the previous section is the reason. Without the architecture and the algorithm selection on record, a numerical difference between two runs cannot be classified as a bug or as documented nondeterminism.

3. Version mutable policy. Store thresholds, routes, flags and schemas separately, with effective times. Channel File 291 is the argument for effective times specifically: the question after an incident is which content version a given host had at a given minute, and only a separately versioned policy record answers it.

4. Reference secrets safely. Record secret identifiers and access policy, never secret values — and assume the identifier will outlive the value. Codecov’s security update of 15 April 2021 discloses that from 31 January 2021 its Bash Uploader script was periodically altered without authorisation, so that material passing through customers’ CI runners was exported to a third-party server. What could be taken was “Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the bash uploader script was executed.” The alteration ran for two months. It was found only after a customer checked the published shasum on the morning of 1 April 2021. HashiCorp then confirmed, on 22 April 2021, that the GPG private key signing its SHA256SUM release files had been exposed by this route. It revoked keypair 91A6 E7F8 5D05 C656 30BE F189 5185 2D87 348F FC4C and published replacement keypair C874 011F 0AB4 0511 0D02 1055 3436 5D94 72D7 468F. Note what collided there. The rotated secret was a release-signing key, so a secrecy event forced a change in how every archived artifact’s integrity is verified. A manifest that stores key identifiers and the signature, rather than the key, survives that rotation. One that entangles the two does not.

5. Test reconstruction. Rebuild in a clean environment and compare outputs within declared tolerances. An archive nobody has restored is a claim, not evidence.

A manifest is valuable only if a clean system can use it. Rebuilding from a clean system now and then exposes vanished packages, inaccessible artifacts, undocumented defaults and secret dependencies before an incident — and, as the Codecov disclosure shows, before an attacker exposes them instead.

The result should state its tolerance, and the three available tolerances are now concrete rather than rhetorical. Byte-identical means what the Reproducible Builds project means: bit-by-bit comparison of all specified artifacts. Numerically equivalent means within a stated margin, because PyTorch will not promise identity across releases, commits or platforms, and NVIDIA will not promise it across architectures. Behaviorally equivalent is what remains when a release has to be rebuilt on a patched substrate, and it is the claim NIST SP 800-218 practice PS.3 supplies the archive and the provenance data to support.

A team that has not named its tolerance has not made a claim anyone can test.