Reproducible model artifacts: preserve the calculation, not just its answer

Choose the reproduction claim

Reproduction can mean obtaining identical file bytes, matching numerical values within a tolerance, or independently confirming a mathematical result. These are related but different claims. The Reproducible Builds definition concerns identical artifacts produced from the same source, environment and instructions. A numerical reference may instead accept tiny floating-point differences across environments. State which comparison is intended before running it. A screenshot with the same rounded answer proves neither byte identity nor agreement at the calculation's working precision. Conversely, different timestamps can change file hashes even when every numerical result agrees, so the artifact boundary needs to be explicit.

Preserve a minimal complete bundle

A useful bundle contains the model identifier and version, exact input values, input contract, source revision, dependency description, invocation and expected output. Add the assertion rules and the actual comparison result. For a synthetic calculation, include the small fixture itself so readers do not need a vendor account to reproduce arithmetic. Distinguish an expected answer derived separately from an output merely copied from the implementation under test. If an optional external dataset is involved, record its identity and access requirements without implying that it is included. A bundle that omits a required private input is a partial reproduction package.

Pin dependencies beyond the top-level package

A package name alone does not identify executable behavior. Record exact versions and relevant transitive dependencies, together with interpreter and platform details when they affect the calculation. pip's hash-checking mode illustrates a stricter installation approach: dependencies must be pinned and their distributions covered by hashes. The principle is useful beyond Python, but a dependency hash verifies an artifact's identity against a recorded expectation, not its correctness or trustworthiness. Separate installation evidence from numerical tests. A successful environment setup says that the specified components were obtained; it does not say that the model used the right formula or input order.

Do not treat a seed as the whole experiment

When random inputs are used, preserve the generator family, seed, method calls and array shapes, not just an integer labeled seed. NumPy's compatibility policy describes strict conditions for reproducing a random stream, including the build and environment. Different linear algebra libraries can also affect derived samples. For a compact educational fixture, publishing the generated numbers is often simpler than requiring every reader to recreate the sampling process. Keep the generation recipe as provenance, while using the frozen fixture for arithmetic checks. A deterministic model evaluated on fixed inputs should not acquire unnecessary randomness merely to make its tests appear realistic.

Declare comparison tolerances in output units

The hypothetical fixture compounds returns of 0.10, -0.10 and 0.05. Multiplying the wealth factors 1.10, 0.90 and 1.05 gives 1.0395, so cumulative return is 0.0395, or 3.95%. The table compares invented candidate outputs against that independently calculated decimal return using only an absolute tolerance of 0.000000001. A difference of 0.0000000002 passes; a difference of 0.0000002 fails. A value of 3.95 is not a close answer in the wrong formatting style: it is a scale error when the output contract requires a decimal return. Specify any relative tolerance separately rather than accepting a library default unknowingly.

Keep identity checks separate from value checks

Hash the exact bytes intended for distribution, and define any canonical serialization used for comparison. Changing whitespace, key order or line endings can change a text file's digest without changing its parsed values. That does not justify ignoring a mismatch; it identifies which comparison needs investigation. Keep volatile run metadata outside a stable calculation payload when exact-byte reproduction is a requirement. Never replace an unexpected digest with the newly observed one merely to obtain a passing check. First determine whether the source, fixture, serialization or dependency changed, and preserve the discrepancy as part of the review evidence.

Report reproduction limits honestly

A reproduction note should say what was rerun, under which environment, and which assertions passed or failed. If only the arithmetic was checked by hand, say that instead of describing an executed software test. If a second implementation shares the same helper routine, it is not fully independent of that routine's mistakes. Include boundary cases such as zero returns and a total-loss return where the model's domain allows them. Reproducible arithmetic is valuable evidence about a calculation, but it is not an empirical investment result. These artifact practices describe a verifiable workflow, not a claim of unattended research or an operating service.

Primary references

These sources support methods and conventions. The example values are original constructions, not observations extracted from these documents.

• Reproducible Builds: definitions and artifact comparison
• pip: secure installs and dependency hash checking
• NumPy: random stream compatibility policy

Related investigations