Arithmetic tests are not empirical validation

Different claims require different evidence

An arithmetic test asks whether a calculation implements a specified formula on specified inputs. An empirical validation asks whether a model or measurement is useful for an intended application under relevant conditions. NASA's systems-engineering distinction between verification against requirements and validation for intended use provides a useful analogy, not a financial certification framework. A volatility routine can return exactly the correct sample standard deviation while its input history is unrepresentative of future conditions. A regression routine can fit exactly the requested observations while those observations contain future information. Computational correctness is necessary for trustworthy analysis but does not settle these separate empirical questions.

Fixtures establish a narrow numerical contract

A deterministic fixture fixes inputs and an independently derived expected result. It can detect the wrong divisor, omitted compounding, sign errors or accidental use of percentages where decimals were required. NIST's statistical reference datasets supply certified numerical values for testing regression implementations, while warning that success on the collection does not guarantee correctness on every dataset. That is a boundary on the inference from tests, not a dismissal of testing. An expected answer generated by the same faulty calculation as the implementation provides weak independence. A second derivation, exact small example or established reference value can expose shared mistakes.

A worked fixture with no performance claim

The original synthetic hypothetical contains decimal returns 0.01, -0.01, 0.02 and -0.02. Their arithmetic mean is zero and centered sum of squares is 0.001000. Sample variance is 0.000333333333 and sample standard deviation is 0.018257418584. Compounding gives 1.01 times 0.99 times 1.02 times 0.98 minus one, or -0.00049996. The table lists these expected values and the claims they can support. The negative compound result despite a zero arithmetic mean is an algebraic consequence of the constructed inputs. These rows are not trades, independent strategy-validation observations or evidence that any investment approach passes an acceptance gate.

Properties supplement individual answers

Some checks concern relationships between calculations rather than one expected output. Reordering returns leaves their sample variance and compounded terminal wealth unchanged, but can change drawdown. Multiplying all returns by a constant multiplies their standard deviation by its absolute value. Multiplying both regression variables by the same positive unit factor leaves the slope unchanged. Such properties can reveal inconsistent implementations across many synthetic inputs. They still inherit the specification's scope: a property valid for a simple return vector need not describe a cash-flow account or a path-dependent trading rule. More property checks increase software coverage, not the number of independent market experiments.

Data meaning survives outside the formula

A formula may be correct while its inputs violate the intended information contract. Examples include misaligned benchmark dates, duplicate rows, survivorship-filtered assets, retrospectively revised predictors and costs attached to the wrong period. Scikit-learn's leakage guidance illustrates how preprocessing fitted on test data contaminates evaluation despite otherwise valid numerical operations. Integration checks examine these boundaries between data, transformations and calculations. A unit test of the standard-deviation function alone cannot demonstrate that real observations were aligned correctly. Likewise, a successful parser does not establish that a return column includes dividends or that a recorded price was available at the claimed decision time.

Undefined inputs and numerical tolerance

The mathematical contract also has boundaries: sample variance needs enough observations, beta needs nonzero benchmark variance and an information ratio needs nonzero active-return dispersion. Nonfinite values and ambiguous missing-data treatment can make an apparent answer meaningless. Floating-point comparisons introduce a different issue. An absolute tolerance near zero and a relative tolerance away from zero address numerical approximation, but neither is an economic materiality threshold. A numerically close result can still be conceptually wrong if it uses a different annualization convention. These are examples of requirements to specify, not claims that a particular calculator already implements every rejection rule or tolerance test.

Empirical evidence begins beyond fixture agreement

Evidence about an investment model additionally concerns data provenance, causal timing, baseline choice, untouched evaluation periods, dependence, costs and the number of alternatives considered. The result may remain uncertain even when every software test passes. Conversely, disappointing empirical performance does not prove an arithmetic bug. A clear record reports calculation checks separately from empirical findings, including unperformed analyses and unresolved assumptions. The synthetic fixture here supports reproducibility of a few defined calculations only. Repeating it many times cannot transform it into a larger statistical sample, and passing it supplies no license to label a model validated, profitable or suitable for deployment.

Primary references

These sources support methods and conventions. The example values are original constructions, not observations extracted from these documents.

• NIST Statistical Reference Datasets: Linear Regression background and limitations
• NASA Systems Engineering Handbook: Product Realization, verification and validation
• Scikit-learn documentation: Common pitfalls and data leakage

Related investigations