Why do backtest results differ between runs? A reproducibility checklist

1. Preserve the two experiments before changing either one

Assign a run identifier to each result and keep the confirmed spec, actual code and configuration. AI can generate different implementations from the same wording. A code fix may also change the strategy's meaning, so the prompt and the executed artifact should both remain available.

Record which inputs are frozen and which are fetched again at rerun time. Historical datasets may receive corrections or corporate-action updates; a request made on another date can therefore return different observations. If a fixed data snapshot is unavailable, retain the retrieval date, provider, identifiers and conventions and state the remaining reproducibility limit.

Run ID and execution timestamp: [value]
Original prompt and confirmed spec version: [value]
Executed code version or content hash: [value]
Data provider, retrieval date and snapshot/version if available: [value]
Universe definition, identifiers and historical membership rule: [value]
Date range, frequency, timezone and trading calendar: [value]
Price adjustment, missing-data and corporate-action conventions: [value]
Capital, sizing, fees, slippage and order/fill settings: [value]
Runtime and dependency versions; random seed if applicable: [value]
Outputs: signals, orders, fills, holdings, valuation and metrics
Known change from the comparison run: [value]

2. Compare the inputs in a fixed order

Begin with differences that change the meaning of the experiment. A different start date or universe can explain a result before any statistical interpretation is needed. Then compare the details that affect execution and reporting.

3. Find the first material divergence in the trade path

Align the runs by date and instrument. If input values differ, investigate data first. If inputs match but signals differ, examine the implementation and numerical conventions. If signals match but orders differ, inspect sizing, cash, existing holdings and scheduling. If orders match but fills differ, inspect execution assumptions. If positions match but equity differs, inspect valuation, costs and corporate actions.

A small early difference can grow through compounding and subsequent position sizing. That makes the final return a poor starting point for debugging. Use the earliest explainable difference to construct a small reproducible case, then rerun after changing only the suspected cause.

1. 01Compare intermediate signals and transactions rather than only headline metrics.
2. 02Use tolerances appropriate to the numeric operation, but inspect differences that alter a trade decision.
3. 03A fixed random seed is useful only when randomness exists and does not freeze data, code or dependency versions.
4. 04Separate a corrected implementation from a newly selected strategy version in the research log.

4. Keep a versioned explanation alongside FiClaw results

FiClaw keeps the research chain around a strategy spec, generated code, backtest and diagnosis. Use those materials to state what changed between iterations and why. Where a required artifact is not included in an output, capture it separately instead of assuming the experiment is fully frozen.

Close the comparison with a short statement: which difference caused the divergence, which evidence supports the explanation, whether the old run remains valid, and what still cannot be reproduced. A reproducible result makes the experiment inspectable; it does not by itself show that the strategy is robust or suitable for deployment. The external randomness and return-calculation references are general methods, not a statement about FiClaw's implementation.

Frequently asked questions