Compare Two Strategy Versions Without Cherry-Picking

A higher reported return after a change is a finding to investigate. First check whether the strategy version, data, period and execution assumptions are comparable. This guide describes a reproducible research comparison without treating a selected historical winner as proof.

Paper research evaluates simulated behavior, with no need for a trading-volume statistic or an implication that real money is being deployed. Keep the intended comparison narrow enough that a result can be traced to the declared change.

What one comparison establishes

Searching many variants and reporting only the selected winner can inflate confidence. The Deflated Sharpe Ratio literature analyzes selection and non-normal-return effects under a specified model. A ledger of trials supplies important context; it is not itself a completed statistical correction.

A result from one historical cohort cannot set a universal predictive-power bound for a new event-contract strategy. Instead, preserve the actual path, available data and selection process used in this comparison, and define which additional evidence would answer its unresolved questions.

A backtest can expose mistakes and investigate a hypothesis under stated assumptions. Improving an implementation is not automatically overfitting; repeatedly choosing a rule because it fits inspected outcomes creates a different selection question. Record why a change was made.

Hold everything constant except the one change

A fair comparison isolates the variable you are testing. Everything else must be identical between version A and version B:

• Period and universe: use the same intended timestamps and exact contracts for a controlled historical comparison.
• Starting capital and quantity convention: keep them explicit and consistent.
• Change: preserve both versions and identify the particular logic or input difference under investigation.
• Execution model: retain the same declared fee, fill, slippage and valuation assumptions unless one of those is the subject of the change.
• Data provenance: preserve dataset identity, cadence, transformations, coverage and settlement handling.

Specify the comparison before inspecting its results. If several changes must be evaluated together, record them as a batch and avoid attributing the result to just one. A later adjusted window is another comparison with its own evidence.

Keep the failures and the zero-trade runs

Keep losses, errors and zero-order outcomes with their reasons. Zero orders can be correct for an observer or an untriggered rule, but can also reflect missing data or failed initialization. A runtime error is not a completed negative financial result; distinguish the two rather than deleting either.

Preserve every attempted version and research choice, including discarded trials. Dependence between variants also matters to formal selection corrections; a count alone cannot establish significance.

A later paper run is a different sample, not a rerun

A forward run uses later observations and can reveal new operational behavior, but it is a different sample and potentially a different simulator. Record its overlap with development information and exact instrument selection. Do not use a direct historical-versus-forward metric difference as proof that the code change caused it.

If forward results disappoint, diagnose before you label. Deterioration can come from overfitting, but also from changed market conditions, data timing, or a signal that was real but temporary. A weak later result does not by itself prove overfitting, just as a strong backtest does not prove a durable edge. Investigate the cause rather than assigning an automatic verdict.

Doing this in practice

Use frozen historical inputs for the controlled comparison and preserve the actual result records. The illustration below is a manual research note, not a shipped automated comparison feature or a command that replays the same sample:

# Illustrative research records, not Banger CLI options or actual results.
comparison = {
    "dataset": "same-frozen-fixture",
    "window": "same-predeclared-window",
    "fee_and_fill_model": "same-explicit-model",
    "runs": [
        {"version": "A", "changed_input": "baseline", "result_id": "save-A"},
        {"version": "B", "changed_input": "one-declared-change", "result_id": "save-B"},
    ],
}
# Preserve failures and zero-order outcomes alongside any computed metrics.

Banger’s public beta supports versioned paper research. Sequential banger run sessions observe different live windows; those commands alone cannot hold the sample fixed. Where the product exposes supported comparison or parameter controls, save their accepted configuration and result identity. Follow the actual CLI/setup documentation for a separate bounded forward observer. Public observation requires no venue trading keys, and provider costs and capacity remain separate considerations.

The short version

• Fix period, capital, parameters, and execution assumptions before looking at results.
• Change one thing at a time so the difference is attributable.
• Keep failed and zero-trade runs, and report how many variants you tried.
• Treat a single backtest as a reason to investigate, not as proof.
• Treat a later paper run as a new sample, and diagnose deterioration instead of labeling it.

A controlled record makes a discrepancy easier to interpret. It does not guarantee that an apparent edge is real. Preserve mode-specific fees, assumed passive fills, unknown coverage and completed shutdown when carrying a candidate into a later paper experiment.