Recognize Selection Bias in a Contract Universe

If you backtest a prediction-market strategy against the contracts you can see today, you are not testing against the market as it was. You are testing against a filtered version of history, and the filter quietly removes exactly the cases that would have hurt you. This is selection bias, and it is the difference between a backtest that looks like edge and a result that holds up.

What selection bias does to a sample

The classic version in trading is survivorship bias. The definition is simple enough to be dangerous: survivorship bias is the error of concentrating on the assets that survived a selection process while overlooking those that no longer exist.

The mechanism matters more than the label. When you build a universe from only the names that are still around, you strip the dataset of its worst outcomes. In equities, a backtest that only includes currently active stocks implicitly selects for winners, which inflates average returns and understates drawdowns. One commonly cited range is that excluding delisted or failed securities can overstate annual returns by roughly 1 to 4 percent.

Prediction markets have their own version of the same trap, and it shows up through more channels than equities do. The universe you can query right now is not the universe that existed during your test period. Three distinct filters are at work.

Three filters that reshape a contract universe

• Open filter. Listing endpoints and dashboards tend to surface active, tradeable markets. If you only pull what is open today, you have already excluded everything that already closed and resolved.
• Liquidity filter. Screening for tight spreads or high reported volume is reasonable for execution, but if you apply it to historical selection you drop the thin, messy, or abandoned markets that are part of the real opportunity set.
• Outcome filter. The most seductive one. Keeping only the contracts where your rule would have worked, or eyeballing a chart and testing the markets you remember resolving cleanly, is picking the winners after the fact.

Prediction markets add a structural wrinkle that equities do not have as sharply. Contracts resolve and leave. On Kalshi, trading stops at a market's close, Kalshi verifies the outcome against the data source named in that market's own rules, and the market settles, usually within a few hours of the outcome being confirmed. A resolved contract is a finished object, not a live one. If your universe is built from live listings, every one of those settled markets is missing by construction.

Why the resolved ones are the information

Binary event contracts carry their answer in the outcome. Kalshi's standard binary contracts settle at $1.00 if the event occurs and $0.00 if it does not, so net profit on a position is $1.00 minus entry price minus fees. (Do not assume this generalizes to every scalar, bracketed, or exceptional contract, which can settle differently.) The point for sampling is that the resolved set is where you learn whether your entry prices were actually mispriced. Drop it and you are left with markets that are, almost by definition, still unresolved and still ambiguous.

Outcome resolution on Kalshi is also not always a clean mechanical read. Settlement is governed by the specific contract terms, which name an official data source and a resolution date before trading opens, and disputed cases have turned on the exact wording of those terms rather than the headline question. A sample that only keeps the tidy resolutions hides precisely the cases where the fine print, not your model, decided the payout.

Preserve the original selection rule

The fix is not a magic dataset. It is discipline about the rule that defined your universe in the first place.

• Write down the selection rule before you look at outcomes. 'All KXBTCD markets in this date range' is a rule. 'The BTC markets I remember going well' is not.
• Apply the rule point-in-time. Decide membership using only information available on each historical date, not today's list of what is still open.
• Keep the dead ones. Resolved, delisted, voided, and thin markets stay in the sample if the rule admitted them. Their absence is the bias.
• Preserve unknowns as unknowns. If a history is unavailable or a resolution rule is ambiguous, record that gap instead of silently dropping the row or imputing a convenient value.
• Report the rejects. Note how many candidates the rule produced and how many you actually had data for. A large gap between those two numbers is itself a finding.

Avoid the reflex to set a universal minimum sample size or to treat one favorable backtest as proof of edge. A clean result on a filtered universe is not evidence of anything except the filter. What you want is a sample whose boundaries you can defend.

Where a paper runtime fits

Banger (bangertrades.com) is a paper research beta for prediction-market strategies. You write a strategy as a Python class or clone one, and it runs against current market data without venue trading keys. Hosted historical backtesting currently supports Kalshi binary markets, which are the $1/$0 contracts described above. That scope is useful to know when you design a universe: a Kalshi-binary backtest is a defined, inspectable model, not the whole prediction-market world.

Two habits keep selection honest inside that kind of tool. First, pin an exact market ticker for a test and preserve it, rather than trusting whatever the first discovery result happens to return, so the run is reproducible. Second, read the result model rather than trusting the summary. A paper fill is evidence under a particular data, fill, fee, and valuation model, not proof that the same trade would have filled at a real venue. The public forward-paper context, for instance, can fill a non-crossing resting limit at that limit and log the assumption it made, and its quantity cap is based on reported volume rather than full order-book depth. Knowing what the model assumed is part of knowing what your sample means.

The short version

A contract universe is a sampling decision disguised as a data-loading step. If you select on being open, liquid, or correct in hindsight, you build a sample that cannot lose the way the real market did. Define the rule first, apply it point-in-time, keep the resolved and the dead and the ambiguous, and treat any single good result as one data model's output rather than a verdict. The boring part, preserving what you were tempted to throw away, is where the honesty lives.