How to Build a Prediction-Market Data Pipeline

If you want to research strategies on prediction markets, you need your own data. The venues give you a live view of the book, but the history you need for backtests is something you have to capture and store yourself. This post walks through the three decisions that matter most: what to pull, how often to sample it, and how to store it so resolution events do not silently corrupt your research.

Know what each venue actually exposes

The two dominant venues have very different plumbing, and your pipeline has to account for both.

Polymarket splits its API into three services. The Gamma API handles market metadata and discovery, the CLOB API is the central limit order book for trading and book data, and the Data API serves user-specific positions and trade history.

The CLOB order-book endpoint returns a snapshot keyed by token ID, with bids and asks as price and size pairs, plus market details like tick size and last trade price. There is a batch endpoint for pulling multiple books in one request, which matters when you are tracking many markets at once.

Kalshi exposes REST, WebSocket, and FIX. For a data pipeline the WebSocket feed is the one you want, because it streams updates without constant polling and carries public market-data channels like ticker and trade. Kalshi also publishes an asyncapi.yaml schema for the WebSocket integration and an openapi.yaml for REST, so you can generate typed clients instead of hand-rolling them.

One gotcha worth flagging up front: the free Polymarket public API returns live market state only. There is no historical endpoint. If you did not capture a snapshot at time T, that state is gone. This is the single biggest reason to stand up your own recorder early rather than assuming you can backfill later.

Pick a sampling frequency you can defend

Two collection modes exist, and most serious pipelines run both:

• Event-driven capture: subscribe to the WebSocket, write every trade and book delta as it arrives. This preserves microstructure and lets you reconstruct the book at any instant.
• Interval snapshots: poll the full book on a fixed cadence (say every 1, 10, or 60 seconds) and store the whole ladder. Simpler, lossy, but far cheaper to store and query.

For context on what is enough, commercial Polymarket history providers settle on 1-minute resolution and argue that for most strategy backtesting, that cadence captures the dynamics that matter. That is a reasonable default for slower markets like politics or macro. For fast-moving markets (live sports, crypto price levels) 1-minute will smear the moves you care about, so record deltas from the WebSocket instead.

Capture the full ladder, not just top-of-book. If you only store best bid and best ask, you cannot estimate slippage or fill quality at realistic size later. Storing every resting level lets you reconstruct fill curves for any position size; top-of-book only tells you the mid.

Schema: store snapshots as append-only, immutable rows

Model the book as time series, not mutable state. Every row is a snapshot or delta stamped with a source timestamp and an ingestion timestamp. Never update in place. A clean starting schema:

-- one row per (market, side, level) per snapshot
CREATE TABLE book_snapshot (
  venue        TEXT NOT NULL,        -- 'polymarket' | 'kalshi'
  market_id    TEXT NOT NULL,        -- CLOB token id / Kalshi ticker
  ts_exchange  TIMESTAMPTZ NOT NULL, -- venue-reported time
  ts_ingest    TIMESTAMPTZ NOT NULL, -- when your recorder wrote it
  side         TEXT NOT NULL,        -- 'bid' | 'ask'
  level        INT  NOT NULL,        -- 0 = best
  price        NUMERIC NOT NULL,     -- 0.00 - 1.00
  size         NUMERIC NOT NULL,
  tick_size    NUMERIC,
  PRIMARY KEY (venue, market_id, ts_exchange, side, level)
);

-- trades captured from the stream
CREATE TABLE trade (
  venue      TEXT NOT NULL,
  market_id  TEXT NOT NULL,
  ts_exchange TIMESTAMPTZ NOT NULL,
  price      NUMERIC NOT NULL,
  size       NUMERIC NOT NULL,
  aggressor  TEXT
);

Keep two timestamps. The venue clock tells you when the book actually changed; your ingest clock exposes recorder lag and gaps. When a backtest looks too good, the gap between those two columns is usually where the bug lives. For columnar storage on disk, partition Parquet by venue and date and use the market slug plus timestamp as a natural join key. Book and trade data live in separate tables but share those keys, so joining them is a straightforward merge on market and timestamp.

The gotcha that ruins backtests: resolution events

A prediction-market contract is not like an equity. It terminates at a known value. On resolution, holders of winning shares receive $1 per share, losing shares become worthless, and trading stops. Your pipeline has to record that terminal event as a first-class fact, because a naive last-price fill will otherwise book fantasy PnL on a contract that actually paid out at 0 or 1.

The two venues resolve very differently, and both differences bite.

Kalshi is a CFTC-regulated Designated Contract Market whose event contracts settle in US dollars. Resolution is administrative and comes from the exchange, so for a data pipeline you mostly need to capture the settlement value and settlement time from the market lifecycle channel.

Polymarket resolves through the UMA Optimistic Oracle. As of November 2025, Polymarket upgraded to the Managed Optimistic Oracle V2 (MOOv2), which restricts the right to propose resolutions to a set of pre-approved addresses, while keeping the dispute process open to anyone. An undisputed proposal resolves after a 2-hour challenge window. If someone disputes, the case escalates to UMA's Data Verification Mechanism (DVM): a 24-48 hour debate period is followed by a tokenholder vote that takes roughly 48 hours, putting total disputed resolution at approximately 3 to 5 days.

That timing matters for your data model. There is a window between a market's real-world end and its on-chain finalization where the book may still show prices, trading may still be possible, and the eventual payout is not yet certain. If you treat the last observed price as the settlement value, you will misprice every disputed market. Record the proposal time, the resolution time, and the final outcome as separate fields, and reconcile fills only against the finalized outcome.

• Store an explicit market status (open, closed, proposed, disputed, resolved) alongside every snapshot.
• Record the resolution outcome and resolution timestamp as their own columns, not derived from last price.
• Flag any trades that occurred after the real-world end but before finalization so you can exclude or stress-test them.
• Read each market's resolution rules, not just its title. The title is the question; the rules define how it actually settles, including edge cases.

From pipeline to research to paper trading

Once you have clean, resolution-aware history, the research loop is: reconstruct the book at each decision point, simulate fills against the recorded ladder, and mark positions to the finalized outcome rather than the last tick. Keep the ingest-lag column visible throughout so you never accidentally use data your strategy could not have seen live.

When a strategy survives that, the next step is to run it against the live book before risking capital. Banger (an open-source strategy-automation runtime for Polymarket and Kalshi) is built for exactly this handoff: you write a Python strategy, paper-trade it against the live order book with `banger run strategy.py --paper`, and only then run it live under a declarative risk envelope (per-trade cap, daily loss stop, max open positions, kill switch). Banger never custodies funds; you bring your own venue keys. Install with `pip install bangertrades`. The data pipeline you build here is what feeds the research; the runtime is what enforces discipline when you go live.

Regulatory and venue status in this space changes month to month, so treat any legal detail above as accurate only as of July 2026 and re-verify before you act on it.