How to Build a Prediction-Market Data Pipeline

A research data pipeline should preserve what was observable and when, including missing information. Its purpose is not merely to accumulate prices. It must let a later investigation distinguish a quote from a trade, a missing feed from a quiet market and an eventual outcome from information available earlier.

Choose the data for the question

Price-history points may be enough for some descriptive studies. They are not equivalent to candles with trade counts, a complete book or order-queue replay. Polymarket provides a price-history endpoint; the claim that all its historical prices must be collected from scratch is false. Check the requested outcome, time range, interval and returned coverage.

A study of displayed liquidity needs appropriately timed depth. A study of settlement needs rules, status and payout evidence. Do not infer a final payout solely from the last traded price. Use the current venue schema rather than assuming every contract is binary or every lifecycle message contains the complete settlement record.

Preserve identity before joining datasets

Store the venue, instrument and outcome identifiers, and the mapping used at ingestion. An event can contain multiple instruments; a condition can have multiple outcome tokens. Similar names or matching timestamps do not prove two rows describe the same instrument or executable moment.

Keep versioned rules and metadata, including changes relevant to the experiment. Store a separate observation identifier for each snapshot. A timestamp alone may collide, be absent or describe many levels. A full empty snapshot needs a header too; otherwise an empty book is indistinguishable from a collector that failed to write rows.

Record receipt and source time separately

Use timezone-aware receipt timestamps. Preserve a source timestamp when supplied and record its unit and clock meaning. Do not invent an exchange time when the source omits one. Sequence numbers belong to their documented stream or subscription scope, not a presumed global ordering across every connection.

This runnable, standalone constructor illustrates a small research header. It is not Banger's database schema or a venue response parser. The caller supplies a validated identity and whether the observation represents complete depth. It preserves a missing source timestamp, rather than replacing it with receipt time.

from datetime import datetime, timezone
from uuid import uuid4

def observation_header(venue, instrument, outcome, received_at,
                       source_at=None, full_depth=False):
    for label in (venue, instrument, outcome):
        if not isinstance(label, str) or not label.strip():
            raise ValueError("nonempty identity required")
    def utc(value):
        if not isinstance(value, datetime) or value.utcoffset() is None:
            raise ValueError("timezone-aware timestamp required")
        return value.astimezone(timezone.utc).isoformat()
    if not isinstance(full_depth, bool):
        raise ValueError("explicit depth flag required")
    return {"observation_id": str(uuid4()), "schema_version": 1,
            "venue": venue, "instrument": instrument, "outcome": outcome,
            "received_at": utc(received_at),
            "source_at": utc(source_at) if source_at is not None else None,
            "full_depth": full_depth}

# Hypothetical identifiers; this performs no network or database operation.
example = observation_header("fixture", "instrument-1", "YES",
                             datetime(2026, 10, 1, tzinfo=timezone.utc))
assert example["source_at"] is None and not example["full_depth"]

Store normalized levels under the header's observation ID, with side, outcome, documented price units and contract quantity. Validate finite values and instrument-appropriate bounds. Keep raw public evidence where permitted, with payload-size and retention limits; do not mix authorization headers or private account data into a public research dataset.

Make gaps and duplicates visible

Record disconnects, failed requests, cursor progress, schema changes and recovery boundaries. A reconnect does not prove continuity. Define deduplication from source identity where the source provides one. A newly generated local observation ID does not itself deduplicate a repeated message.

Avoid joining all rows by market and timestamp alone, which can multiply levels and trades. Reconcile row counts and coverage under explicit join keys. Keep missing intervals and unresolved outcomes in the report instead of selecting only instruments with complete successful histories.

Validate a bounded slice first

Inspect one finite collection window before building a larger service. Verify the exact response schema, pagination, outcome mapping and clock handling. Reproduce the same feature calculation from retained evidence. Separate training, decision-time information and later labels so an outcome cannot leak backward into a backtest.

Banger's public paper observations do not guarantee a full research archive or real execution replay. Record the actual adapter path and simulator assumptions. A useful pipeline supports a limited, reproducible claim about the collected slice; it does not automatically establish complete historical coverage, lossless streams or future profitable execution.