01
Definitions first
What counts as a resolved question, a stale price, or a comparable contract is written down before anything is measured. Most disagreements about accuracy are really disagreements about definitions.
Data & methods
How would we know if these prices were any good?
Prediction markets are sold on accuracy, and accuracy is the one claim almost nobody documents. The data programme exists to make that claim checkable — by fixing the definitions, the scoring rules and the thresholds before any result is published.
Current state
No dataset and no accuracy figures have been published.
The Institute has not released a dataset, a venue comparison, or a calibration result. The method below is what we are building against. Anyone quoting an Institute accuracy number today is quoting something that does not exist.
Preview — data programme, not yet populated
Every value below is a placeholder
—
Calibration error
—
Sample size
—
Venues compared
No calibration curve has been plotted. This is the shape of the chart, not a result.
Commitments
Four rules, fixed in advance.
02
Scoring rules in the open
Brier, log score and calibration error are computed with published code paths and stated bucketing. A number without its method is not a result.
03
Thresholds before results
Minimum sample sizes and publication thresholds are fixed in advance, so a flattering early sample cannot become a headline.
04
Whole-sample reporting
Where a series is reported, it is reported entire. Excluded observations are named and the reason for exclusion is published with them.
Scope
Drawing the line before it is convenient to move it.
- What we will publish
- Method notes, scoring definitions, reproducible code paths, and — once thresholds are met — measured results with their uncertainty.
- What we will not publish
- Aggregate accuracy claims below the stated sample threshold, venue league tables without a published comparison method, or any figure we cannot show the working for.
- Sources
- Publicly available market and resolution data, and data contributed under terms that permit publication. Contributed data is disclosed as such.
- Privacy
- Nothing is published at a granularity that identifies an individual participant or their positions.
Flagship products in development
Two things we intend to publish.
01
Not yet published
Orphan Risk Index
A measure of exposures that no existing instrument prices well. Design work is under way; no index level has been calculated or released.
02
Not yet published
Regulatory-outcome curves
Term structures for the resolution of pending regulatory decisions. Methodology first; no curve has been produced or distributed.
When published, Institute data products will be indicative research output only. They are not price-forming, not a quotation, not a benchmark for settlement, and not a recommendation to enter into any transaction.
Related