Experimental Charts

Not all the charts we dream up are perfect on the first try. Here are some experiments which don't quite meet our standards for the main chart page.

Charts Home
Home page with all charts for all platforms
Experimental Charts
Some experiments
Just Kalshi
Only Kalshi markets
Just Manifold
Only Manifold markets
Just Metaculus
Only Metaculus markets
Just Polymarket
Only Polymarket markets

Differing Predictions

Up until this point we've used the midpoint of each market as the refrence value. What if we used something else, such as the average probability or a fixed point in time?

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (Midpoint) →

n=671,732 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (Time-Average) →

n=671,732 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (24h before resolution) →

n=276,111 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (30d before resolution) →

n=82,460 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (90d before resolution) →

n=49,402 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (one year before resolution) →

n=9,058 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (24h after open) →

n=276,111 markets

Source: brier.fyi

Weighted Averages

Typically all markets are trated equally with a raw average. What if we used a weighted average instead? For instance, markets with $100 in trade volume might be weighted 10 times as heavily as one with $10. This allows us to prioritize markets that have properties we think are beneficial without ignoring others.

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (Midpoint) →

n=671,732 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution, weighted by volume0102030405060708090100Prediction (Midpoint) →

n=652,288 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution, weighted by number of traders0102030405060708090100Prediction (Midpoint) →

n=233,996 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution, weighted by duration0102030405060708090100Prediction (Midpoint) →

n=671,732 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution, weighted by recency0102030405060708090100Prediction (Midpoint) →

n=671,732 markets

Source: brier.fyi

Pick a Score

We use the Brier score by default in most plots, but we calculate many more. Take a look at how the platforms compare when using these alternative scoring methods.
Are there significant spikes or outliers in the accuracy distribution? For instance, a spike around Brier score 0.25 would indicate lots of markets that rested at 50% probability.

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
050,000100,000150,000200,000250,000300,000350,000↑ Count0.00.10.20.30.40.50.60.70.80.91.0Brier score, midpoint →
KalshiManifoldMetaculusPolymarket0.00.10.20.30.40.50.60.70.80.91.0Brier score, midpoint →

n=671,732 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
050,000100,000150,000200,000250,000300,000350,000↑ Count0.00.10.20.30.40.50.60.70.80.91.0Brier score, time-average →
KalshiManifoldMetaculusPolymarket0.00.10.20.30.40.50.60.70.80.91.0Brier score, time-average →

n=671,732 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
05,00010,00015,00020,00025,00030,00035,00040,00045,000↑ Count0.00.10.20.30.40.50.60.70.80.91.0Brier score, 30d before close →
KalshiManifoldMetaculusPolymarket0.00.10.20.30.40.50.60.70.80.91.0Brier score, 30d before close →

n=84,357 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
050100150200250300350400↑ Count−0.2−0.10.00.10.20.30.40.5Brier score, relative to others →
KalshiManifoldMetaculusPolymarket−0.2−0.10.00.10.20.30.40.5Brier score, relative to others →

n=971 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
050,000100,000150,000200,000250,000300,000350,000↑ Count−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, midpoint →
KalshiManifoldMetaculusPolymarket−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, midpoint →

n=671,619 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
050,000100,000150,000200,000250,000300,000↑ Count−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, time-average →
KalshiManifoldMetaculusPolymarket−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, time-average →

n=671,703 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
05,00010,00015,00020,00025,00030,00035,00040,000↑ Count−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, 30d before close →
KalshiManifoldMetaculusPolymarket−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, 30d before close →

n=84,333 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
050100150200250300350400↑ Count−1.4−1.2−1.0−0.8−0.6−0.4−0.20.00.20.4Logarithmic score, relative to others →
KalshiManifoldMetaculusPolymarket−1.4−1.2−1.0−0.8−0.6−0.4−0.20.00.20.4Logarithmic score, relative to others →

n=971 markets

Source: brier.fyi

Other Experiments

Calibration Plot

0102030405060708090100↑ Resolution0102030405060708090100Prediction (Midpoint) →

Calibration for all markets, aggregated across platforms. n=671,732 markets

Source: brier.fyi

Calibration Stem Plot

−120,000−100,000−80,000−60,000−40,000−20,000020,00040,000↑ Resolution0102030405060708090100Prediction (Midpoint) →

Total count resolved yes vs no for each prediction bucket. n=671,732 markets

Source: brier.fyi

Scoring Methods

BrierLogarithmicSpherical
−2.0−1.8−1.6−1.4−1.2−1.0−0.8−0.6−0.4−0.20.00.20.40.60.81.0↑ Score0102030405060708090100Error, percent →

Derivative of Scoring Methods

BrierLogarithmicSpherical
−5.0−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.00.51.01.52.0↑ Score Derivative0102030405060708090100Error, percent →

Source: brier.fyi