Experimental Charts

Not all the charts we dream up are perfect on the first try. Here are some experiments which don't quite meet our standards for the main chart page.

Charts Home
Home page with all charts for all platforms
Experimental Charts
Some experiments
Just Kalshi
Only Kalshi markets
Just Manifold
Only Manifold markets
Just Metaculus
Only Metaculus markets
Just Polymarket
Only Polymarket markets

Differing Predictions

Up until this point we've used the midpoint of each market as the refrence value. What if we used something else, such as the average probability or a fixed point in time?

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (Midpoint) →

n=443,535 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (Time-Average) →

n=443,535 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (24h before resolution) →

n=200,303 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (30d before resolution) →

n=75,595 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (90d before resolution) →

n=45,671 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (one year before resolution) →

n=8,275 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (24h after open) →

n=200,303 markets

Source: brier.fyi

Weighted Averages

Typically all markets are trated equally with a raw average. What if we used a weighted average instead? For instance, markets with $100 in trade volume might be weighted 10 times as heavily as one with $10. This allows us to prioritize markets that have properties we think are beneficial without ignoring others.

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution0102030405060708090100Prediction (Midpoint) →

n=443,535 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution, weighted by volume0102030405060708090100Prediction (Midpoint) →

n=431,894 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution, weighted by number of traders0102030405060708090100Prediction (Midpoint) →

n=173,728 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution, weighted by duration0102030405060708090100Prediction (Midpoint) →

n=443,535 markets

Source: brier.fyi

Calibration Plot

KalshiManifoldMetaculusPolymarket
0102030405060708090100↑ Resolution, weighted by recency0102030405060708090100Prediction (Midpoint) →

n=443,535 markets

Source: brier.fyi

Pick a Score

We use the Brier score by default in most plots, but we calculate many more. Take a look at how the platforms compare when using these alternative scoring methods.
Are there significant spikes or outliers in the accuracy distribution? For instance, a spike around Brier score 0.25 would indicate lots of markets that rested at 50% probability.

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
020,00040,00060,00080,000100,000120,000140,000160,000180,000200,000220,000240,000260,000↑ Count0.00.10.20.30.40.50.60.70.80.91.0Brier score, midpoint →
KalshiManifoldMetaculusPolymarket0.00.10.20.30.40.50.60.70.80.91.0Brier score, midpoint →

n=443,535 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
020,00040,00060,00080,000100,000120,000140,000160,000180,000200,000220,000240,000260,000↑ Count0.00.10.20.30.40.50.60.70.80.91.0Brier score, time-average →
KalshiManifoldMetaculusPolymarket0.00.10.20.30.40.50.60.70.80.91.0Brier score, time-average →

n=443,535 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
05,00010,00015,00020,00025,00030,00035,00040,000↑ Count0.00.10.20.30.40.50.60.70.80.91.0Brier score, 30d before close →
KalshiManifoldMetaculusPolymarket0.00.10.20.30.40.50.60.70.80.91.0Brier score, 30d before close →

n=76,941 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
050100150200250300350400↑ Count−0.2−0.10.00.10.20.30.40.5Brier score, relative to others →
KalshiManifoldMetaculusPolymarket−0.2−0.10.00.10.20.30.40.5Brier score, relative to others →

n=942 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
020,00040,00060,00080,000100,000120,000140,000160,000180,000200,000220,000240,000↑ Count−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, midpoint →
KalshiManifoldMetaculusPolymarket−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, midpoint →

n=443,430 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
020,00040,00060,00080,000100,000120,000140,000160,000180,000200,000220,000↑ Count−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, time-average →
KalshiManifoldMetaculusPolymarket−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, time-average →

n=443,508 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
05,00010,00015,00020,00025,00030,00035,000↑ Count−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, 30d before close →
KalshiManifoldMetaculusPolymarket−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.0Logarithmic score, 30d before close →

n=76,921 markets

Source: brier.fyi

Distribution of Market Accuracy Scores

KalshiManifoldMetaculusPolymarket
050100150200250300350400↑ Count−1.4−1.2−1.0−0.8−0.6−0.4−0.20.00.20.4Logarithmic score, relative to others →
KalshiManifoldMetaculusPolymarket−1.4−1.2−1.0−0.8−0.6−0.4−0.20.00.20.4Logarithmic score, relative to others →

n=942 markets

Source: brier.fyi

Other Experiments

Calibration Plot

0102030405060708090100↑ Resolution0102030405060708090100Prediction (30d before resolution) →

Calibration for all markets, aggregated across platforms. n=443,535 markets

Source: brier.fyi

Calibration Stem Plot

−10,000−8,000−6,000−4,000−2,00002,000↑ Resolution0102030405060708090100Prediction (30d before resolution) →

Total count resolved yes vs no for each prediction bucket. n=443,535 markets

Source: brier.fyi

Scoring Methods

BrierLogarithmicSpherical
−2.0−1.8−1.6−1.4−1.2−1.0−0.8−0.6−0.4−0.20.00.20.40.60.81.0↑ Score0102030405060708090100Error, percent →

Derivative of Scoring Methods

BrierLogarithmicSpherical
−5.0−4.5−4.0−3.5−3.0−2.5−2.0−1.5−1.0−0.50.00.51.01.52.0↑ Score Derivative0102030405060708090100Error, percent →

Source: brier.fyi