A research group from the University of Edinburgh and the University of Bristol (Serafini et al., 2022) has recently published a new paper dealing with ranking earthquake forecasts by using proper scoring rules. Operational earthquake forecasting for risk management and communication during seismic sequences depends on our ability to select an optimal forecasting model. To do this, the researchers need to compare the performance of competing models in prospective experiments, and to rank their performance according to the outcome using a fair, reproducible and reliable method. It is crucial that the metrics used to rank the competing forecasts are ‘proper’, meaning that, on average, they prefer the data generating model.
→ The authors prove that the Parimutuel Gambling score, proposed, and in some cases applied, as a metric for comparing probabilistic seismicity forecasts, is in general ‘improper’. In the special case where it is proper, we show it can still be used improperly.
→ Furthermore, they demonstrate the conclusions both analytically and graphically providing a set of simulation based techniques that can be used to assess if a score is proper or not.
→ The researchers compare the Parimutuel Gambling score’s performance with two commonly used proper scores (the Brier and logarithmic scores) using confidence intervals to account for the uncertainty around the observed score difference.
→ They also suggest that using confidence intervals enables a rigorous approach to distinguish between the predictive skills of candidate forecasts, in addition to their rankings.
In general, this work provides a framework to check if (and when) a score is proper, and to explore and compare the capabilities of different scores in distinguishing between competing forecasts with a focus on the penalty applied by different scores.
Ranking earthquake forecasts using proper scoring rules: binary events in a low probability environment
Serafini, F., Naylor, M., Lindgren, F., Werner, M.J, Main, I.
Geophysical Journal International, Volume 230, Issue 2, September 2022, Pages 1419–1440, https://doi.org/10.1093/gji/ggac124