RISE in brief
Earthquakes are the deadliest natural hazard. Developing tools and measures to reduce future human and economic losses is the aim of RISE. RISE stands for Real-time earthquake rIsk reduction for a reSilient Europe and is a three-year project financed by the Horizon 2020 programme of the European Commission. It started in September 2019 and will end in February 2023. RISE is coordinated by ETH Zurich, it brings together 19 organisations from across Europe and five international partners.
2022-06-30
Ranking earthquake forecasts using proper scoring rules: binary events in a low probability environment
A research group from the University of Edinburgh and the University of Bristol (Serafini et al., 2022) has recently published a new paper dealing with ranking earthquake forecasts by using proper scoring rules. Operational earthquake forecasting for risk management and communication during seismic sequences depends on our ability to select an optimal forecasting model. To do this, the researchers need to compare the performance of competing models in prospective experiments, and to rank their performance according to the outcome using a fair, reproducible and reliable method. It is crucial that the metrics used to rank the competing forecasts are ‘proper’, meaning that, on average, they prefer the data generating model.
→ The authors prove that the Parimutuel Gambling score, proposed, and in some cases applied, as a metric for comparing probabilistic seismicity forecasts, is in general ‘improper’. In the special case where it is proper, we show it can still be used improperly.
→ Furthermore, they demonstrate the conclusions both analytically and graphically providing a set of simulation based techniques that can be used to assess if a score is proper or not.
→ The researchers compare the Parimutuel Gambling score’s performance with two commonly used proper scores (the Brier and logarithmic scores) using confidence intervals to account for the uncertainty around the observed score difference.
→ They also suggest that using confidence intervals enables a rigorous approach to distinguish between the predictive skills of candidate forecasts, in addition to their rankings.
In general, this work provides a framework to check if (and when) a score is proper, and to explore and compare the capabilities of different scores in distinguishing between competing forecasts with a focus on the penalty applied by different scores.
Ranking earthquake forecasts using proper scoring rules: binary events in a low probability environment
Serafini, F., Naylor, M., Lindgren, F., Werner, M.J, Main, I.
Geophysical Journal International, Volume 230, Issue 2, September 2022, Pages 1419–1440, https://doi.org/10.1093/gji/ggac124
Upcoming events
International Symposium on Seismic Hazard, Risk, and Recovery Modelling
#R2R2022
Aveiro, Portugal / zoom
RISE Session:
Seismology, geoethics and society: risk communication at
the service of risk reduction
25-26 May 2023
RISE Final Meeting
Lugano, Switzerland