Activities

Good Practices

Publications

Deliverables

Subscribe to our newsletter

RISE in brief

Earthquakes are the deadliest natural hazard. Developing tools and measures to reduce future human and economic losses is the aim of RISE. RISE stands for Real-time earthquake rIsk reduction for a reSilient Europe and is a three-year project financed by the Horizon 2020 programme of the European Commission. It started in September 2019 and will end in February 2023. RISE is coordinated by ETH Zurich, it brings together 19 organisations from across Europe and five international partners.

News

2022-06-30

Ranking earthquake forecasts using proper scoring rules: binary events in a low probability environment

A research group from the University of Edinburgh and the University of Bristol (Serafini et al., 2022) has recently published a new paper dealing with ranking earthquake forecasts by using proper scoring rules. Operational earthquake forecasting for risk management and communication during seismic sequences depends on our ability to select an optimal forecasting model. To do this, the researchers need to compare the performance of competing models in prospective experiments, and to rank their performance according to the outcome using a fair, reproducible and reliable method. It is crucial that the metrics used to rank the competing forecasts are ‘proper’, meaning that, on average, they prefer the data generating model.

→ The authors prove that the Parimutuel Gambling score, proposed, and in some cases applied, as a metric for comparing probabilistic seismicity forecasts, is in general ‘improper’. In the special case where it is proper, we show it can still be used improperly.

→ Furthermore, they demonstrate the conclusions both analytically and graphically providing a set of simulation based techniques that can be used to assess if a score is proper or not.

→ The researchers compare the Parimutuel Gambling score’s performance with two commonly used proper scores (the Brier and logarithmic scores) using confidence intervals to account for the uncertainty around the observed score difference.
 
→ They also suggest that using confidence intervals enables a rigorous approach to distinguish between the predictive skills of candidate forecasts, in addition to their rankings.
 
In general, this work provides a framework to check if (and when) a score is proper, and to explore and compare the capabilities of different scores in distinguishing between competing forecasts with a focus on the penalty applied by different scores.


Ranking earthquake forecasts using proper scoring rules: binary events in a low probability environment
Serafini, F., Naylor, M., Lindgren, F., Werner, M.J, Main, I.
Geophysical Journal International, Volume 230, Issue 2, September 2022, Pages 1419–1440, https://doi.org/10.1093/gji/ggac124

Twitter

Twitter

Follow us on Twitter @research_rise!

Upcoming events

20 June 2022
Society for Earthquake and Civil Engineering Dynamics (SECED)
Webinar (free and open to everyone):
"Effectiveness & Usefulness of Earthquake Early Warning Systems"
More information

14 July 2022
International Symposium on Seismic Hazard, Risk, and Recovery Modelling
#R2R2022
Aveiro, Portugal / zoom
More information
Registration

4–9 September 2022
3rd European Conference on Earthquake Engineering and Seismology 

More information

RISE Session:
Seismology, geoethics and society: risk communication at
the service of risk reduction


25-26 May 2023
RISE Final Meeting
Lugano, Switzerland