Exploring new statistical metrics to evaluate the magnitude distribution of earthquake forecasting models

Clicks: 10
ID: 331650
2026
Free PDF
Article Quality & Performance Metrics
Overall Quality
Not rated
Combines reader engagement with the AI quality analysis. This article has not been analysed, so there is no overall score — reader engagement is measured and shown alongside.
AI Quality Assessment
Not analyzed
Readership in this journal
Emerging

Ranked #320 of 337 articles by views in geophysical journal international

Most read Least read

Bar heights use a square-root scale. Only the 120 most-read articles are drawn; the journal has 337 in total.

Mint this article as an NFT
Not yet minted

Create a permanent, verifiable on-chain record of this article on the Scimatic Network. The NFT is held in your Journament account, and you can withdraw it to your own wallet at any time.

5 SUSD one-off · no wallet required
Sign in to mint
Abstract
Summary Evaluating earthquake forecasts is a crucial step in understanding and improving the capabilities of forecasting models. The use of specific metrics to assess the consistency between forecasts and data on one particular aspect of the process is important to understand which aspects of seismicity a model is failing to describe, and, consequently, highlight where new versions of the model should improve. This can be done effectively only by metrics unaffected by inconsistencies in other aspects of the process. The Collaboratory for the Study of Earthquake Predictability (CSEP), which organises earthquake forecasting experiments around the globe, has developed different tests targeting different aspects of the process, such as the number N-test or the magnitude M-test, assessing (in)consistency between the observed and forecasted number and magnitude distributions of events, respectively. We find that the results of the recently proposed M-test for catalog-based forecasts (composed of a collection of synthetic catalogs from the model) depend on the N-test, i.e. the two tests do not isolate the desired aspects appropriately, rendering uninformative forecast results. Here, we address this problem using simulated data and provide a solution based on resampling of simulated catalogs. We implement this new M-test with resampling in the pyCSEP software toolkit, conduct two analyses using actual earthquake forecasts for Europe (1990-2015) and Switzerland (1933-1962, 1962-1992, 1993-2022) that provide inconsistent earthquake counts compared to observations, and analyse how the test results change using this proposed test. Lastly, we investigate alternative metrics, namely an unnormalised M-test, two Chi-square formulations, the Hellinger distance, the Brier score, and a novel Multinomial Log-Likelihood (MLL) score, and compare them based on the ability to find (in)consistency between data and forecast in various synthetic scenarios. We find that there are scenarios in which the alternative metrics outperform the resampled M-test. In particular, the MLL test outperforms the M-test in all the scenarios considered. We also study how the ability of finding inconsistency changes with the number of observations, and the cutoff magnitude, comparing one of the Chi-square metrics, the MLL, and the M-test against classical statistical methods such as the Kolmogorov-Smirnoff (KS) test, the Wilcoxon test, and the Anderson-Darling (AD) test. The MLL and the AD test are the ones providing the highest probability of finding inconsistencies for all combinations of number of observations and cutoff magnitude. This study shows how realistic synthetic examples can be used to compare the ability of different metrics in finding inconsistencies between data and forecasts, and shows that the MLL is the metric providing the best tradeoff between interpretability and probability of finding inconsistencies and, therefore, should be used.
Reference Key
openalex_W7220846874 Use this key to autocite in the manuscript while using SciMatic Manuscript Manager or Thesis Manager
Authors Francesco Serafini, L Mizrahi, M Han, K Bayliss, P Iturrieta, José A. Bayona, Mark Naylor, Maximilian Jonas Werner
Are you one of these authors? Claim your profile
Journal geophysical journal international
Year 2026
DOI
10.1093/gji/ggag419
URL
https://doi.org/10.1093/gji/ggag419
Keywords Keywords not found

Citations

No citations found. To add a citation, contact the admin at info@scimatic.org

Comments

No comments yet. Be the first to comment on this article.