Nice idea! But a mall question: is the Brier Index still strictly proper? My understanding is that propriety is preserved under positive affine transformations, but not necessarily under arbitrary monotonic ones? E.g. for n = 1, the BI reduces to 100(1 − |f−o|)(absolute error), which is generally not strictly proper. I might be missing something about the aggregation though.
Thank you for spotting this technical error! Yes, a monotone transformation of a proper scoring rule remains proper for affine transformations, but the Brier Index is a nonlinear transformation. We’ve just published an addendum clarifying this issue and explaining why it doesn’t affect the ForecastBench ranking.
Nice idea! But a mall question: is the Brier Index still strictly proper? My understanding is that propriety is preserved under positive affine transformations, but not necessarily under arbitrary monotonic ones? E.g. for n = 1, the BI reduces to 100(1 − |f−o|)(absolute error), which is generally not strictly proper. I might be missing something about the aggregation though.
https://en.wikipedia.org/wiki/Scoring_rule#Affine_transformation
https://stats.stackexchange.com/questions/145875/alternative-notions-to-that-of-proper-scoring-rules-and-using-scoring-rules-to
Thank you for spotting this technical error! Yes, a monotone transformation of a proper scoring rule remains proper for affine transformations, but the Brier Index is a nonlinear transformation. We’ve just published an addendum clarifying this issue and explaining why it doesn’t affect the ForecastBench ranking.