Example of a (strictly) proper scoring rule on a general measurable space?

Question

Most of the literature on scoring rules that I know of deals with discrete measurable spaces, but in this paper by Gneiting and Raferty a very general definition of a scoring rule is given. I don't see in the paper, however, any concrete examples of (strictly) proper scoring rules for general measurable spaces. I'm hoping someone can provide such an example.

Let me recall the setup (with some simplifications and minor modifications to suit my purposes). Let $(\Omega, \mathcal A)$ be a measurable space and let $\mathcal P$ be the set of probability measure on this space (Gneiting and Rafferty actually allow $\mathcal P$ to be any convex set of probability measures). A scoring rule $S: \mathcal P \times \Omega \to [-\infty, \infty)$ is a function that is measurable in its second argument. Write $$S(P,Q) = \int S(P, \omega)Q(d\omega).$$ Say that $S$ is (strictly) proper if $$S(P,P) \geq S(Q,P)$$ holds for all $P,Q \in \mathcal P$ (with equality iff $P=Q$).

A scoring rule $S$ is regular if $S(P,P) > -\infty$ for all $P \in \mathcal P$. Gneiting and Rafferty prove the following representation theorem.

A regular scoring rule is (strictly) proper if and only if there exists a (strictly) convex, real-valued function $G$ on $\mathcal P$ such that $$S(P, \omega) = G(P) - \int G^*(P, \omega')P(d\omega') + G^*(P,\omega)$$ for all $P \in \mathcal P$ and $\omega \in \Omega$, where the function $G^*(P, \cdot): \Omega \to [-\infty, \infty]$ (the subtangent of $G$ at $P$) is measurable and satisfies $$G(Q) \geq G(P) + \int G^*(P, \omega')(Q-P)(d\omega')$$ for all $Q \in \mathcal P$.

I'm looking for an example of a strictly proper scoring rule that illustrates the theorem and doesn't depend on $\Omega$ being countable. Since the strict properness fo $S$ implies that $S$ is regular, I suppose this amounts to choosing a strictly convex function $G$ that has a subtangent at every $P$. Is there an obvious or natural choice of such a function?

Answer 1

Turns out that Gneiting and Raftery give an example in Section 4.2 of the continuous ranked probability score (CRPS), which is strictly proper for $\mathcal{P}$ equal to the Borel probability measures on $\mathbb{R}$ with finite first moment. Writing a forecast as a cumulative distribution function $F$, the score is

$$ CRPS(F, x) = -\int_{-\infty}^{\infty} \Big(F(y) - \mathbf{1}[y \geq x] \Big)^2 dy . $$

It "corresponds to the integral of the Brier scores for the associated probability forecasts at all real-valued thresholds." Apparently this is a popular scoring rule in statistics.

Answer 2

Well, it might be important to limit $\mathcal{P}$ here. If we consider the space $\Omega = \mathbb{R}$ with Lebesgue measure, we might take $\mathcal{P}$ to be the set of distributions with a continuous density function (no point masses). Then I believe the log scoring rule works: $S(f, \omega) = \log f(\omega)$ where $f$ is a density function. $G$ should be the negative of differential entropy. If we in addition limit $\mathcal{P}$ to be square integrable, quadratic score should work: something like $S(f, \omega) = 2 f(\omega) - \|f\|_2^2$. Here $G(f) = \|f\|_2^2$ or something similar.

I'm much more used to the finite or countable setting - sorry I don't have more general examples! Only one paper[1] comes to my mind in this setting, and it makes the restriction above. Maybe others will know of more references.

I can mention some nontrivial weakly proper scoring rules. Let's take $\Omega$ to be the interval $[0,1]$. We know that a scoring rule for the mean of a distribution is $s: [0,1] \times [0,1] \to \mathbb{R}$ defined by $s(\mu, \omega) = -(\mu - \omega)^2$. We can lift this to a weakly proper scoring rule for distributions, $S(p, \omega) = -(\mu_p - \omega)^2$ where $\mu_p$ is the mean of $p$. Here I guess $G(p) = \mu_p^2$, ish. This scoring rule is also weakly proper on $\Omega = \mathbb{R}$ as long as $\mathcal{P}$ only contains distributions with finite mean. You can also lift scoring rules for other "properties" of distributions, e.g. obtaining $S(p,\omega) = -|m_p - \omega|$ where $m_p$ is a median of $p$.

[1] Proper Local Scoring Rules. Parry, Dawid, and Lauritzen. Annals of Statistics, 2012. https://arxiv.org/abs/1101.5011

Answer 3

Suppose $\scr F$ is a countably generated $\sigma$-algebra on any $\Omega$ (in particular, the Borel $\sigma$-algebra on any Polish space will work) and $\scr P$ is all countably additive probabilities on $\scr F$. Suppose $A_1,A_2,...$ generate $\scr F$. Let $S_n(P,\omega) = -(1_{A_n}(\omega)-P(A_n))^2$ be a Brier accuracy score with respect to $A_n$ and let $S=\sum_n 2^{-n} S_n$. Then it's elementary that $S_n$ has the following strict propriety condition: $S_n(P,P) \ge S_n(Q,P)$ with equality iff $P(A_n)=Q(A_n)$. Hence, $S$ is proper, and $S(P,P) > S(Q,P)$ if $P$ and $Q$ differ on at least one of the $A_n$. But since the $A_n$ generate $\scr F$, if $P$ and $Q$ differ anywhere, they differ on at least one of the $A_n$, so $S$ is strictly proper. Moreover, it's regular, and in fact it's uniformly bounded, and $P\mapsto S(P,\omega)$ is continuous for each fixed $\omega$ in the $\ell^\infty(\scr F)$ metric. Finding $G$ should then be an easy exercise.