# Strictly Proper Scoring Rules and f-Divergences

Let $$S$$ be a scoring rule for probability functions. Define

$$EXP_{S}(Q|P) = \sum \limits_{w} P(w)S(Q, w)$$.

Say that $$S$$ is striclty proper if and only if $$P$$ always minimises $$EXP_{S}(Q|P)$$ as a function of $$Q$$. Define

$$D_{S}(P, Q) = EXP_{S}(Q|P) - EXP_{S}(P|P)$$.

If $$S$$ is the logarithmic scoring rule defined by $$S(P, w) = -ln(P(w))$$, then $$D_{S}(P, Q)$$ is just the Kullback-Leibler divergence between $$P$$ and $$Q$$, or equivalently, the inverse Kullback-Leibler divergence between $$Q$$ and $$P$$. Note that the inverse Kullback-Leibler divergence is an $$f$$-divergence.

My question is this: is there any other strictly proper scoring rule $$S$$ such that $$D_{S}(P, Q)$$ is equal to $$F(Q, P)$$ for some $$f$$-divergence $$F$$?

I think that $$D_{S}(P, Q)$$ is always a Bregman divergence, and Amari proved that the only $$f$$-divergence that is also a Bregman divergence is the Kullback-Leibler divergence (on the space of probability functions). Is this enough to imply that there are no other strictly proper scoring rules with this property?

• Could you explain your definition of $S$ being strictly proper? What is the space of functions that $S$ is a minimiser among (and perhaps what is the precise optimization problem)? Mar 19 '19 at 15:08
• @Steve: Thanks, I just realised that there's a typo in the question, which I've now edited. $S$ being strictly proper means that $P$ always minimises $EXP_{S}(Q|P)$ as a function of $P$, i.e. $EXP_{S}(P|P) < EXP_{S}(Q, P)$ for all $Q \neq P$. Mar 19 '19 at 17:38

In a word, yes, KL is the only one. You're correct that $$S$$ is strictly proper if and only if $$D_S$$ is a Bregman divergence of some strictly convex function[1] (note you should swap the terms in your definition of $$D_S$$). You're also apparently right (going from the abstract) that the only f-divergence on the simplex that is a Bregman divergence is KL-divergence[2], so your conclusion follows.

One more direct way to see this is that $$D_S$$ is of the form $$D_S(Q;P) = \sum_w P(w) \left[ S(P,w) - S(Q,w) \right]$$ while an $$f$$-divergence is of the form \begin{align} D_f(Q;P) = \sum_w P(w) \left[ f\left(\frac{Q(w)}{P(w)}\right) \right] \end{align} Non-rigorously, for these to be equal, at the very least we would need $$S(P,w)$$ to be only a function of $$P(w)$$ and not the rest of $$P$$, and we already know the log scoring rule is the only one that satisfies this; magically it also converts the difference in score into the log of the ratio, as is needed.

[1] e.g. Gneiting and Raftery 2007 https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf

• I have one more question about swapping the order of the arguments in the definition of $D_{S}$. On my definition, $D_{S}(P, Q) = KL(P, Q) = IKL(Q, P)$ (where IKL is the inverse KL divergence). But if I swap the terms, then we have $D_{S}(Q, P) = KL(P, Q) = IKL(Q, P)$, which would seem to imply that IKL is a Bregman divergence, which should not be the case, right? Mar 20 '19 at 8:57
• p.s. Gneiting and Rafferty seem to use the same order of arguments that I do: see page 361 stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf Mar 20 '19 at 9:23
• @KingKong, apologies for not being clear - the order of arguments $D_S(P,Q)$ is good, but the right hand side is currently the negative of Bregman divergence . $EXP_S(P|P)$ is the larger term (since the scoring rule is proper).
– usul
Mar 20 '19 at 12:51
• Thanks! I think it's my fault for being unclear. I'm thinking of scoring rules as measures of inaccuracy, rather than measures of accuracy. So being strictly proper means that $EXP_{S}(P|P)$ is always the smaller term. So by `log score' I mean $- log(P(w))$. I'm right in thinking that this way of defining things means that $D_{S}(P, Q)$ (as I define it) is always a Bregman divergence, right? Mar 20 '19 at 13:06
• Ah, okay, yes., but I would recommend the term "loss function" for something the reporter wants to minimize and "scoring rule" for maximize (higher score is better, lower loss is better).
– usul
Mar 20 '19 at 15:01