Let $S$ be a scoring rule for probability functions. Define

$EXP_{S}(Q|P) = \sum \limits_{w} P(w)S(Q, w)$.

Say that $S$ is striclty proper if and only if $P$ always minimises $EXP_{S}(Q|P)$ as a function of $Q$. Define

$D_{S}(P, Q) = EXP_{S}(Q|P) - EXP_{S}(P|P)$.

If $S$ is the logarithmic scoring rule defined by $S(P, w) = -ln(P(w))$, then $D_{S}(P, Q)$ is just the Kullback-Leibler divergence between $P$ and $Q$, or equivalently, the inverse Kullback-Leibler divergence between $Q$ and $P$. Note that the inverse Kullback-Leibler divergence is an $f$-divergence.

My question is this: is there any other strictly proper scoring rule $S$ such that $D_{S}(P, Q)$ is equal to $F(Q, P)$ for some $f$-divergence $F$?

I think that $D_{S}(P, Q)$ is always a Bregman divergence, and Amari proved that the only $f$-divergence that is also a Bregman divergence is the Kullback-Leibler divergence (on the space of probability functions). Is this enough to imply that there are no other strictly proper scoring rules with this property?

  • $\begingroup$ Could you explain your definition of $S$ being strictly proper? What is the space of functions that $S$ is a minimiser among (and perhaps what is the precise optimization problem)? $\endgroup$
    – Steve
    Mar 19 '19 at 15:08
  • $\begingroup$ @Steve: Thanks, I just realised that there's a typo in the question, which I've now edited. $S$ being strictly proper means that $P$ always minimises $EXP_{S}(Q|P)$ as a function of $P$, i.e. $EXP_{S}(P|P) < EXP_{S}(Q, P)$ for all $Q \neq P$. $\endgroup$
    – King Kong
    Mar 19 '19 at 17:38

In a word, yes, KL is the only one. You're correct that $S$ is strictly proper if and only if $D_S$ is a Bregman divergence of some strictly convex function[1] (note you should swap the terms in your definition of $D_S$). You're also apparently right (going from the abstract) that the only f-divergence on the simplex that is a Bregman divergence is KL-divergence[2], so your conclusion follows.

One more direct way to see this is that $D_S$ is of the form $$ D_S(Q;P) = \sum_w P(w) \left[ S(P,w) - S(Q,w) \right] $$ while an $f$-divergence is of the form \begin{align} D_f(Q;P) = \sum_w P(w) \left[ f\left(\frac{Q(w)}{P(w)}\right) \right] \end{align} Non-rigorously, for these to be equal, at the very least we would need $S(P,w)$ to be only a function of $P(w)$ and not the rest of $P$, and we already know the log scoring rule is the only one that satisfies this; magically it also converts the difference in score into the log of the ratio, as is needed.

[1] e.g. Gneiting and Raftery 2007 https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf

[2] https://ieeexplore.ieee.org/document/5290302

  • $\begingroup$ I have one more question about swapping the order of the arguments in the definition of $D_{S}$. On my definition, $D_{S}(P, Q) = KL(P, Q) = IKL(Q, P)$ (where IKL is the inverse KL divergence). But if I swap the terms, then we have $D_{S}(Q, P) = KL(P, Q) = IKL(Q, P)$, which would seem to imply that IKL is a Bregman divergence, which should not be the case, right? $\endgroup$
    – King Kong
    Mar 20 '19 at 8:57
  • $\begingroup$ p.s. Gneiting and Rafferty seem to use the same order of arguments that I do: see page 361 stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf $\endgroup$
    – King Kong
    Mar 20 '19 at 9:23
  • $\begingroup$ @KingKong, apologies for not being clear - the order of arguments $D_S(P,Q)$ is good, but the right hand side is currently the negative of Bregman divergence . $EXP_S(P|P)$ is the larger term (since the scoring rule is proper). $\endgroup$
    – usul
    Mar 20 '19 at 12:51
  • $\begingroup$ Thanks! I think it's my fault for being unclear. I'm thinking of scoring rules as measures of inaccuracy, rather than measures of accuracy. So being strictly proper means that $EXP_{S}(P|P)$ is always the smaller term. So by `log score' I mean $- log(P(w))$. I'm right in thinking that this way of defining things means that $D_{S}(P, Q)$ (as I define it) is always a Bregman divergence, right? $\endgroup$
    – King Kong
    Mar 20 '19 at 13:06
  • $\begingroup$ Ah, okay, yes., but I would recommend the term "loss function" for something the reporter wants to minimize and "scoring rule" for maximize (higher score is better, lower loss is better). $\endgroup$
    – usul
    Mar 20 '19 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.