3
$\begingroup$

Let $S$ be a scoring rule for probability functions. Define

$EXP_{S}(Q|P) = \sum \limits_{w} P(w)S(Q, w)$.

Say that $S$ is striclty proper if and only if $P$ always minimises $EXP_{S}(Q|P)$ as a function of $Q$. Define

$D_{S}(P, Q) = EXP_{S}(Q|P) - EXP_{S}(P|P)$.

If $S$ is the logarithmic scoring rule defined by $S(P, w) = -ln(P(w))$, then $D_{S}(P, Q)$ is just the Kullback-Leibler divergence between $P$ and $Q$, or equivalently, the inverse Kullback-Leibler divergence between $Q$ and $P$. Note that the inverse Kullback-Leibler divergence is an $f$-divergence.

My question is this: is there any other strictly proper scoring rule $S$ such that $D_{S}(P, Q)$ is equal to $F(Q, P)$ for some $f$-divergence $F$?

I think that $D_{S}(P, Q)$ is always a Bregman divergence, and Amari proved that the only $f$-divergence that is also a Bregman divergence is the Kullback-Leibler divergence (on the space of probability functions). Is this enough to imply that there are no other strictly proper scoring rules with this property?

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you explain your definition of $S$ being strictly proper? What is the space of functions that $S$ is a minimiser among (and perhaps what is the precise optimization problem)? $\endgroup$
    – Steve
    Commented Mar 19, 2019 at 15:08
  • $\begingroup$ @Steve: Thanks, I just realised that there's a typo in the question, which I've now edited. $S$ being strictly proper means that $P$ always minimises $EXP_{S}(Q|P)$ as a function of $P$, i.e. $EXP_{S}(P|P) < EXP_{S}(Q, P)$ for all $Q \neq P$. $\endgroup$
    – King Kong
    Commented Mar 19, 2019 at 17:38

1 Answer 1

Reset to default
1
$\begingroup$

In a word, yes, KL is the only one. You're correct that $S$ is strictly proper if and only if $D_S$ is a Bregman divergence of some strictly convex function[1] (note you should swap the terms in your definition of $D_S$). You're also apparently right (going from the abstract) that the only f-divergence on the simplex that is a Bregman divergence is KL-divergence[2], so your conclusion follows.

One more direct way to see this is that $D_S$ is of the form $$ D_S(Q;P) = \sum_w P(w) \left[ S(P,w) - S(Q,w) \right] $$ while an $f$-divergence is of the form \begin{align} D_f(Q;P) = \sum_w P(w) \left[ f\left(\frac{Q(w)}{P(w)}\right) \right] \end{align} Non-rigorously, for these to be equal, at the very least we would need $S(P,w)$ to be only a function of $P(w)$ and not the rest of $P$, and we already know the log scoring rule is the only one that satisfies this; magically it also converts the difference in score into the log of the ratio, as is needed.

[1] e.g. Gneiting and Raftery 2007 https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf

[2] https://ieeexplore.ieee.org/document/5290302

Improve this answer
$\endgroup$
5
  • $\begingroup$ I have one more question about swapping the order of the arguments in the definition of $D_{S}$. On my definition, $D_{S}(P, Q) = KL(P, Q) = IKL(Q, P)$ (where IKL is the inverse KL divergence). But if I swap the terms, then we have $D_{S}(Q, P) = KL(P, Q) = IKL(Q, P)$, which would seem to imply that IKL is a Bregman divergence, which should not be the case, right? $\endgroup$
    – King Kong
    Commented Mar 20, 2019 at 8:57
  • $\begingroup$ p.s. Gneiting and Rafferty seem to use the same order of arguments that I do: see page 361 stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf $\endgroup$
    – King Kong
    Commented Mar 20, 2019 at 9:23
  • $\begingroup$ @KingKong, apologies for not being clear - the order of arguments $D_S(P,Q)$ is good, but the right hand side is currently the negative of Bregman divergence . $EXP_S(P|P)$ is the larger term (since the scoring rule is proper). $\endgroup$
    – usul
    Commented Mar 20, 2019 at 12:51
  • $\begingroup$ Thanks! I think it's my fault for being unclear. I'm thinking of scoring rules as measures of inaccuracy, rather than measures of accuracy. So being strictly proper means that $EXP_{S}(P|P)$ is always the smaller term. So by `log score' I mean $- log(P(w))$. I'm right in thinking that this way of defining things means that $D_{S}(P, Q)$ (as I define it) is always a Bregman divergence, right? $\endgroup$
    – King Kong
    Commented Mar 20, 2019 at 13:06
  • $\begingroup$ Ah, okay, yes., but I would recommend the term "loss function" for something the reporter wants to minimize and "scoring rule" for maximize (higher score is better, lower loss is better). $\endgroup$
    – usul
    Commented Mar 20, 2019 at 15:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .