Skip to main content
1 of 2
apc
  • 111
  • 2

Adding weights to the Brier score

Fix $n > 0$, and consider the space $\cal P$ of probability functions defined over the Boolean closure of a fixed $\cal S = \{ s_1, \ldots, s_n \}$. The Brier score of $P \in \cal P$ at $s_i \in \cal S$ is given by $$BS(P,s_i) = \sum_{1 \leq j \leq n} (P(s_i) - \mathbb 1_{i = j})^2,$$ where $\mathbb 1_{i=j}$ is 1 if $i=j$ and 0 otherwise. The Brier score is strictly proper in that, for any $P \neq Q \in \cal P$,

$$\sum_i P(s_i) BS(P,s_i) < \sum_i P(s_i) BS(Q,s_i).$$

Fix $\lambda_i$, with $(1 \leq i \leq n)$, $\lambda_i > 0$. The following variant of the Brier score is also proper: $$BS_\lambda(P,s_i) = \sum_{1 \leq j \leq n} \lambda_j \cdot (P(s_j) - \mathbb 1_{i = j})^2.$$ This is essentially because $$a\cdot(1-x)^2 + (1-a)\cdot x^2$$ takes its minimum at $x=a$, and because $$\sum_i P(s_i) \cdot BS_\lambda(Q,s_i) = \sum_i \lambda_i \left(P(s_i)\cdot(1-Q(s_i))^2 + (1 - P(s_i))\cdot Q(s_i)^2\right).$$

Now, let $\cal F$ denote the Boolean closure of $\cal S$, and set $$BS_{\cal F}(P,s_i) = \sum_{X \in \cal F} (P(X) - \chi_X(s_i))^2,$$ where $\chi_X$ is the characteristic function of $X$. It is easy to see that $BS_{\cal F}$ is also strictly proper.

But again, fix $\lambda_X > 0$, $X \in \cal F$, and set $$BS^\lambda_{\cal F}(P,s_i) = \sum_{X \in \cal F} \lambda_X \cdot(P(X) - \chi_X(s_i))^2.$$

Is $BS^\lambda_{\cal F}$ proper?

If so, given $\lambda_X > 0$, $X \in \cal F$, is it possible to find $\mu_i > 0$, $1 \leq i \leq n$ such that:

$$BS^\lambda_{\cal F}(P,s_i) = BS^\mu(P,s_i).$$

apc
  • 111
  • 2