How would one extend the Brier score to an infinite number of forecasts?

Question

Is there a neat way to use something like the Brier score to score an infinite set of forecasts/outcomes?

Answer 1

A general form of the Brier score, for an essentially arbitrary outcome space $\cal X$, is as follows.

Let $q(\cdot)$ be your quoted density for a random quantity $X$, with respect to a dominating measure $\mu$ over $\cal X$. Then your score, when outcome $X=x$ is realised, is $$S(x, q(\cdot)) = \int q(t)^2 d\mu(t) - 2q(x).$$

The Brier score is just one of an infinity of proper scoring rules. For some alternatives, see e.g.:

Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102, 359-378.
Dawid, A. P. (2007). The geometry of proper scoring rules. Annals of the Institute of Statistical Mathematics 59, 77-93.
doi:10.1007/s10463-006-0099-8