Let $\mu:\mathbb{R}^n\to\mathbb{R}$ and $\sigma:\mathbb{R}^n\to\mathbb{R}_+$ be two unknown functions, and consider a stochastic model of the form $$ \mathbb{E}[Y\mid\mathbf{x}] = \mu(\mathbf{x}),\qquad \mathbb{V}[Y\mid\mathbf{x}] = \sigma(\mathbf{x})^2. $$

We have some data consisting of samples $(\mathbf{x}_i,y_i)$, where $y_i$ is a realization of the random variable $Y\mid\mathbf{x}_i$.

We assume that the data is split in training and validation sets, and that the training set is used to infer some predictors $(\hat{\mu},\hat{\sigma})$; (NB: How the functions $\hat{\mu}$ and $\hat{\sigma}$ are computed is not the object of this question.)


I am wondering about natural error metrics that can be used to measure the performance of a predictor $(\hat{\mu},\hat{\sigma})$ on a validation set. More precisely, given some quantities $m_i:=\hat{\mu}(\mathbf{x}_i)$ and $s_i:=\hat{\sigma}(\mathbf{x}_i)$ computed across the validation set, what function of the $m_i$, $s_i$ and $y_i$'s can be used to attest the quality of the prediction ?

First ideas:

Of course, if we are only interested in pointwise estimation, then any standard error metric can be used, e.g. the Root-mean-square deviation $\text{RMSD}=\sqrt{\sum_i (m_i-y_i)^2}$.

However, here the problem is more complex, we want a pointwise estimate $m_i$ AND some uncertainty estimate $s_i$ as well. So there might be a trade-off between having small $s_i$'s and having small ratios $\frac{|y_i-m_i|}{s_i}$. Assuming normality of $Y\mid\mathbf{x}$, a possibility might be to use the negative-log likelihood $\text{NLL} = \text{constant} + \sum_i \frac{1}{2}(\frac{m_i-y_i}{s_i})^2 + \log(s_i),$ but this quantity is not necessarily nonnegative, and so it is difficult to interprete its value. An alternative might me to observe that the $z_i=\frac{m_i-y_i}{s_i}$ should be normally distributed. So we could compute their sample mean $m_Z$ and sample variance $s_Z^2$, and then use any statistical distance (e.g. a KL-divergence) between $\mathcal{N}(m_Z,s_Z^2)$ and $\mathcal{N}(0,1)$.

Any thoughts or good reference on this subject will be welcome !


1 Answer 1


It turns out that I found a good reference to answer my question. A better keyword would have been "loss function for prediction intervals". In this paper, the distribution-free case is discussed: https://arxiv.org/abs/1802.07167


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.