Background:
Let $\mu:\mathbb{R}^n\to\mathbb{R}$ and $\sigma:\mathbb{R}^n\to\mathbb{R}_+$ be two unknown functions, and consider a stochastic model of the form $$ \mathbb{E}[Y\mid\mathbf{x}] = \mu(\mathbf{x}),\qquad \mathbb{V}[Y\mid\mathbf{x}] = \sigma(\mathbf{x})^2. $$
We have some data consisting of samples $(\mathbf{x}_i,y_i)$, where $y_i$ is a realization of the random variable $Y\mid\mathbf{x}_i$.
We assume that the data is split in training and validation sets, and that the training set is used to infer some predictors $(\hat{\mu},\hat{\sigma})$; (NB: How the functions $\hat{\mu}$ and $\hat{\sigma}$ are computed is not the object of this question.)
Question:
I am wondering about natural error metrics that can be used to measure the performance of a predictor $(\hat{\mu},\hat{\sigma})$ on a validation set. More precisely, given some quantities $m_i:=\hat{\mu}(\mathbf{x}_i)$ and $s_i:=\hat{\sigma}(\mathbf{x}_i)$ computed across the validation set, what function of the $m_i$, $s_i$ and $y_i$'s can be used to attest the quality of the prediction ?
First ideas:
Of course, if we are only interested in pointwise estimation, then any standard error metric can be used, e.g. the Root-mean-square deviation $\text{RMSD}=\sqrt{\sum_i (m_i-y_i)^2}$.
However, here the problem is more complex, we want a pointwise estimate $m_i$ AND some uncertainty estimate $s_i$ as well. So there might be a trade-off between having small $s_i$'s and having small ratios $\frac{|y_i-m_i|}{s_i}$. Assuming normality of $Y\mid\mathbf{x}$, a possibility might be to use the negative-log likelihood $\text{NLL} = \text{constant} + \sum_i \frac{1}{2}(\frac{m_i-y_i}{s_i})^2 + \log(s_i),$ but this quantity is not necessarily nonnegative, and so it is difficult to interprete its value. An alternative might me to observe that the $z_i=\frac{m_i-y_i}{s_i}$ should be normally distributed. So we could compute their sample mean $m_Z$ and sample variance $s_Z^2$, and then use any statistical distance (e.g. a KL-divergence) between $\mathcal{N}(m_Z,s_Z^2)$ and $\mathcal{N}(0,1)$.
Any thoughts or good reference on this subject will be welcome !
