**Background**:

Let $\mu:\mathbb{R}^n\to\mathbb{R}$ and $\sigma:\mathbb{R}^n\to\mathbb{R}_+$ be two unknown functions, and consider a stochastic model of the form $$ \mathbb{E}[Y|\mathbf{x}] = \mu(\mathbf{x}),\qquad \mathbb{V}[Y|\mathbf{x}] = \sigma(\mathbf{x})^2. $$

We have some data consisting of samples $(\mathbf{x}_i,y_i)$, where $y_i$ is a realization of the random variable $Y|\mathbf{x}_i$.

We assume that the data is split in training and validation sets, and that the training set is used to infer some predictors $(\hat{\mu},\hat{\sigma})$; *(NB: How the functions $\hat{\mu}$ and $\hat{\sigma}$ are computed is not the object of this question.)*

**Question**:

I am wondering about natural error metrics that can be used to measure the performance of a predictor $(\hat{\mu},\hat{\sigma})$ on a validation set. More precisely, given some quantities $m_i:=\hat{\mu}(\mathbf{x}_i)$ and $s_i:=\hat{\sigma}(\mathbf{x}_i)$ computed across the validation set, what function of the $m_i$, $s_i$ and $y_i$'s can be used to attest the quality of the prediction ?

**First ideas**:

Of course, if we are only interested in pointwise estimation, then any standard error metric can be used, e.g. the Root-mean-square deviation $RMSD=\sqrt{\sum_i (m_i-y_i)^2}$.

However, here the problem is more complex, we want a pointwise estimate $m_i$ **AND** some uncertainty estimate $s_i$ as well. So there might be a trade-off between having small $s_i$'s and having small ratios $\frac{|y_i-m_i|}{s_i}$. Assuming normality of $Y|\mathbf{x}$, a possibility might be to use the negative-log likelihood
$NLL = constant + \sum_i \frac{1}{2}(\frac{m_i-y_i}{s_i})^2 + \log(s_i)$,
but this quantity is not necessarily nonnegative, and so it is difficult to interprete its value.
An alternative might me to observe that the $z_i=\frac{m_i-y_i}{s_i}$ should be normally distributed. So we could compute their sample mean $m_Z$ and sample variance $s_Z^2$, and then use any statistical distance (e.g. a KL-divergence) between $\mathcal{N}(m_Z,s_Z^2)$ and $\mathcal{N}(0,1)$.

Any thoughts or good reference on this subject will be welcome !