# Error metric for joint estimation of mean and variance

Background:

Let $$\mu:\mathbb{R}^n\to\mathbb{R}$$ and $$\sigma:\mathbb{R}^n\to\mathbb{R}_+$$ be two unknown functions, and consider a stochastic model of the form $$\mathbb{E}[Y\mid\mathbf{x}] = \mu(\mathbf{x}),\qquad \mathbb{V}[Y\mid\mathbf{x}] = \sigma(\mathbf{x})^2.$$

We have some data consisting of samples $$(\mathbf{x}_i,y_i)$$, where $$y_i$$ is a realization of the random variable $$Y\mid\mathbf{x}_i$$.

We assume that the data is split in training and validation sets, and that the training set is used to infer some predictors $$(\hat{\mu},\hat{\sigma})$$; (NB: How the functions $$\hat{\mu}$$ and $$\hat{\sigma}$$ are computed is not the object of this question.)

Question:

I am wondering about natural error metrics that can be used to measure the performance of a predictor $$(\hat{\mu},\hat{\sigma})$$ on a validation set. More precisely, given some quantities $$m_i:=\hat{\mu}(\mathbf{x}_i)$$ and $$s_i:=\hat{\sigma}(\mathbf{x}_i)$$ computed across the validation set, what function of the $$m_i$$, $$s_i$$ and $$y_i$$'s can be used to attest the quality of the prediction ?

First ideas:

Of course, if we are only interested in pointwise estimation, then any standard error metric can be used, e.g. the Root-mean-square deviation $$\text{RMSD}=\sqrt{\sum_i (m_i-y_i)^2}$$.

However, here the problem is more complex, we want a pointwise estimate $$m_i$$ AND some uncertainty estimate $$s_i$$ as well. So there might be a trade-off between having small $$s_i$$'s and having small ratios $$\frac{|y_i-m_i|}{s_i}$$. Assuming normality of $$Y\mid\mathbf{x}$$, a possibility might be to use the negative-log likelihood $$\text{NLL} = \text{constant} + \sum_i \frac{1}{2}(\frac{m_i-y_i}{s_i})^2 + \log(s_i),$$ but this quantity is not necessarily nonnegative, and so it is difficult to interprete its value. An alternative might me to observe that the $$z_i=\frac{m_i-y_i}{s_i}$$ should be normally distributed. So we could compute their sample mean $$m_Z$$ and sample variance $$s_Z^2$$, and then use any statistical distance (e.g. a KL-divergence) between $$\mathcal{N}(m_Z,s_Z^2)$$ and $$\mathcal{N}(0,1)$$.

Any thoughts or good reference on this subject will be welcome !