This is a bit different and doesn't address the question, but hopefully close enough to be useful: we know some things about $\mathbb{E} L(Z,Y)$ for other loss functions $L$.

If and only if $L$ is a Bregman divergence, the mean is the minimizer. This is attributed to Banerjee et al. 2005; try Mark Reid's blog post and its references.

By phrasing things differently and asking to maximize an expected score rather than minimize an expected loss (though there is no formal difference), the setting starts to look like **proper scoring rules** (e.g. Savage 1971, Gneiting and Raftery 2007). These are loss functions for eliciting distributions and they are well-understood; again, essentially just Bregman divergences.

A generalization of this question which is just receiving study recently is *property elicitation*: other loss functions and how they connect to the "property" of the distribution that solves the minimization problem. For example, if the loss is $|Y-Z|$, the minimizer is the median; if it is $\mathbf{1}_{Y=Z}$ then it is the mode, etc.

For references there I would just point to the Information Elicitation page created by Rafael Frongillo and I for this purpose, which has some slides as well. I think the paper to look at first there would be Lambert et al. 2008.

...

(1971) Leonard J. Savage. "Elicitation of personal probabilities and expectations".

(2005) Banerjee et al. "On the Optimality of Conditional Expectation as a Bregman Predictor".

(2007) Tilman Gneiting and Adrian E. Raftery. "Strictly proper scoring rules, prediction, and estimation".

(2008) Nicolas S. Lambert, David M. Pennock, and Yoav Shoham. "Eliciting properties of probability distributions".

2more comments