6
$\begingroup$

In my recent work I've become interested in working with the minimizer of $$ \mathbb{E}[(Y-Z)^2] + \lambda P(Z), $$ $Y$ is an observed random variable, $P$ is a positive-convex penalty function, $Z$ is a measurable random variable with respect to the $\sigma$-algebra generated by $Y$ and $\lambda\geq 0$.

If $\lambda=0$ this is the MSE and is minimized by the conditional expectation. My question is, is there a well developed theory of "penalized conditional expectation"?
That is a theory studying the above equation's minimizer? So far I have found nothing really. All help is greatly appreciated.

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ I would not expect as elegant results in the penalized case: the original minimization problem (with $\lambda = 0$) is quadratic, optimizer is linear and can be nicely characterized as a projection of $Z$ onto $\sigma(Y)$. I'd expect you can get some local deviation results for $\lambda \ll 1$, however not sure whether you'd find them useful. $\endgroup$
    – SBF
    Commented Sep 28, 2016 at 8:55
  • $\begingroup$ What do you mean by local deviation? $\endgroup$
    – ABIM
    Commented Sep 29, 2016 at 2:06
  • 1
    $\begingroup$ I mean something of the kind "for $\lambda$ small enough, the optimal solution is $F = \Bbb E[Z|Y] + ...$" $\endgroup$
    – SBF
    Commented Sep 29, 2016 at 11:33
  • 1
    $\begingroup$ This is a bit different, but we know a fair amount about $\mathbb{E} L(Z,Y)$ for other loss functions $L$. If $L$ is any Bregman divergence, then the solution is still the conditional expectation. For example of other cases, if $L(Z,Y) = |Z-Y|$ then the solution is the median, and so on. If this approach interests you I can try to give references. $\endgroup$
    – usul
    Commented Oct 1, 2016 at 0:21
  • 1
    $\begingroup$ You may already know this, but "penalized likelihood methods" or "penalized empirical risk minimization" in statistics are based on this type of equation. Keywords would include "ridge regression" or Tikhonov regularization as well as the lasso estimator. There are huge bodies of literature on these topics. $\endgroup$
    – R Hahn
    Commented Oct 1, 2016 at 6:00

1 Answer 1

Reset to default
3
$\begingroup$

This is a bit different and doesn't address the question, but hopefully close enough to be useful: we know some things about $\mathbb{E} L(Z,Y)$ for other loss functions $L$.

If and only if $L$ is a Bregman divergence, the mean is the minimizer. This is attributed to Banerjee et al. 2005; try Mark Reid's blog post and its references.

By phrasing things differently and asking to maximize an expected score rather than minimize an expected loss (though there is no formal difference), the setting starts to look like proper scoring rules (e.g. Savage 1971, Gneiting and Raftery 2007). These are loss functions for eliciting distributions and they are well-understood; again, essentially just Bregman divergences.

A generalization of this question which is just receiving study recently is property elicitation: other loss functions and how they connect to the "property" of the distribution that solves the minimization problem. For example, if the loss is $|Y-Z|$, the minimizer is the median; if it is $\mathbf{1}_{Y=Z}$ then it is the mode, etc.

For references there I would just point to the Information Elicitation page created by Rafael Frongillo and I for this purpose, which has some slides as well. I think the paper to look at first there would be Lambert et al. 2008.

...

(1971) Leonard J. Savage. "Elicitation of personal probabilities and expectations".

(2005) Banerjee et al. "On the Optimality of Conditional Expectation as a Bregman Predictor".

(2007) Tilman Gneiting and Adrian E. Raftery. "Strictly proper scoring rules, prediction, and estimation".

(2008) Nicolas S. Lambert, David M. Pennock, and Yoav Shoham. "Eliciting properties of probability distributions".

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I see you accepted this answer, but that seems premature - hopefully there is more (relevant) info out there! $\endgroup$
    – usul
    Commented Oct 1, 2016 at 23:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .