In supervised machine learning, we typically take a Risk Minimization (RM) point of view when formulating a problem. So, what we typically solve for is the following:

$$ R^L(h) = \underset{h\in\mathcal{H}}{\text{min}}\,\mathbb{E}_{(x,y_x)\sim\mathcal{D}} \Big[ L(h(x), y_x)\Big]$$

where $h(\cdot): \mathbb{R}^m \to \mathbb{R}^d$ ($m$- dimension of input/feature vector and $d$ denotes the number of classes for the classification task) denotes a classifier function, $\mathcal{H}$ is hypothesis class/family of classifiers (essentially a Hilbert Space over functions on $\mathbb{R}^m$), $L: \mathbb{R}^d \to \mathbb{R}^+$ is a loss function, and $(x,y_x)$ are i.i.d. samples from a distribution $\mathcal{D}$.

In practice, however, one often employs what's called Regularized Risk Minimization (RRM):

$$ R_\Omega^L(h) = \underset{h\in\mathcal{H}}{\text{min}}\,\mathbb{E}_{(x,y_x)\sim\mathcal{D}} \Big[ L(h(x), y_x) + \lambda\Omega(h)\Big]$$

where $\Omega: \mathcal{H} \to \mathbb{R}^+$ is a differentiable regularization function.

I want to understand under what necessary and/or sufficient conditions can the minimizers of $R^L(h)$ and $R_\Omega^L(h)$ be the same. That is, under what conditions do we get:

$$ \text{arg}\, \underset{h\in\mathcal{H}}{\text{min}} \, R^L(h) = \text{arg}\, \underset{h\in\mathcal{H}}{\text{min}} \, R_\Omega^L(h)$$

Is it even possible in the first place?

In supervised machine learning, we typically take a Risk Minimization (RM) point of view when formulating a problem. So, what we typically solve for is the following:

$$ R^L(h) = \underset{h\in\mathcal{H}}{\text{min}}\,\mathbb{E}_{(x,y_x)\sim\mathcal{D}} \Big[ L(h(x), y_x)\Big]$$

where $h(\cdot): \mathbb{R}^m \to \mathbb{R}^d$ ($m$- dimension of input/feature vector and $d$ denotes the number of classes for the classification task) denotes a classifier function, $\mathcal{H}$ is hypothesis class/family of classifiers (essentially a Hilbert space over functions on $\mathbb{R}^m$), $L: \mathbb{R}^d \to \mathbb{R}^+$ is a loss function, and $(x,y_x)$ are i.i.d. samples from a distribution $\mathcal{D}$.

In practice, however, one often employs what's called Regularized Risk Minimization (RRM):

$$ R_\Omega^L(h) = \underset{h\in\mathcal{H}}{\text{min}}\,\mathbb{E}_{(x,y_x)\sim\mathcal{D}} \Big[ L(h(x), y_x) + \lambda\Omega(h)\Big]$$

where $\Omega: \mathcal{H} \to \mathbb{R}^+$ is a differentiable regularization function.

I want to understand under what necessary and/or sufficient conditions can the minimizers of $R^L(h)$ and $R_\Omega^L(h)$ be the same. That is, under what conditions do we get:

$$ \text{arg}\, \underset{h\in\mathcal{H}}{\text{min}} \, R^L(h) = \text{arg}\, \underset{h\in\mathcal{H}}{\text{min}} \, R_\Omega^L(h)$$

Is it even possible in the first place?

In supervised machine learning, we typically take a Risk Minimization (RM) point of view when formulating a problem. So, what we typically solve for is the following:

$$ R^L(h) = \underset{h\in\mathcal{H}}{\text{min}}\,\mathbb{E}_{(x,y_x)\sim\mathcal{D}} \Big[ L(h(x), y_x)\Big]$$

where $h(\cdot)$ denotes a classifier function, $\mathcal{H}$ is the hypothesis class (i.e. family of classifiers), $L$ is a loss function, and $(x,y_x)$ are i.i.d. samples from a distribution $\mathcal{D}$.

In practice, however, one often employs what's called Regularized Risk Minimization (RRM):

$$ R_\Omega^L(h) = \underset{h\in\mathcal{H}}{\text{min}}\,\mathbb{E}_{(x,y_x)\sim\mathcal{D}} \Big[ L(h(x), y_x) + \lambda\Omega(h)\Big]$$

where $\Omega$ is a regularizer function.

I want to understand under what conditions can the minimizers of $R^L(h)$ and $R_\Omega^L(h)$ be the same. That is, under what conditions do we get:

$$ \text{arg}\, \underset{h\in\mathcal{H}}{\text{min}} \, R^L(h) = \text{arg}\, \underset{h\in\mathcal{H}}{\text{min}} \, R_\Omega^L(h)$$

Is it even possible in the first place?

In supervised machine learning, we typically take a Risk Minimization (RM) point of view when formulating a problem. So, what we typically solve for is the following:

$$ R^L(h) = \underset{h\in\mathcal{H}}{\text{min}}\,\mathbb{E}_{(x,y_x)\sim\mathcal{D}} \Big[ L(h(x), y_x)\Big]$$

where $h(\cdot): \mathbb{R}^m \to \mathbb{R}^d$ ($m$- dimension of input/feature vector and $d$ denotes the number of classes for the classification task) denotes a classifier function, $\mathcal{H}$ is hypothesis class/family of classifiers (essentially a Hilbert Space over functions on $\mathbb{R}^m$), $L: \mathbb{R}^d \to \mathbb{R}^+$ is a loss function, and $(x,y_x)$ are i.i.d. samples from a distribution $\mathcal{D}$.

In practice, however, one often employs what's called Regularized Risk Minimization (RRM):

$$ R_\Omega^L(h) = \underset{h\in\mathcal{H}}{\text{min}}\,\mathbb{E}_{(x,y_x)\sim\mathcal{D}} \Big[ L(h(x), y_x) + \lambda\Omega(h)\Big]$$

where $\Omega: \mathcal{H} \to \mathbb{R}^+$ is a differentiable regularization function.

I want to understand under what necessary and/or sufficient conditions can the minimizers of $R^L(h)$ and $R_\Omega^L(h)$ be the same. That is, under what conditions do we get:

$$ \text{arg}\, \underset{h\in\mathcal{H}}{\text{min}} \, R^L(h) = \text{arg}\, \underset{h\in\mathcal{H}}{\text{min}} \, R_\Omega^L(h)$$

Is it even possible in the first place?