# Representer theorem for a loss / functional of the form $L(h) := \sum_{i=1}^n (|h(x_i)-y_i|+t\|h\|)^2$

Let $$K:X \times X \to \mathbb R$$ be a (positive-definite) kernel and let $$H$$ be the induced reproducing kernel Hilbert space (RKHS). Fix $$(x_1,y_1),\ldots,(x_n,y_n) \in X \times \mathbb R$$. For $$t \ge 0$$, consider the functional $$L:H \to \mathbb R$$ defined by

$$\tag{1} L(h) := \sum_{i=1}^n (|h(x_i)-y_i|+t\|h\|)^2.$$

Question. Is there a representer theorem for minimizers of $$L$$? That is, is it true that every minimizer $$h$$ of $$L$$ can be written as $$h = \sum_{i=1}^n c_i K_{x_i}$$, where the $$c_i$$'s are scalars and $$K_{x_i} \in H$$ is defined by $$K_{x_i}(x) = K(x_i,x)$$ for any $$x \in X$$?

Note that if $$t=0$$, then $$L = \sum_{i=1}^n (h(x_i)-y_i)^2$$, and the classical representer theorem applies.

More generally, given functions $$G:\mathbb R^{n+1} \to \mathbb R$$ and $$u:\mathbb R \to \mathbb R$$, I'm interested in conditions under which the functional $$L:H \to \mathbb R$$ defined by $$L(h) := G(u(x_1,y_1),\ldots,u(x_n,y_n),\|h\|),$$ admits a representer theorem. In particular, (1) corresponds to the case where $$G(r_1,\ldots,r_n,b) = \sum_i^n (r_i + t b)^2$$ and $$u(c,d) = |c-d|$$, for all $$r_1,\ldots,r_n,b,c,d \in \mathbb R$$.

## A perhaps useful observation

For any $$\alpha \in (0,1)$$, define the function $$L_{\alpha}:H \to \mathbb R$$ by $$L_{\alpha}(h) := \dfrac{1}{\alpha}\sum_{i=1}^n(h(x_i)-y_i)^2 + \dfrac{t^2}{1-\alpha}\|h\|^2.$$

Then, for the function $$L:H \to \mathbb R$$ defined in (1), one has the identity

$$L(h) = \inf_{\alpha \in (0,1)}L_{\alpha}(h).$$

It is clear that the $$L_{\alpha}$$'s admit a representer theorem. I don't know if any of this is useful.

• Did you know this text? Mar 23, 2022 at 0:31
• @JoséCFerreira No, but would be happy to know how this connects with my problem (section number, etc.). Thanks in advance. Mar 23, 2022 at 1:16

Let $$h=h_1 + h_2$$, with $$h_1\in\mathrm{span}(\{K(\cdot, x_i)\}_{i=1}^n)$$ and $$h_2$$ in the orthogonal complement of this last set. It follows that $$\|h\|_H^2 = \|h_1\|_H^2+ \|h_2\|_H^2$$, and $$\|h\|_H^2 \geq \|h_1\|_H^2$$.
The reproducing property, gives you the equality $$h(x_i) = \langle h,K(\cdot,x_i)\rangle = \langle h_1,K(\cdot,x_i)\rangle + \langle h_2,K(\cdot,x_i)\rangle = \langle h_1,K(\cdot,x_i)\rangle =h_1(x_i)$$ Because $$h_2$$ is orthogonal to $$\mathrm{span}(\{K(\cdot, x_i)\}_{i=1}^n)$$.
It follows that, if $$\|h_2\|>0$$ then $$L(h) = \sum_{i=1}^n (|h(x_i)-y_i|+t\|h\|)^2>\sum_{i=1}^n (|h_1(x_i)-y_i|+t\|h_1\|)^2=L(h_1).$$
1. To a more general case, let $$L(h) := G(u(h(x_1),y_1),\ldots,u(h(x_n),y_n),\|h\|),$$ being an increasing function, with respect to $$\|h\|$$. It holds the inequality $$L(h)\geq L(h_1),$$ and all minimizers of $$L(h)$$, if they exists, are in $$\mathrm{span}(\{K(\cdot, x_i)\}_{i=1}^n)$$.
2. Searching for "$$\langle f,K(x,\cdot)\rangle$$" on SearchOnMath perhaps you find some usefull results about the representer theorem.