Orthonormal bases on Reproducing Kernel Hilbert Spaces

Question

Recall that a Hilbert space $\mathcal{H}$ is a reproducing kernel Hilbert space (RKHS) if the elements of $\mathcal{H}$ are functions on a certain set $X$ and for any $a\in X$, the linear functional $f\mapsto f(a)$ is bounded on $\mathcal{H}$. By Riesz Representation Theorem, there exists an element $K_a\in\mathcal{H}$ such that $$f(a) = \langle f, K_a\rangle\ \text{ for all } \ f\in\mathcal{H}.$$ The function $K(x,y) = K_y(x) = \langle K_y, K_x\rangle$ defined on $X\times X$ is called the reproducing kernel function of $\mathcal{H}$.

It is well known and easy to show that for any orthonormal basis $\{e_m\}_{m=1}^{\infty}$ for $\mathcal{H}$, we have the formula $$K(x,y) = \sum_{m=1}^{\infty}e_m(x)\overline{e_m(y)},\tag{Eqn 1}$$ where the convergence is pointwise on $X\times X$.

My question concerns the converse of the above statement.

Question: if $\{g_m\}_{m=1}^{\infty}$ is a sequence of functions in $\mathcal{H}$ such that $$K(x,y) = \sum_{m=1}^{\infty}g_m(x)\overline{g_m(y)}\tag{Eqn 2}$$ for all $x,y\in X$. Is the sequence $\{g_m\}_{m=1}^{\infty}$ an orthonormal basis for $\mathcal{H}$?

The answer to this question is clearly negative since equation (Eqn 1) can be re-written as $$K(x,y) = \frac{e_1(x)}{\sqrt{2}}\overline{\frac{e_1(y)}{\sqrt{2}}}+\frac{e_1(x)}{\sqrt{2}}\overline{\frac{e_1(y)}{\sqrt{2}}}+\sum_{m=2}^{\infty}e_m(x)\overline{e_m(y)}$$ and clearly $\{e_1/\sqrt{2}, e_1/\sqrt{2}, e_2, \ldots\}$ is not an orthonormal basis for $\mathcal{H}$. So the following additional condition should be added: the sequence $\{g_m\}_{m=1}^{\infty}$ is linearly independent.

The following proof suggests that the answer is affirmative. (For those who are familiar with the proof of the Moore-Aronszajn's Theorem in the theory of RKHS, the proof here looks similar.) Assume that we have (Eqn 2) and the sequence $\{g_m\}_{m=1}^{\infty}$ is linearly independent. Let $\mathcal M$ be the linear space spanned by the functions $\{g_m\}_{m=1}^{\infty}$. Define a sesquilinear form on $\mathcal M$ as \begin{align*} \left\langle\sum_{\text{finite sum}}a_jg_j, \sum_{\text{finite sum}}b_kg_k\right\rangle_{\mathcal M} = \sum_{\text{finite sum}} a_j\overline{b}_j. \end{align*} Since $\{g_m\}_{m=1}^{\infty}$ is a linearly independent set, the above definition is well-defined. Note that $\{g_m\}_{m=1}^{\infty}$ is an orthonormal set in $\langle,\rangle_{\mathcal M}$. For any $f\in\mathcal M$ and $x\in X$, we have \begin{align*} f(x) = \sum_{\text{finite sum}}\langle f,g_m\rangle_{\mathcal M}\,g_m(x). \end{align*} Cauchy-Schwarz's inequality gives \begin{align*} |f(x)| & \leq \Big(\sum_{\text{finite sum}}|\langle f,g_m\rangle_{\mathcal M}|^2\Big)^{1/2}\Big(\sum_{\text{finite sum}}|g_m(x)|^2\Big)^{1/2} \leq \|f\|_{\mathcal M}\sqrt{K(x,x)}. \end{align*} Let $\widetilde{\mathcal M}$ be the Hilbert space completion of $\mathcal M$. The standard argument shows that $\widetilde{\mathcal M}$ is a RKHS of functions on $X$. What is the kernel of $\widetilde{\mathcal M}$? Since $\{g_m\}_{m=1}^{\infty}$ is an orthonormal set and its span is dense in $\widetilde{\mathcal M}$, it is an orthonormal basis for $\widetilde{\mathcal M}$. The kernel of $\widetilde{\mathcal M}$ then can be computed as $$\sum_{m=1}^{\infty}g_m(x)\bar{g}_m(y),$$ which is the same as $K(x,y)$. Therefore, $\widetilde{\mathcal M}$ is the same as $\mathcal H$ (they consist of the same functions and the inner products on the two spaces are equal). Consequently, $\{g_m\}_{m=1}^{\infty}$ is an orthonormal basis for $\mathcal{H}$. This completes the proof.

Counterexample: On the other hand, there are counterexamples that provide a negative answer to the question in the infinite dimensional case.

What part of the above proof is incorrect? I have checked but could not figure out what went wrong.

Thank you.

Answer 1

The error is in this line:

The standard argument shows that $\widetilde{\mathcal{M}}$ is an RKHS of functions on $X$.

In fact, this is not generally true. The completion $\widetilde{\mathcal{M}}$ may not be naturally identified with a space of functions on $X$.

The "obvious" way that one would try to prove this is as follows. Consider an element $\phi \in \widetilde{\mathcal{M}}$. Since $\mathcal{M}$ is dense in its completion, there is a sequence $\{f_n\} \subset \mathcal{M}$ such that $f_n \to \phi$ in $\widetilde{\mathcal{M}}$-norm, which is an extension of the $\mathcal{M}$-norm. In particular, $\{f_n\}$ is $\mathcal{M}$-norm Cauchy. Because of your inequality $|f(x)| \le \|f\|_{\mathcal{M}} \sqrt{K(x,x)}$ (*), we have that $\{f_n(x)\}$ is Cauchy in $\mathbb{R}$ for each $x$. So $\{f_n(x)\}$ converges to a number which we may call $f_\phi(x)$; that is, $f_n \to f_\phi$ pointwise. It is also easy to show that $f_{\phi}$ does not depend on the choice of sequence $f_n \to \phi$, so the linear map $\phi \mapsto f_\phi$ of $\widetilde{\mathcal{M}}$ into $\mathbb{R}^X$ is well defined. Let's call this map $T$.

So the "obvious" thing to do is to look at the image of $T$, which is of course a function space, and make it a Hilbert space by pushing forward the $\widetilde{\mathcal{M}}$-norm. If this works, then the same inequality (*) will show that the evaluation map is continuous in this norm, and we have ourselves an RKHS.

The problem is that $T$ might fail to be injective. In other words, we could have a nonzero $\phi$ for which $f_\phi = 0$. In that case, pushing forward the norm will not work; it will not be well defined on the image of $T$.

This is exactly what happens in the counterexample you discuss in the comments. As you suggest, consider $\ell^2$ as an RKHS on $X = \mathbb{N} = \{1,2,\dots\}$, with its usual orthonormal basis $\{e_n\}$, and let $v = \sum_n \frac{1}{n} e_n$. Let $H = \{v\}^\perp$ with the same $\ell^2$ inner product; being a closed subspace of $\ell^2$, $H$ is an RKHS.

Let $P : \ell^2 \to H$ be the orthogonal projection and let $g_n = P e_n$. The $g_n$ are linearly independent; if $0 = a_1 g_1 + \dots + a_n g_n = P(a_1 e_1 + \dots + a_n e_n)$ then $a_1 e_1 + \dots + a_n e_n$ is a scalar multiple of $v$, which is only possible if all $a_i$ are 0.

Following your construction, let $\mathcal{M} \subset H$ be the linear span of $\{g_n\}$ and let $\langle\cdot, \cdot\rangle_{\mathcal{M}}$ be the inner product on $\mathcal{M}$ which makes the $\{g_n\}$ orthonormal.

Set $h_m = \sum_{n=1}^m \frac{1}{n} g_n$. Clearly the sequence $\{h_m\}$ is Cauchy in $\mathcal{M}$, so it converges in the completion $\widetilde{\mathcal{M}}$ to some $\phi$. It is also clear that $\|\phi\|_\widetilde{\mathcal{M}}^2 = \frac{\pi^2}{6}$ so in particular $\phi \ne 0$.

But on the other hand, we have $h_m = \sum_{n=1}^m \frac{1}{n} P e_n = P\left(\sum_{n=1}^m \frac{1}{n} e_n\right)$. By continuity of $P$, we have $h_m \to Pv = 0$ in $\ell^2$, and thus also pointwise. So $f_\phi = 0$.

---

This is basically an example of a somewhat paradoxical fact that's bitten me before. Let $X,Y$ be Banach spaces, and suppose $E \subset X$ is dense. It's well known that every bounded operator $T : E \to Y$ has a unique bounded extension $\tilde{T} : X \to Y$. But it's possible that $T$ is injective while $\tilde{T}$ is not. Indeed, we've just constructed an example, by letting $E = \mathcal{M}$, $X = \widetilde{\mathcal{M}}$, $Y = \ell^2$, and $T : E \to Y$ the inclusion map.