I'm following the gentle introduction to Reproducing Kernel Hilbert Spaces From Zero to Reproducing Kernel Hilbert Spaces in Twelve Pages or Less by Hal Daumé III. I believe the author fully intended to hand-wave in several places, and that's been mostly fine for my purposes since I've been able to extract the main idea and fill in some of the details, but I do not have a slight idea about the following.

We pick it up in Section 6.4 on page 9 and follow the author's notation. Given a positive definite kernel $K$ over $X$, we wish to construct an RKHS $\mathcal{H}$ of functions $f : X \to \mathbb{R}$. For this construction, we need an inner product, so the author writes the obvious thing

$$ \langle k_x , k_y \rangle_{\mathcal{H}_K} = \Big\langle \sum_i \alpha_i k_{x_{i}}, \sum_i \beta_i k_{y_{i}} \Big\rangle_{X} $$

since we have already added the span of $\{k_x\}_{x\in X}$ to the RKHS under construction.

One page later, the author appeals to Mercer-Hilbert-Schmit theorems and Fourier analysis to argue that the dot product can be defined by

$$ \langle y, y^{\prime}\rangle_{\mathcal{H}_{K}} = \sum_{i=0}^{\infty} \dfrac{y_i y_{i}^{\prime}}{\lambda_i} $$

where $\lambda_i$ are eigenvalues and

$$ y_i := \int y(x) \phi_i (x) dx $$

for eigenfunctions $\phi$ (the decision to switch to $y\in \mathcal{H}$ was the author's).

My question is NOT the derivation of this, but why it is needed. What's so wrong with the first inner product we defined that forces us to come up with this second representation based on eigenfunctions? Thanks.

  • 1
    $\begingroup$ Nothing is wrong with the first definition, but perhaps the second one is "nicer" because it won't be obvious how to expand an arbitrary function into a linear combination of kernel functions, and you have to do this because the first definition only applies to kernel functions. $\endgroup$ – Nik Weaver Feb 3 at 18:07
  • 3
    $\begingroup$ But the meta answer is that maybe you should try reading something else. Literally any of the top results when you google "reproducing kernel Hilbert space" is better (more correct, less confused) than the paper you're looking at. $\endgroup$ – Nik Weaver Feb 3 at 18:08
  • 1
    $\begingroup$ Finally --- the question seems high enough level to merit posting on mathoverflow, however the community does try to discourage posting on both forums at once. Post one one of them, and if you haven't got any good answers after a day or two, maybe try posting on the other. $\endgroup$ – Nik Weaver Feb 3 at 18:09
  • 1
    $\begingroup$ thesis.bilkent.edu.tr/0002953.pdf In section 2 the author does a good job in the introducing RKHS's $\endgroup$ – erz Feb 4 at 0:57
  • 1
    $\begingroup$ @NikWeaver Regarding your take on the inner product, thanks, that makes sense. Regarding the tutorial, I think you're right, but the bird's eye view it provides is helping me place the more in-depth reviews in context. Also, the quick terminology refresher was useful to me. Regarding the math overflow etiquette, thanks for the clarification. $\endgroup$ – RMurphy Feb 5 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.