Monotonicity of kernel matrices with respect to hyperparameters

Question

Let $\mathcal{X}$ be some nice space, let $\Phi$ be some ordered space, and let $K :\mathcal{X} \times \mathcal{X} \times \Phi \to \mathbf{R}$ be a positive-semidefinite kernel indexed by a hyperparameter $\phi$, i.e. for any $\phi \in \Phi$, the mapping $K(\cdot, \cdot, \phi) :\mathcal{X} \times \mathcal{X} \to \mathbf{R}$ is a positive-semidefinite kernel in the sense of Mercer.

Now, fix a finite subset $\mathcal{X}_N = \{ x_1 , \ldots, x_N \} \subseteq \mathcal{X}$ of size $N$, and consider the mapping $\mathbf{K}$ which takes a hyperparameter $\phi$ and maps it to the corresponding $N \times N$ kernel matrix, i.e.

\begin{align} \mathbf{K} ( \phi) \in \mathrm{PSD}_{N \times N}, \quad \\ \mathbf{K} ( \phi)_{i, j} = K (x_i, x_j, \phi). \end{align}

I'm interested in examples of parametric kernels $K$ for which this mapping is order-preserving (for any set $\mathcal{X}_N$), in the sense that when $\phi_1 \prec \phi_2$ with respect to the ordering on $\Phi$, it follows that $\mathbf{K} ( \phi_1 ) \prec \mathbf{K} ( \phi_2)$ in the semidefinite ordering on matrices.

Some simple examples of which I'm already aware:

• Let $K_1$, $K_2$ be two PSD kernels, let $\Phi = \mathbf{R}_+$, and let $K(x, y, \phi) = K_1 (x, y) + \phi \cdot K_2 (x, y)$.
• Let $\{ K_i : i \in \mathbf{N} \}$ be a countable collection of PSD kernels, let $\Phi = \mathbf{N}$, and let $K(x, y, \phi) = \sum_{i \leqslant \phi} K_i (x, y)$.

I'd be particularly interested in cases where such an ordering holds as e.g. the length scale of a kernel varies, or similar.

Answer 1

This example may be a little bit ridiculous, but suppose we take $\mathcal{X}=\mathbb{R}$ and let $\Phi$ be any parametric subset of the set of PSD kernels itself. We define $$ \mathbf{K}(\phi)_{i,j} = \sum_{i,j}\phi(x_i, x_j) $$

We will say that $\phi_1 \preceq \phi_2$ iff for any $N$, $x_1,...,x_N$, and $c_1,...,c_N\geq 0$, we have $$\sum_{i,j} c_i c_j (\phi_2(x_i,x_j)-\phi_1(x_i,x_j)) \geq 0$$ This gives a partial order on $\Phi$ that (by construction) respects the OP's property of interest.

Possibly the OP is less interested in the preceding construction and more interested in concrete parametric examples. If so, geodesics of the cone of PSD matrices provide some interesting examples. For example, take $K_1, K_2$ as two PSD kernels, and let $\Phi=[0,1]$. For a fixed $N$, let $A$ and $B$ be the two corresponding PSD matrices. Then define $$K(x,y,\phi)=A^{1/2}\exp(\phi \log(A^{-1/2}BA^{-1/2}))A^{1/2}$$ More inspiration along that line might be found at Bonnabel and Sepulchre's "Riemannian Metric and Geometric Mean for Positive Semidefinite Matrices of Fixed Rank"

Answer 2

An example which occurred to me is the following: suppose that $K$ indexes a family of stationary kernels (on e.g. $\mathbf{R}^d$) whose Fourier transforms are ordered pointwise. Following e.g. the proof of Bochner's theorem, one can show that the corresponding kernel matrices are ordered. This furnishes a number of practical examples, e.g. consider the kernels with Fourier transform given by $\hat{K} (\omega; \phi) = (1 + \phi \cdot \| \omega \|_2^2 )^{-s}$, which correspond to suitable scalings of Matèrn kernels.

Note that simply ordering the kernels pointwise need not have the same effect (roughly because symmetric matrices with positive entries need not be positive-definite).