What is the "natural" or "physical" norm on the Hessian matrix (and other higher derivatives)?

Question

Let $u : \mathbb R^n \rightarrow \mathbb R$ and let $H : \mathbb R^n \rightarrow \mathbb R^{n \times n}$ be its Hessian matrix. What is the "natural" choice of pointwise norm on the Hessian matrix?

Here, a norm is "natural" if it satisfies favorable physical or geometric properties. I strongly suspect that this related to the "natural" choice of norms on symmetric matrices, though the context of taking derivatives may entail some changes.

Some candidates include:

1. The Frobenius Norm, which is the $\ell^2$ norm of the vector of entries: if $A = ( a_{ij} )$ is any $n \times n$ matrix, then the Frobenius norm is $$ \|A\|_F = \sqrt{ \sum_{ 1 \leq i, j \leq n } |a_{ij}|^2 }. $$
2. The Spectral Norm, which is defined as $$ \|A\|_\sigma = \sup_{ u,v \in \mathbb R^n \setminus \{0\} } \frac{ u^t A v }{\|u\| \cdot \|v\|} $$ This agrees with the largest singular value of the matrix: $ \|A\|_2 = \sigma_{\text{max}} (A)$
3. Inspired by the previous examples, the $p$-Schatten norm is the $\ell^p$-norm of the vector of singular values of the matrix: $$ \|A\|_{\sigma,p} = \sqrt[p]{ \sum_{i=1}^n \sigma_{i}(A)^p }. $$ We know that $\|A\|_{\sigma,1} = \|A\|_{F}$ and $\|A\|_{\sigma,\infty} = \|A\|_{\sigma}$.

The Hessian matrix is $2$-tensor field, and hence we may be interested in its pullback along orthogonal transformations. All the above norms are invariant under the pullback along orthogonal transformations, that is, $$ \| O^T A O \| = \| A \| $$ whenever $O \in \mathbb R^{n \times n}$ is an orthogonal matrix. (Note that the $\ell^p$-version of the Frobenius norm does not survive this invariance requirement unless $p=2$).

I am wondering under what circumstances any of these norms (or yet another norm) is the natural/physical/geometric choice of the norm on the Hessian.

Note that the Frobenius norm and the spectral norm have natural analogous for general symmetric $k$-tensors, and hence for the $k$-th derivative of a function. I suspect the same is true for the $p$-Schatten norms, though I am not terribly familiar with them.

Answer 1

We would like a matrix norm $\|\cdot\|$ to be invariant under orthogonal transformations in the sense that $\|A\|=\|UAV\|$ whenever $U,V$ are orthogonal mappings. But by the singular value decomposition, we know that every matrix norm invariant under orthogonal transformations is of the form $\|A\|=f(\sigma_1(A),\dots,\sigma_n(A))$ for some (symmetric) function $f:[0,\infty)^n\rightarrow[0,\infty)$ where $\sigma_1(A)\geq\dots\geq\sigma_n(A)$ are the singular values of $A$.

The power of the Schatten norm $\|A\|_p^p$ for $p$ even is notable since $\|A\|_p^p$ is both a homogeneous degree $p$ polynomial in the singular values of $A$ and also in the coefficients of $A$.

The Schatten norms $\|A\|_p^p$ for positive integer $p$ are natural in the sense that the ring of symmetric polynomials in $n$-variables over the field of real numbers is generated by the polynomials $x_1^p+\dots+x_n^p$. Furthermore, the ring of symmetric polynomials in $n$-variables over the field of real numbers is dense in the space of all symmetric continuous functions $f:\mathbb{R}^n\rightarrow\mathbb{R}$ in the topology of uniform convergence on compact sets by the Stone-Weierstrass theorem. In particular, the Schatten norms $\|A\|_p$ completely determine $A$ up to the equivalence relation $A\simeq UAV$ where $U,V$ are orthogonal.

Schatten norms of Jacobians

Suppose that $f:\mathbb{R}^m\rightarrow\mathbb{R}^n$ and $A=J(f)(\mathbf{x}_0)$. Then $$ \lim_{\epsilon\rightarrow 0}\frac{f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0)}{\epsilon}=A\mathbf{x}, $$ so $$\lim_{\epsilon\rightarrow 0}\left\|\frac{f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0)}{\epsilon}\right\|^2=\|A\mathbf{x}\|^2. $$ If $\mathbf{x}$ is a random variable, then the distribution of $\|A\mathbf{x}\|^2$ tells us how much $f(\mathbf{z})$ varies when $\mathbf{z}$ is approximately $\mathbf{x}_0$, but when $\mathbf{x}$ is standard Gaussian, the moment generating function of the distribution of $\|A\mathbf{x}\|^2$ can be characterized in terms of a Schatten norm generating function.

Suppose that $\mathbf{x}$ is a Gaussian random variable with mean $0$ and identity covariance matrix. Then $\|A\mathbf{x}\|^2=\sigma_1^2 X_1^2+\dots+\sigma_n^2 X_n^2$ where $X_1,\dots,X_n$ are independent standard normally distributed random variables of one variable. Therefore, $$ \begin{split} E\big(\exp(t\|A\mathbf{x}\|^2)\big) & =E\Big(\exp\big(t\sigma(A)_1^2X_1^2+\cdots+t\sigma_n(A)^2X_n^2\big)\Big)\\ & =E\Big(\exp\big(t\sigma_1(A)^2X_1^2)\cdots\exp(t\sigma_n(A)^2X_n^2\big)\Big)\\ & =E\Big(\exp\big(t\sigma_1(A)^2X_1^2\big)\Big)\cdots E\Big(\exp\big(t\sigma_n(A)^2X_n^2\big)\Big)\\ & =\frac{1}{\sqrt{1-2t\sigma_1(A)}}\cdots\frac{1}{\sqrt{1-2t\sigma_n(A)}}. \end{split}$$

We then take logarithms to obtain $$ \begin{split} \ln\Big(E\big(\exp(t\|A\mathbf{x}\|^2\big)\Big) &=-\frac{1}{2}\big(\ln(1-2t\sigma_1(A)^2)+\dots+\ln(1-2t\sigma_n(A)^2)\big)\\ &=\frac{1}{2} \sum_{k=1}^n\sum_{j=1}^\infty\frac{(2t\sigma_k(A)^2)^j}{j}\\ &=\sum_{k=1}^n\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\sigma_k(A)^{2j}\\ &=\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\|A\|_{2j}^{2j}. \end{split}$$

Schatten norms of Hessians at local minima

Suppose that $f:\mathbb{R}^m\rightarrow\mathbb{R}$ is a function that attains a local minimum on the input $\mathbf{x}_0$ and $H(f)(\mathbf{x}_0)=B$. Then $\lim_{\epsilon\rightarrow 0}\frac{1}{\epsilon^2}\cdot(f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0))=\frac{1}{2}\cdot\langle B\mathbf{x},\mathbf{x}\rangle.$ Therefore, the distribution of $\langle B\mathbf{x},\mathbf{x}\rangle$ when $\mathbf{x}$ is a standard Gaussian random variable tells how much, but this distribution is also characterized by a Schatten norm generating function.

Suppose that $B$ is a positive semidefinite matrix and $A=\sqrt{B}$ and $\mathbf{x}$ is still a standard Gaussian random variable. Then since $\langle B\mathbf{x},\mathbf{x}\rangle=\langle A^2\mathbf{x},\mathbf{x}\rangle=\langle A\mathbf{x},A\mathbf{x}\rangle=\|A\mathbf{x}\|^2$, we have $$ \ln\Big(E\big(\exp(t\cdot\langle B\mathbf{x},\mathbf{x}\rangle)\big)\Big)=\sum_{j=1}^\infty\frac{1}{2}\cdot\frac{(2t)^j}{j}\|A\|_{2j}^{2j}=\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\|B\|_j^j.$$