# What is the "natural" or "physical" norm on the Hessian matrix (and other higher derivatives)?

Let $$u : \mathbb R^n \rightarrow \mathbb R$$ and let $$H : \mathbb R^n \rightarrow \mathbb R^{n \times n}$$ be its Hessian matrix. What is the "natural" choice of pointwise norm on the Hessian matrix?

Here, a norm is "natural" if it satisfies favorable physical or geometric properties. I strongly suspect that this related to the "natural" choice of norms on symmetric matrices, though the context of taking derivatives may entail some changes.

Some candidates include:

1. The Frobenius Norm, which is the $$\ell^2$$ norm of the vector of entries: if $$A = ( a_{ij} )$$ is any $$n \times n$$ matrix, then the Frobenius norm is $$\|A\|_F = \sqrt{ \sum_{ 1 \leq i, j \leq n } |a_{ij}|^2 }.$$
2. The Spectral Norm, which is defined as $$\|A\|_\sigma = \sup_{ u,v \in \mathbb R^n \setminus \{0\} } \frac{ u^t A v }{\|u\| \cdot \|v\|}$$ This agrees with the largest singular value of the matrix: $$\|A\|_2 = \sigma_{\text{max}} (A)$$
3. Inspired by the previous examples, the $$p$$-Schatten norm is the $$\ell^p$$-norm of the vector of singular values of the matrix: $$\|A\|_{\sigma,p} = \sqrt[p]{ \sum_{i=1}^n \sigma_{i}(A)^p }.$$ We know that $$\|A\|_{\sigma,1} = \|A\|_{F}$$ and $$\|A\|_{\sigma,\infty} = \|A\|_{\sigma}$$.

The Hessian matrix is $$2$$-tensor field, and hence we may be interested in its pullback along orthogonal transformations. All the above norms are invariant under the pullback along orthogonal transformations, that is, $$\| O^T A O \| = \| A \|$$ whenever $$O \in \mathbb R^{n \times n}$$ is an orthogonal matrix. (Note that the $$\ell^p$$-version of the Frobenius norm does not survive this invariance requirement unless $$p=2$$).

I am wondering under what circumstances any of these norms (or yet another norm) is the natural/physical/geometric choice of the norm on the Hessian.

Note that the Frobenius norm and the spectral norm have natural analogous for general symmetric $$k$$-tensors, and hence for the $$k$$-th derivative of a function. I suspect the same is true for the $$p$$-Schatten norms, though I am not terribly familiar with them.

• Explain the downvote. It is a reasonable question. Commented May 8 at 12:24
• Which type of "physical" are you interested in? Presumably the answer is parameterized by whichever specific question you intend. Commented May 16 at 15:44
• The Hessian generalizes to functions of several variables $f:\mathbb{R}^n\rightarrow\mathbb{R}^r$ by setting $H(f)=(H(\pi_1\circ f),\dots,H(\pi_r\circ f))$ where $\pi_i$ is the projection onto the $i$-th variable. The Hessian of the gradient therefore is a collection of Hermitian matrices $(A_1,\dots,A_n)$, and one can define a completely positive superoperator from $(A_1,\dots,A_n)$ a collection of Hermitian matrices defined by $\Phi(A_1,\dots,A_n)(X)=A_1XA_1^*+\dots+A_rXA_r^*$. Commented May 18 at 11:14
• The 2004 paper "Notes on super-operator norms induced by Schatten norms" by John Watrous goes over a few superoperator norms including the norm $\|\Phi\|_{p\rightarrow q}=\sup_{X\neq 0}\frac{\|\Phi(X)\|_p}{\|X\|_q}$ (the operator norm induced by the Schatten $p$ and $q$ norms). Commented May 18 at 11:54

We would like a matrix norm $$\|\cdot\|$$ to be invariant under orthogonal transformations in the sense that $$\|A\|=\|UAV\|$$ whenever $$U,V$$ are orthogonal mappings. But by the singular value decomposition, we know that every matrix norm invariant under orthogonal transformations is of the form $$\|A\|=f(\sigma_1(A),\dots,\sigma_n(A))$$ for some (symmetric) function $$f:[0,\infty)^n\rightarrow[0,\infty)$$ where $$\sigma_1(A)\geq\dots\geq\sigma_n(A)$$ are the singular values of $$A$$.

The power of the Schatten norm $$\|A\|_p^p$$ for $$p$$ even is notable since $$\|A\|_p^p$$ is both a homogeneous degree $$p$$ polynomial in the singular values of $$A$$ and also in the coefficients of $$A$$.

The Schatten norms $$\|A\|_p^p$$ for positive integer $$p$$ are natural in the sense that the ring of symmetric polynomials in $$n$$-variables over the field of real numbers is generated by the polynomials $$x_1^p+\dots+x_n^p$$. Furthermore, the ring of symmetric polynomials in $$n$$-variables over the field of real numbers is dense in the space of all symmetric continuous functions $$f:\mathbb{R}^n\rightarrow\mathbb{R}$$ in the topology of uniform convergence on compact sets by the Stone-Weierstrass theorem. In particular, the Schatten norms $$\|A\|_p$$ completely determine $$A$$ up to the equivalence relation $$A\simeq UAV$$ where $$U,V$$ are orthogonal.

Schatten norms of Jacobians

Suppose that $$f:\mathbb{R}^m\rightarrow\mathbb{R}^n$$ and $$A=J(f)(\mathbf{x}_0)$$. Then $$\lim_{\epsilon\rightarrow 0}\frac{f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0)}{\epsilon}=A\mathbf{x},$$ so $$\lim_{\epsilon\rightarrow 0}\left\|\frac{f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0)}{\epsilon}\right\|^2=\|A\mathbf{x}\|^2.$$ If $$\mathbf{x}$$ is a random variable, then the distribution of $$\|A\mathbf{x}\|^2$$ tells us how much $$f(\mathbf{z})$$ varies when $$\mathbf{z}$$ is approximately $$\mathbf{x}_0$$, but when $$\mathbf{x}$$ is standard Gaussian, the moment generating function of the distribution of $$\|A\mathbf{x}\|^2$$ can be characterized in terms of a Schatten norm generating function.

Suppose that $$\mathbf{x}$$ is a Gaussian random variable with mean $$0$$ and identity covariance matrix. Then $$\|A\mathbf{x}\|^2=\sigma_1^2 X_1^2+\dots+\sigma_n^2 X_n^2$$ where $$X_1,\dots,X_n$$ are independent standard normally distributed random variables of one variable. Therefore, $$\begin{split} E\big(\exp(t\|A\mathbf{x}\|^2)\big) & =E\Big(\exp\big(t\sigma(A)_1^2X_1^2+\cdots+t\sigma_n(A)^2X_n^2\big)\Big)\\ & =E\Big(\exp\big(t\sigma_1(A)^2X_1^2)\cdots\exp(t\sigma_n(A)^2X_n^2\big)\Big)\\ & =E\Big(\exp\big(t\sigma_1(A)^2X_1^2\big)\Big)\cdots E\Big(\exp\big(t\sigma_n(A)^2X_n^2\big)\Big)\\ & =\frac{1}{\sqrt{1-2t\sigma_1(A)}}\cdots\frac{1}{\sqrt{1-2t\sigma_n(A)}}. \end{split}$$

We then take logarithms to obtain $$\begin{split} \ln\Big(E\big(\exp(t\|A\mathbf{x}\|^2\big)\Big) &=-\frac{1}{2}\big(\ln(1-2t\sigma_1(A)^2)+\dots+\ln(1-2t\sigma_n(A)^2)\big)\\ &=\frac{1}{2} \sum_{k=1}^n\sum_{j=1}^\infty\frac{(2t\sigma_k(A)^2)^j}{j}\\ &=\sum_{k=1}^n\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\sigma_k(A)^{2j}\\ &=\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\|A\|_{2j}^{2j}. \end{split}$$

Schatten norms of Hessians at local minima

Suppose that $$f:\mathbb{R}^m\rightarrow\mathbb{R}$$ is a function that attains a local minimum on the input $$\mathbf{x}_0$$ and $$H(f)(\mathbf{x}_0)=B$$. Then $$\lim_{\epsilon\rightarrow 0}\frac{1}{\epsilon^2}\cdot(f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0))=\frac{1}{2}\cdot\langle B\mathbf{x},\mathbf{x}\rangle.$$ Therefore, the distribution of $$\langle B\mathbf{x},\mathbf{x}\rangle$$ when $$\mathbf{x}$$ is a standard Gaussian random variable tells how much, but this distribution is also characterized by a Schatten norm generating function.

Suppose that $$B$$ is a positive semidefinite matrix and $$A=\sqrt{B}$$ and $$\mathbf{x}$$ is still a standard Gaussian random variable. Then since $$\langle B\mathbf{x},\mathbf{x}\rangle=\langle A^2\mathbf{x},\mathbf{x}\rangle=\langle A\mathbf{x},A\mathbf{x}\rangle=\|A\mathbf{x}\|^2$$, we have $$\ln\Big(E\big(\exp(t\cdot\langle B\mathbf{x},\mathbf{x}\rangle)\big)\Big)=\sum_{j=1}^\infty\frac{1}{2}\cdot\frac{(2t)^j}{j}\|A\|_{2j}^{2j}=\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\|B\|_j^j.$$