5
$\begingroup$

Let $u : \mathbb R^n \rightarrow \mathbb R$ and let $H : \mathbb R^n \rightarrow \mathbb R^{n \times n}$ be its Hessian matrix. What is the "natural" choice of pointwise norm on the Hessian matrix?

Here, a norm is "natural" if it satisfies favorable physical or geometric properties. I strongly suspect that this related to the "natural" choice of norms on symmetric matrices, though the context of taking derivatives may entail some changes.

Some candidates include:

  1. The Frobenius Norm, which is the $\ell^2$ norm of the vector of entries: if $A = ( a_{ij} )$ is any $n \times n$ matrix, then the Frobenius norm is $$ \|A\|_F = \sqrt{ \sum_{ 1 \leq i, j \leq n } |a_{ij}|^2 }. $$
  2. The Spectral Norm, which is defined as $$ \|A\|_\sigma = \sup_{ u,v \in \mathbb R^n \setminus \{0\} } \frac{ u^t A v }{\|u\| \cdot \|v\|} $$ This agrees with the largest singular value of the matrix: $ \|A\|_2 = \sigma_{\text{max}} (A)$
  3. Inspired by the previous examples, the $p$-Schatten norm is the $\ell^p$-norm of the vector of singular values of the matrix: $$ \|A\|_{\sigma,p} = \sqrt[p]{ \sum_{i=1}^n \sigma_{i}(A)^p }. $$ We know that $\|A\|_{\sigma,1} = \|A\|_{F}$ and $\|A\|_{\sigma,\infty} = \|A\|_{\sigma}$.

The Hessian matrix is $2$-tensor field, and hence we may be interested in its pullback along orthogonal transformations. All the above norms are invariant under the pullback along orthogonal transformations, that is, $$ \| O^T A O \| = \| A \| $$ whenever $O \in \mathbb R^{n \times n}$ is an orthogonal matrix. (Note that the $\ell^p$-version of the Frobenius norm does not survive this invariance requirement unless $p=2$).

I am wondering under what circumstances any of these norms (or yet another norm) is the natural/physical/geometric choice of the norm on the Hessian.

Note that the Frobenius norm and the spectral norm have natural analogous for general symmetric $k$-tensors, and hence for the $k$-th derivative of a function. I suspect the same is true for the $p$-Schatten norms, though I am not terribly familiar with them.

$\endgroup$
4
  • $\begingroup$ Explain the downvote. It is a reasonable question. $\endgroup$ Commented May 8 at 12:24
  • $\begingroup$ Which type of "physical" are you interested in? Presumably the answer is parameterized by whichever specific question you intend. $\endgroup$ Commented May 16 at 15:44
  • $\begingroup$ The Hessian generalizes to functions of several variables $f:\mathbb{R}^n\rightarrow\mathbb{R}^r$ by setting $H(f)=(H(\pi_1\circ f),\dots,H(\pi_r\circ f))$ where $\pi_i$ is the projection onto the $i$-th variable. The Hessian of the gradient therefore is a collection of Hermitian matrices $(A_1,\dots,A_n)$, and one can define a completely positive superoperator from $(A_1,\dots,A_n)$ a collection of Hermitian matrices defined by $\Phi(A_1,\dots,A_n)(X)=A_1XA_1^*+\dots+A_rXA_r^*$. $\endgroup$ Commented May 18 at 11:14
  • $\begingroup$ The 2004 paper "Notes on super-operator norms induced by Schatten norms" by John Watrous goes over a few superoperator norms including the norm $\|\Phi\|_{p\rightarrow q}=\sup_{X\neq 0}\frac{\|\Phi(X)\|_p}{\|X\|_q}$ (the operator norm induced by the Schatten $p$ and $q$ norms). $\endgroup$ Commented May 18 at 11:54

1 Answer 1

3
$\begingroup$

We would like a matrix norm $\|\cdot\|$ to be invariant under orthogonal transformations in the sense that $\|A\|=\|UAV\|$ whenever $U,V$ are orthogonal mappings. But by the singular value decomposition, we know that every matrix norm invariant under orthogonal transformations is of the form $\|A\|=f(\sigma_1(A),\dots,\sigma_n(A))$ for some (symmetric) function $f:[0,\infty)^n\rightarrow[0,\infty)$ where $\sigma_1(A)\geq\dots\geq\sigma_n(A)$ are the singular values of $A$.

The power of the Schatten norm $\|A\|_p^p$ for $p$ even is notable since $\|A\|_p^p$ is both a homogeneous degree $p$ polynomial in the singular values of $A$ and also in the coefficients of $A$.

The Schatten norms $\|A\|_p^p$ for positive integer $p$ are natural in the sense that the ring of symmetric polynomials in $n$-variables over the field of real numbers is generated by the polynomials $x_1^p+\dots+x_n^p$. Furthermore, the ring of symmetric polynomials in $n$-variables over the field of real numbers is dense in the space of all symmetric continuous functions $f:\mathbb{R}^n\rightarrow\mathbb{R}$ in the topology of uniform convergence on compact sets by the Stone-Weierstrass theorem. In particular, the Schatten norms $\|A\|_p$ completely determine $A$ up to the equivalence relation $A\simeq UAV$ where $U,V$ are orthogonal.

Schatten norms of Jacobians

Suppose that $f:\mathbb{R}^m\rightarrow\mathbb{R}^n$ and $A=J(f)(\mathbf{x}_0)$. Then $$ \lim_{\epsilon\rightarrow 0}\frac{f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0)}{\epsilon}=A\mathbf{x}, $$ so $$\lim_{\epsilon\rightarrow 0}\left\|\frac{f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0)}{\epsilon}\right\|^2=\|A\mathbf{x}\|^2. $$ If $\mathbf{x}$ is a random variable, then the distribution of $\|A\mathbf{x}\|^2$ tells us how much $f(\mathbf{z})$ varies when $\mathbf{z}$ is approximately $\mathbf{x}_0$, but when $\mathbf{x}$ is standard Gaussian, the moment generating function of the distribution of $\|A\mathbf{x}\|^2$ can be characterized in terms of a Schatten norm generating function.

Suppose that $\mathbf{x}$ is a Gaussian random variable with mean $0$ and identity covariance matrix. Then $\|A\mathbf{x}\|^2=\sigma_1^2 X_1^2+\dots+\sigma_n^2 X_n^2$ where $X_1,\dots,X_n$ are independent standard normally distributed random variables of one variable. Therefore, $$ \begin{split} E\big(\exp(t\|A\mathbf{x}\|^2)\big) & =E\Big(\exp\big(t\sigma(A)_1^2X_1^2+\cdots+t\sigma_n(A)^2X_n^2\big)\Big)\\ & =E\Big(\exp\big(t\sigma_1(A)^2X_1^2)\cdots\exp(t\sigma_n(A)^2X_n^2\big)\Big)\\ & =E\Big(\exp\big(t\sigma_1(A)^2X_1^2\big)\Big)\cdots E\Big(\exp\big(t\sigma_n(A)^2X_n^2\big)\Big)\\ & =\frac{1}{\sqrt{1-2t\sigma_1(A)}}\cdots\frac{1}{\sqrt{1-2t\sigma_n(A)}}. \end{split}$$

We then take logarithms to obtain $$ \begin{split} \ln\Big(E\big(\exp(t\|A\mathbf{x}\|^2\big)\Big) &=-\frac{1}{2}\big(\ln(1-2t\sigma_1(A)^2)+\dots+\ln(1-2t\sigma_n(A)^2)\big)\\ &=\frac{1}{2} \sum_{k=1}^n\sum_{j=1}^\infty\frac{(2t\sigma_k(A)^2)^j}{j}\\ &=\sum_{k=1}^n\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\sigma_k(A)^{2j}\\ &=\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\|A\|_{2j}^{2j}. \end{split}$$

Schatten norms of Hessians at local minima

Suppose that $f:\mathbb{R}^m\rightarrow\mathbb{R}$ is a function that attains a local minimum on the input $\mathbf{x}_0$ and $H(f)(\mathbf{x}_0)=B$. Then $\lim_{\epsilon\rightarrow 0}\frac{1}{\epsilon^2}\cdot(f(\mathbf{x}_0+\epsilon\mathbf{x})-f(\mathbf{x}_0))=\frac{1}{2}\cdot\langle B\mathbf{x},\mathbf{x}\rangle.$ Therefore, the distribution of $\langle B\mathbf{x},\mathbf{x}\rangle$ when $\mathbf{x}$ is a standard Gaussian random variable tells how much, but this distribution is also characterized by a Schatten norm generating function.

Suppose that $B$ is a positive semidefinite matrix and $A=\sqrt{B}$ and $\mathbf{x}$ is still a standard Gaussian random variable. Then since $\langle B\mathbf{x},\mathbf{x}\rangle=\langle A^2\mathbf{x},\mathbf{x}\rangle=\langle A\mathbf{x},A\mathbf{x}\rangle=\|A\mathbf{x}\|^2$, we have $$ \ln\Big(E\big(\exp(t\cdot\langle B\mathbf{x},\mathbf{x}\rangle)\big)\Big)=\sum_{j=1}^\infty\frac{1}{2}\cdot\frac{(2t)^j}{j}\|A\|_{2j}^{2j}=\sum_{j=1}^\infty\frac{1}{2}\frac{(2t)^j}{j}\|B\|_j^j.$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.