Third order matrix differential norm

Question

Suppose we have a function $f:\mathbb{R}^n\to\mathbb{R}$ that is at least three times differentiable. Clearly, there is a relationship between the symmetric trilinear form $$T_1=\nabla^3f(x),$$ and the $n^2 \times n$ matrix $$T_2=\frac{\partial \operatorname{vec}(\nabla^2f(x))}{\partial x^T}.$$ That relationship being a particular reshape operation that can take you from one to the other. In other words, they both contain the same information, but in different forms. According to "Matrix Differential Calculus with Applications in Statistics and Econometrics" (Magnus and Neudecker), at least in my understanding, $T_2$ is the true derivative, as it is the matrix of all third order partial derivatives of $f$. Although, the text I have just cited doesn't consider third-order differentials extensively.

Let's define $\|\cdot\|$ as the norm induced by the vector 2-norm, so that $\|T_2\|$ is the largest singular value of $T_2$, and the following holds for $T_1$: $$\|T_1\|=\max_{h:\|h\|_2\leq 1}|\nabla^3f(x)[h,h,h]|.$$

Now, my question is as follows: is there any connection between the norms of the two operators For example, if $\|T_1\|=m$, then does that imply $\|T_2\|=m$?

Alternatively, would a Lipschitz bound on $\nabla^2f(x)$ provide a bound on the norm of $T_2$ in a similar way as it does for $T_1$? If $\nabla^2f(x)$ is $m$-Lipschitz, then my understanding is that $\|T_1\|\leq m$, but this seems to imply that $T_1$ is the correct derivative.

Answer 1

First, we will denote the $n^2\times n$ matrix of third-order derivatives as $\mathbf{K}$, which has the following structure: $$ \mathbf{K}=\frac{\partial\operatorname{vec}(\mathbf{H})}{\partial\mathbf{x}^T}=\begin{bmatrix} \frac{\partial f}{\partial \mathbf{x}_1\mathbf{x}_1\mathbf{x}_1} & \cdots & \frac{\partial f}{\partial \mathbf{x}_1\mathbf{x}_1\mathbf{x}_{n-1}} & \frac{\partial f}{\partial\mathbf{x}_1\mathbf{x}_1\mathbf{x}_n} \\ \frac{\partial f}{\partial \mathbf{x}_1\mathbf{x}_2\mathbf{x}_1} & \cdots & \frac{\partial f}{\partial \mathbf{x}_1\mathbf{x}_2\mathbf{x}_{n-1}} & \frac{\partial f}{\partial \mathbf{x}_1\mathbf{x}_2\mathbf{x}_n} \\ \vdots & \ddots & \vdots & \vdots \\ \frac{\partial f}{\partial \mathbf{x}_n\mathbf{x}_n\mathbf{x}_1} & \cdots & \frac{\partial f}{\partial\mathbf{x}_n\mathbf{x}_n\mathbf{x}_{n-1}} & \frac{\partial f}{\partial \mathbf{x}_n\mathbf{x}_n\mathbf{x}_n} \end{bmatrix} $$ and we note that the tri-linear operator is defined as follows: $$ \nabla^3f(\mathbf{x})[\mathbf{a},\mathbf{b},\mathbf{c}]=\sum_{i,j,k}^n\frac{\partial f}{\partial\mathbf{x}_i\partial\mathbf{x}_j\partial\mathbf{x}_k}\mathbf{a}_{i}\mathbf{b}_{j}\mathbf{c}_{k} $$

We will show the following relationship holds: $$ \|\nabla^3f(\mathbf{x})\|\leq\|\mathbf{K}\|\leq \sqrt{n}\|\nabla^3f(\mathbf{x})\| $$

Proof:

I claim that $\nabla^3f(\mathbf{x})[\mathbf{a},\mathbf{b},\mathbf{c}]=(\mathbf{a}\otimes \mathbf{b})^T\mathbf{K}\mathbf{c}$ -- that is to say $\mathbf{K}$ can perform the same action as the tri-linear operator. This can be seen by simply writing out the RHS and comparing it to the form of the tri-linear operator. Taking the absolute value and maximizing over all vectors with norm less than 1 gives us the following: $$ \|\nabla^3f(\mathbf{x})\|=\max_{\|\mathbf{a},\mathbf{b},\mathbf{c}\|\leq 1}|\nabla^3f(\mathbf{x})[\mathbf{a},\mathbf{b},\mathbf{c}]|=\max_{\|\mathbf{a},\mathbf{b},\mathbf{c}\|\leq 1}|(\mathbf{a}\otimes \mathbf{b})^T\mathbf{K}\mathbf{c}| $$ This immediately implies the LHS of the bound in question, as the RHS of the preceding equation is bounded above by $\|\mathbf{K}\|$. Now we can bound it below by choosing appropriate vectors. In particular, take $\mathbf{c}$ to be the left-most right singular vector of $\mathbf{K}$, and let $\mathbf{u}$ be the left-most left singular vector of $\mathbf{K}$. For now, we leave $\mathbf{a}$ and $\mathbf{b}$ unspecified. Computing the matrix-vector product $\mathbf{K}\mathbf{c}$ gives the following result: $$ \|\mathbf{K}\|\cdot|(\mathbf{a}\otimes \mathbf{b})^T\mathbf{u}|=|(\mathbf{a}\otimes \mathbf{b})^T\sigma_{1}\mathbf{u}|\leq\|\nabla^3f(\mathbf{x})\| $$ where $\sigma_1$ is the largest singular value of $\mathbf{K}$. Now, we will rewrite the LHS as $\|\mathbf{K}\|\cdot|\mathbf{a}^T\operatorname{mat}(\mathbf{u})\mathbf{b}|$. We know that $\|\operatorname{mat}(\mathbf{u})\|_F=1$ as the vector $\mathbf{u}$ has unit norm. Thus we can bound the largest singular vector of $\operatorname{mat}(\mathbf{u})$ below by $1/\sqrt{n}$.

Then let $\mathbf{b}$ be the left-most right singular vector of $\operatorname{mat}(\mathbf{u})$, and let $\mathbf{a}$ be the left-most left singular vector of the same matrix. We can now write the following: $$ \|\mathbf{K}\|\cdot|\mathbf{a}^T\operatorname{mat}(\mathbf{u})\mathbf{b}|\geq\frac{1}{\sqrt{n}}\|\mathbf{K}\| $$ Rearranging gives us the upper bound of the result in question.