1
$\begingroup$

Say I have a function $L : (W_1,..,W_{H+1}) \rightarrow \mathbb{R}$ i.e it takes a tuple of $n$ matrices of different dimensions and computes a number from them.

Then I see being defined a particular kind of a matrix called the ``Hessian" of $L$ (say $\nabla^2 L$) which is a matrix of $(H+1)\times (H+1)$ submatrices. And the $(i,j)^{th}$ submatrix is defined as, $(\nabla^2 L)_{ij} = D_{vec(W^T_{H+2-j})} [ (D_{vec(W^T_{H+2-i})}L )^T ] $.

So if $W^T_{H+2-i}$ is $a \times b$ dimensional then its vectorization is a $ab$ dimensional column vector and hence $(D_{vec(W^T_{H+2-i})}L )^T$ is an $ab$ dimensional row vector. If $W^T_{H+2-j}$ is $c \times d$ dimensional then its vectorization is $cd$ dimensional column vector. Then $(\nabla^2 L)_{ij}$ becomes a $cd \times ab$ sized matrix of first derivatives of the components of $(D_{vec(W^T_{H+2-i})}L )^T$ w.r.t the components of $W^T_{H+2-j}$.

  • Can someone help motivate why this matrix is the correct generalization of the notion of a Hessian which is usually defined as the matrix of all double derivaties of a function mapping $\mathbb{R}^n \rightarrow \mathbb{R}$?

  • Any literature reference which helps parse this would be helpful!


  • One could have vectorized each of the $W$ matrices and concatenated the tuple $(W_1,..,W_n)$ (say in the sequence as in this tuple) into a single large vector and then taken the usual Hessian of the function w.r.t this new big vector. But that would not have produced this matrix.

  • It seems to me that papers are defining this function $L$ to be "convex" or "concave" depending on whether this above Hessian matrix is p.s.d or n.s.d This looks a bit tricky because I am not aware of any separate notion of convexity or concavity of functions whose domain is a product of Euclidean spaces.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.