Say I have a function $L : (W_1,..,W_{H+1}) \rightarrow \mathbb{R}$ i.e it takes a tuple of $n$ matrices of different dimensions and computes a number from them.

Then I see being defined a particular kind of a matrix called the ``Hessian" of $L$ (say $\nabla^2 L$) which is a matrix of $(H+1)\times (H+1)$ submatrices. And the $(i,j)^{th}$ submatrix is defined as, $(\nabla^2 L)_{ij} = D_{vec(W^T_{H+2-j})} [ (D_{vec(W^T_{H+2-i})}L )^T ] $.

So if $W^T_{H+2-i}$ is $a \times b$ dimensional then its vectorization is a $ab$ dimensional column vector and hence $(D_{vec(W^T_{H+2-i})}L )^T$ is an $ab$ dimensional row vector. If $W^T_{H+2-j}$ is $c \times d$ dimensional then its vectorization is $cd$ dimensional column vector. Then $(\nabla^2 L)_{ij}$ becomes a $cd \times ab$ sized matrix of first derivatives of the components of $(D_{vec(W^T_{H+2-i})}L )^T$ w.r.t the components of $W^T_{H+2-j}$.

  • Can someone help motivate why this matrix is the correct generalization of the notion of a Hessian which is usually defined as the matrix of all double derivaties of a function mapping $\mathbb{R}^n \rightarrow \mathbb{R}$?

  • Any literature reference which helps parse this would be helpful!

  • One could have vectorized each of the $W$ matrices and concatenated the tuple $(W_1,..,W_n)$ (say in the sequence as in this tuple) into a single large vector and then taken the usual Hessian of the function w.r.t this new big vector. But that would not have produced this matrix.

  • It seems to me that papers are defining this function $L$ to be "convex" or "concave" depending on whether this above Hessian matrix is p.s.d or n.s.d This looks a bit tricky because I am not aware of any separate notion of convexity or concavity of functions whose domain is a product of Euclidean spaces.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.