(Posted this on math.stackexchange and cross.correlated over more than a week ago, but didn't get an answer, and this is a question in my research so this seems like it might have been the better place to ask...)

I'm wondering if there exists any higher-order SVD for dimensionality reduction. Note that I do not mean multilinear PCA, which operates on data tensors, but some form of SVD which can produce, say, a quadratic approximation of a dataset.

Intuitively, SVD takes in a set of vectors as a datset, and computes the ranked eigenvalues and eigenvectors which correspond to a reduced subspace and a "weight" representing their importance. Given a decomposition:

$$A = U\Sigma V^T$$

Where $A$ is a matrix where the columns are data vectors, the truncated matrix $U$ (keeping only the leftmost $n$ columns; call this truncated version $\mathcal{U}$) provides a reduced approximate basis for the original dataset $A$. However, this approximation is purely linear. It can thus be thought of as a linear approximation of the data on some $n$ reduced variable set. For a particular column of $a_i$ of $A$, $a_i \approx \mathcal{U} x_i$ for some dimensionality-reduced vector $x_i$.

I am wondering if it is possible to do something akin to a Taylor expansion here, and recover a higher-order approximation of the data. For example, for a quadratic system of order $n$, I'd like to generate the $\mathcal{U}$ matrix, but also a third-order tensor $\mathcal{W}$ such that a "quadratic" approximation of the data could be written, as, say:

$a_i \approx \mathcal{U} x_i + \frac{1}{2} x_i^T (\mathcal{W} ~ \vdots ~ x_i)$

where $\vdots$ is a tensor-vector contraction. (This is a form I made up now for discussion's sake; it is possible that the true form of some quadratic approximation is slightly different). Here, the best $x_i$ the quadratic approximation will be better than the linear one.

Ideally, I'd be searching for some general theory which would allow for arbitrary order approximation.

Is there any general established method for this (or something similar)? If so, is it tractable? If not, is there a mathematically grounded reason why not?

assumethat they span a linear space in some sense. $\endgroup$ – Dirk Aug 6 '18 at 10:143more comments