# Why are matrices ubiquitous but hypermatrices rare?

I am puzzled by the amazing utility and therefore ubiquity of two-dimensional matrices in comparison to the relative paucity of multidimensional arrays of numbers, hypermatrices. Of course multidimensional arrays are useful: every programming language supports them, and I often employ them myself. But these uses treat the arrays primarily as convenient data structures rather than as mathematical objects. When I think of the generalization of polygon to $d$-dimensional polytope, or of two-dimensional surface to $n$-dimensional manifold, I see an increase in mathematical importance and utility; whereas with matrices, the opposite.

One answer to my question that I am prepared to acknowledge is that my perception is clouded by ignorance: hypermatrices are just as important, useful, and prevalent in mathematics as 2D matrices. Perhaps tensors, especially when viewed as multilinear maps, fulfill this role. Certainly they play a crucial role in physics, fluid mechanics, Riemannian geometry, and other areas. Perhaps there is a rich spectral theory of hypermatrices, a rich decomposition (LU, QR, Cholesky, etc.) theory of hypermatrices, a rich theory of random hypermatrices—all analogous to corresponding theories of 2D matrices, all of which I am unaware.

I do know that Cayley explored hyperdeterminants in the 19th century, and that Gelfand, Kapranov, and Zelevinsky wrote a book entitled Discriminants, Resultants and Multidimensional Determinants (Birkhäuser, Boston, 1994) about which I know little.

If, despite my ignorance, indeed hypermatrices have found only relatively rare utility in mathematics, I would be interested to know if there is some high-level reason for this, some reason that 2D matrices are inherently more useful than hypermatrices?

I am aware of how amorphous is this question, and apologize if it is considered inappropriate.

• Just a guess, but it may have to do with the difficulty of defining a (canonical) product of hypermatrices; you can't view them naturally as linear maps between vector spaces, and define a product via composition. Dec 2, 2010 at 13:25
• If tensors are ubiquitous, and tensors are hypermatrices, then aren't hypermatrices ubiquitous? (Not a rhetorical question. I can't tell which of these statements you believe less.) Dec 2, 2010 at 14:20
• Here is a related question (related at least in my mind). Why are groups, rings and fields far more ubiquitous than sets $S$ equipped with a function $f:S \times S \times S \to S$ having "nice" properties? Dec 2, 2010 at 15:01
• Also, have you seen hyperdeterminant.wordpress.com ? Dec 2, 2010 at 15:07
• Furthermore, $n$-tuples are immensely more common than matrices... Dec 2, 2010 at 15:16

Note that in linear algebra matrices describe at least two different things: linear maps between vector spaces (we consider only finite-dimensional vector spaces here) and bilinear forms. When thinking of matrices as tensors, linear maps between $V$ and $W$ are elements of the space $V^* \otimes W$, whereas bilinear forms between $V$ and $W$ are elements of $V^* \otimes W^*$. Now you can easily generalize the latter case to more than two spaces, but not the former. But it is the former case where several concepts like composition (matrix multiplication), determinants, eigenvalues etc. apply. (Note that eigenvalues and determinants can be defined for bilinear forms on a vector space equipped with an inner product, but not for bilinear forms on plain vector spaces). Of course you can consider spaces like $V^* \otimes W^* \otimes X$, but elements of this space are better thought as linear maps between $V\otimes W$ and $X$ than as three-dimensional hypermatrices. So what is special about the number 2 is that there is a notion of duality for vector spaces, but no "n-ality".

• Related to this, there is a very fruitful interplay between graphs and matrices, particularly with the use of eigenvalues of adjacency matrices. One can say a few things along these lines about hypergraphs, but they are (to date) much less satisfactory. Dec 2, 2010 at 15:49
• @Florian: Your point about duality is a great insight! Dec 2, 2010 at 16:11
• Recent paper on spectral hypergraph theory, for eigenvalues of adjacency hypermatrices: "Spectra of Hypergraphs", Joshua Cooper, Aaron Dutle, arxiv.org/abs/1106.4856 Sep 2, 2011 at 15:40
• "Note that eigenvalues and determinants can be defined for bilinear forms on a vector space equipped with an inner product, but not for bilinear forms on plain vector spaces" Determinants can be defined for bilinear forms on plain vector spaces. The catch is that it isn't a scalar, but an element of the one-dimensional vector space $(\wedge^nV)^{\otimes2}$. An inner product gives you an isomorphism $(\wedge^nV)^{\otimes2}\rightarrow\mathbb{R}$. This can be generalized to multilinear forms, although getting a scalar requires an orientation as well as an inner product for $(2n+1)$-linear forms. Apr 23, 2018 at 16:36
• I meant $(\wedge^nV^*)^{\otimes2}$, but I can't edit my comment anymore. Apr 23, 2018 at 18:49

An awfully simplistic answer: we work on two-dimensional paper, so two-dimensional matrices are very convenient to write down and compute with, while higher-dimensional hypermatrices are not.

So while we could represent multilinear forms, tensors, etc. as hypermatrices, we often don’t, because doing so is not nearly as fruitful as representing linear maps, bilinear forms etc. as matrices. Instead, we usually use other notations when working with higher tensors by hand.

In computer algebra, the dimension of the paper is not significant, while some kinds of abstraction are harder, so in this context, higher tensors are much more often represented as hypermatrices.

• I really think this is the answer. Dec 2, 2010 at 18:22
• And the fact that we are speaking of this now, when computers make it easier to treat hypermatrices, seems to me a sort of confirm of ypur thesis. Dec 2, 2010 at 19:34
• I do not think this is the answer. Dec 6, 2012 at 20:07
• I'm sure that the comparative awkwardness of notation for hypermatrices at least partly explains their lack of popularity amongst mathematicians, and lack of popularity leads to lack of theory. So I think this answer is partly correct, but it needs to be combined with answers like Florian's that expose genuine features that differ for $D=2$ and $D>2$. Dec 6, 2012 at 22:37

To such a complex problem, there cannot be a unique answer. I see many, which all justify the tremendous interest that mathematicians have devoted so far to matrices, rather than to hypermatrices.

Ubiquity. Matrices are used by every species of mathematicians, and beyond, by a large fraction of scientists. This is perhaps the only mathematical area to enjoy this versatility. Let me provide a few examples. Matrix exponential is fundamental in differential equations (more generally in dynamical systems) and Lie theory of groups. Symmetric matrices are used in quantum mechanics, statistics, optimisation and numerical analysis; they have deep relations with representation theory and combinatorics (see the solution of Horn's conjecture by Tao & Knutson). Positive matrices are encountered in probability and numerical analysis (discrete maximum principle). Matrix groups are used in representation theory, in number theory (including modular forms), in dynamical systems (because of symmetries). When depending on parameters, matrices enter in PDE theory as symbols.

Simplicity. The concept of matrix is by definition simpler than that of hypermatrices. It is natural that the study of matrices precedes that of HM. This argument will fade as time increases, of course.

Richness. What makes a field particularly attractive is that it involves several apparently unrelated concepts in order to produce unexpected results. This happens in matrix theory, because on the one hand, we may view them as linear maps (where conjugation is relevant) and on the other hand we may see them as bilinear or sesquilinear maps (where congruence is relevant). It becomes especially fruitful when we go back and forth between both points of view. This happens in the remarkable theorem that normal matrices are unitarily diagonalizable, but also in the parametrization of a Lie group by its Lie algebra via the exponential and the Hermitian square root. I am not at all aware of the theory of HM, but if they do not form naturally an algebra, I doubt that their theory could be so rich, or if it is, it will be for completely different mathematical reasons.

To temperate this pledge, let me say that hypermatrices have been studied (although not so deeply) under the name tensors. They are of great importance in differential geometry (Ricci curvature tensor, with the many identities named after Christoffel, Gauss, Codazzi, ...) and in its applications: general relativity, elasticity. These are undoubtedly difficult topics, where even simple problems are not well understood. To mention one of them, there is still no satisfactory description of the twice-symmetric tensors of fourth order ($a_{ijkl}=a_{jikl}=a_{ijlk}$) that satisfy the Legendre-Hadamard condition $$\sum_{i,j,k,l}a_{ijkl}x_ix_j\xi_k\xi_l\ge0,\qquad\forall x\in\mathbb R^n,\xi\in\mathbb R^d.$$ It seems to me that the use of HM is too scattered, and therefore there is no research community specializing on all their aspects. Edit. Likewise, the notion of rank, although correctly defined in the case of tensors, is hard to manipulate and to compute explicitly. This is the reason why the exact algorithmic complexity of the multiplication of matrices is still not known (the operation $(A,B)\mapsto AB$ in $M_n(k)$ may be viewed as a $3$-tensor, and its tensorial rank governs the number of operations needed in an $n\times n$ mulitplication).

• @Denis: I could not ask for a more knowledgeable and informative answer. Your points about richness are especially enlightening. I am grateful! Dec 2, 2010 at 15:25
• The Legendre-Hadamard condition is a great example of what Gjergji asked in the comments above, one where the multilinear setting is much less understood than the linear case. Whereas it is well known that the convex cone generated by $\xi\otimes\xi$ "squares of vectors" in the space of square matrices is equivalent to the cone of positive semi-definite matrices, and as such that PSD cone is self-dual, an analogous statement is known to be false for Legendre-Hadamard tensors. Dec 2, 2010 at 15:33
• In particular, it'd be great to find out what is the difference set between the Legendre-Hadamard tensors and the rank-one cone generated by elements of the form $x\otimes x\otimes\xi\otimes\xi$, or even find a self-dual cone sitting between the two convex cones. Dec 2, 2010 at 15:35

I think much of what I'm about to say has been said already, but I wanted to repeat it: I don't agree with the premise of the question at all. A "hypermatrix" is simply another name for a tensor. All aspects of matrix operations that I know (multiplication, determinant, etc.) have direct generalizations to tensors. All of this is best developed and understood in the abstract setting using tensor and exterior algebras over abstract vector spaces, as well as representation theory. Moreover, tensors along with these operations are used in many settings, including but not restricted to algebraic and differential geometry. What is true is that you don't necessarily have the depth of theorems that you have for matrices, but my view is that is because of two reasons: 1) Tensors are more general, so they don't satisfy all of the properties of matrices, and 2) Tensors are more complicated, so it is takes longer to develop them to the same depth.

• This is pretty much exactly the answer I would have given. I'd add one more reason: 3) it's harder to write the data of a tensor on a chalkboard or piece of paper. Dec 2, 2010 at 21:58
• Theo, an excellent point! Dec 2, 2010 at 22:29
• Theo, I just said a week ago to colleagues of mine (partly joking, but not entirely) that I specialized on matrices because I can write them on the blackboard. Dec 3, 2010 at 3:57
• I entirely agree with you, and I think Theo makes a very good point. I don't think I've ever seen a concrete tensor... Dec 6, 2012 at 21:44
• "All aspects of matrix operations that I know (multiplication, determinant, etc.) have direct generalizations to tensors." How do you propose to define multiplication of tensors in such a way as to generalize multiplication of matrices? Apr 21, 2016 at 6:00

Bhargava explained Gauss composition of binary quadratic forms using 2x2x2-cubes, which can be identified with 2x2x2-hypermatrices on which $SL_2({\mathbb Z})^3$ acts naturally; the hyperdeterminant of this hypermatrix is the common discriminant of the three associated quadratic forms. Already Cayley realized the connection with composition of binary quadratic forms.

One reason linear algebra is so useful is that the basic notions, like rank, have so many equivalent definitions. Some are better for formulating problems, some for proving theorems, and some for doing computations. The ability to freely move between these is the key to solving many problems.

Some of these definitions of course won't make sense for hypermatrices, but many of them do. The problem is that they don't usually end up being equivalent.

For example: you can define rank one hypermatrices as outer products of vectors ("simple tensors") and define the minimum number of such terms which must be summed to yield a given hypermatrix to be the "hyperrank". But this does not properly classify all hypermatrices up to changes of basis along all the "axes" of the hypermatrix as it does for matrices (in fact the number of equivalence classes is no longer even finite). And except in very simple cases this does not agree with what you'd get by the vanishing of certain "hyperdeterminants" -- indeed, the set of hypermatrices with hyperrank at most $k$ isn't even closed.

So the things you can compute aren't the same as the things you'd like to compute, and everything ends up feeling much more ad hoc.

Of course this complexity makes for a lot of interesting things to study, but not a simple widely applicable tool every undergrad should learn. Though perhaps they should learn why they don't learn it!

Although the following paper: Most tensor problems are NP Hard by Hillar and Lim does not explain why tensors are not so ubiquitous, it suggests that they might continue remaining non-ubiquitous. There seem to be no easy generalizations of standard notions in the matrix case: eigenvalues, singular value, spectral norm, etc., are shown two be NP-Hard to compute, even for 3-D tensors.

• IIRC the jump in tensor complexity from 2 to 3 dimensions has a relation to the same jump in the Boolean satisfiability problem, from 2-SAT to 3-SAT. Sep 17 at 17:52

It seems to me is that there are a lot of things in mathematics that one could call hypermatrices if one were so inclined, but which people generally don't (and if they call them anything, they call them tensors). For example, one use of matrices $M$ is that they represent bilinear forms $x^T M y$ and quadratic forms $x^T M x$. Three-dimensional hypermatrices then represent trilinear forms and cubic forms (and so forth for higher dimensions), and these do appear in various places in mathematics, for example in Lie theory. The norm map on a cubic number field is also an example of a cubic form. More generally, alternating multilinear forms appear as differential forms, and the same can be said about more general types of tensors. It could be said that studying a projective hypersurface defined by a homogeneous polynomial $f(x_1, ... x_n) = 0$ is the same as studying a certain $n$-dimensional hypermatrix associated to $f$. It could be said that studying a finite-dimensional algebra or Lie algebra $A$ is the same as studying the hypermatrix giving the structure constants of its multiplication or bracket $m : A \times A \to A$.

So, as I said in the comments, I'm not sure what you mean when you say that hypermatrices are rare. I suppose you are trying to draw a distinction between basis-dependent and basis-independent ideas?

• When you mention the connection between trilinear and cubic forms, and Lie theory, I think of the realisation of certain groups as stabilisers of degree-$3$ tensors, and of triality $\mathsf D_4$. Is that what you had in mind? Sep 18 at 0:32
• I wrote this long enough ago that I don't know what I had in mind; maybe the realization of Lie groups as stabilizers of $3$-forms, maybe the Cartan $3$-form $K([x, y], z)$. Sep 18 at 2:29

As others have pointed out, higher-order tensors (i.e., hypermatrices) are in fact ubiquitous in mathematics, but they often aren’t discussed in detail as such because there’s not a lot you can say about them in general. Matrices are far more tractable, so there’s more you can deduce from linear algebra once you identify a matrix.

The key issue is how big the groups acting on these objects are. If you have an $$n$$-by-$$n$$ matrix, it generally has a roughly $$n^2$$-dimensional group acting on it. (The group in question depends on what the matrix represents.) This means the group size is comparable to the total number of degrees of freedom in the matrix, so we can expect substantial simplifications via the group action. This is why various canonical forms are so simple in comparison with general matrices. For example, row echelon form, rational canonical form, Jordan canonical form, Sylvester’s law of inertia: in each case a matrix can be described using at most a linear number of free parameters, because the relevant symmetry group is nearly as large as the space of matrices.

By contrast, an $$n$$-by-$$n$$-by-$$n$$ hypermatrix doesn’t get a substantially bigger group: you still end up with $$O(n^2)$$ dimensions, through matrix groups acting on two coordinates at a time. Because there are $$n^3$$ degrees of freedom total, there’s no hope of simplification. (For example, if we are describing the multiplication map in an algebra, there are a cubic number of structure constants describing the map, but change of basis lets you control only a quadratic number of degrees of freedom.) No matter how you apply symmetries, you cannot expect to reach any canonical form that’s genuinely simpler than the original hypermatrix.

The upshot is that matrices can often be classified using canonical forms, but hypermatrices almost never can be. That’s why researchers are happy to see matrices but find hypermatrices frustrating.

• Yes, I also think the small-ness of orbits, a.k.a. simplicity of canonical forms, is a large element here. Nov 15, 2020 at 23:07

Since a matrix is just how we write down a linear map $V\to W$ from one vector space to another, it seems to me that the prevalance of matrices over hypermatrices is just a reflection of the fact that we use categories so much more often than multicategories (where a morphism has a list of objects as its domain). And I feel that the large role categories play, with morphisms that just go from one object to another, is due to the way we look at the world in terms of states and processes, of where you are now and how to get where you're going, of being and becoming.

Of course, by using duals of vector spaces we can also use matrices to represent either functionals $V\otimes W\to k$ or elements $k\to V\otimes W$, but I feel that these uses are usually no more special than their generalizations for hypermatrices.

I think that the difference between matrices and hypermatrices is closely related to the difference between graphs and hypergraphs.

1) Many of the miracles for graphs/matrices (characterization, duality, efficient algorithms) do not extend to high dimensions. Gauss elimination and the efficincy of computing determinants is at the roots of some of these miracles.

2) For modeling, in many cases graphs /matrices suffices. You can model a hypergraph easily using a graphs.

• Probably you are using 'modelling' in a technical sense, but, at least in an informal sense, I'm not sure that (2) is convincing. You can model graphs easily using sets, but we nonetheless study graph theory on its own, rather than as a special case of set theory. Sep 17 at 18:44

This answer is relatively simpleton but I think it gets to the heart of the matter.

Groups are very ubiquitious in mathematics. More so than non binary n-ary groups.

Wherever there are/can be groups used there is a way to use matrices (via representation theory) and vice versa wherever you see matrices acting/relevant in a theory, there is a group action hidden there.

Hypermatrices do NOT lend themselves to a natural binary operation but rather an n-ary operation with some kind of unknown associativity. These are objects we currently ALMOST never use/think in terms of, and thus we almost never find ourselves using hypermatrices/discovering unexpected hypermatrix behavior for the same reasons.

There are many reasons for this. One already mentioned by others is simplicity. The simpler a structure is, the more often you will encounter it. Abelian groups are everywhere ; non-abelian groups already more rare, rings still less frequent, etc. Matrices fit into many basic schemes and structures: Groups, rings, vector spaces, algebras.

The second very important reason is that math is used in science to describe nature, and basic laws in nature are simple, at least in the first approximation which is most often sufficient in applications. Most laws of nature are linear, more so, the relations which are most used. Functions or expressions that depend on more than one variable are so much less frequently used than those which just express proportionality. Matrices correspond to linear relations, proportionality, and yet they even encompass also nearly all (et least the most frequent) cases of bilinear relations (when the result is a scalar). Hypermatrices correspond to bilinear vector-valued, or trilinear and higher order relations, which are much less frequent in nature and use. Do you know any popular laws of that nature? See!

• "Do you know any popular laws of that nature?" Yes, Hooke's Law is a linear relationship between stress (a 2-tensor) and strain (another 2-tensor) and the 'stiffness tensor' has rank 4. Apr 24, 2018 at 14:56

Hypermatrices are as ubiquitous and as important as matrices. Take for example the Riemann tensor, $$R^a_{ijk}$$ or the Weyl tensor $$C^a_{ijk}$$. This involves the notion of a manifold and hence that of non-Euclidean geometry, a hugely significant advance in maths and physics - the standard model in both particle and physics involves both.

Now whilst these are hypermatrices - they are also not. What I mean by this is that the definition of a tensor stipulates it must be covariant. Hence a tensor is a family of hypermatrices indexed by the set of all frames and satisfying covariance. A tensor is not a single hypermatrix.

In this sense differential forms are also hypermatrices too. And these are ubiquitous in maths and physics.

Generally, we teach matrices first before we teach their covariant version, linear algebra. This suggests we ought to teach hypermatrices before teach 'hyper' linear algebra, that is linear algebra using tensors. It's well known that a linear transformation, $$V \rightarrow V \in Vect$$ between a fixed vector space in a fixed basis yields a matrix; likewise, a linear transformation between tensor spaces on a fixed vector space yields a hypermatrix, $$\otimes^p V \rightarrow \otimes^q V \in Vect^{\otimes}$$. This might help make the step between matrix mathematics and linear algebra to that of tensors significantly easier to understand for students - I certainly recall being confused by the definition of a tensor field as it is explained in GR texts when I first came across it simply because I expected an invariant definition in the way vectors are presented invariantly. Instead, they are presented covariantly.

An invariant definition is manifestly covariant but a covariant definition - despite the name - is usually not.

However an invariant definition of tensor fields is possible, they are sections of vector bundle maps $$\otimes^p TM \rightarrow \otimes^q TM \in VBun^{\otimes}M$$ and this is manifestly - just look at the definitions - the definition of a tensor lifted to a manifold.

There is also a recent book, whose title and authors I fail to recall bit I do recall their ethnicity - they were either Chinese or had Chinese heritage. They extensively list analogues of matrix operations for hypermatrices. They also came up with an intriguing terminological innovation: they suggest calling a hypermatrix a tentrix in analogy to matrix and keeping the term tensor solely for the invariant notion.

Hi Joseph,

it seems that the basic difference between hypermatrices and matrices is the indexation.

Let me explain what I mean. Consider a finite graph $G=(V,E)$ (the graph structure is not important I mentioning this just to refer to some geometrical interpretation of the index) and a collection of numbers $$(A_{v,w})_{(v,w)\in V\times V}.$$

Despite this list have a "two dimensional" character (bi-indexed) it is in fact a hypermatrice.

The way to import to hypermatrices the results of standard theory of matrix is to define the operations as usual. The product, for instance, is $$(AB)_{(v,w)}=\sum_{z\in V}A_{v,z}B_{z,w}.$$

The determinant and other important objects can also be defined in this way by replacing the group $\mathbb{S}_n$ by automorphisms of $V$ and so on.

So I guess that do not pop up any interesting feature that justify to give a great attention to this object. Because one can see them just as a replacement of the indexation process.

For the other hand in fields like statistical mechanics and percolation they are much more natural in high dimensional problems than the usual matrices. When we define $p_{uv}$ the probability to see the edge $\{u,v\}$ open or when we are leading with the coupling constants $J_{i,j}$ in the Ising model, all of them are hypermatrices and its linear algebra structure are frequently arise in proofs of important results in correlations inequalities.