Why are matrices ubiquitous but hypermatrices rare?

Question

I am puzzled by the amazing utility and therefore ubiquity of two-dimensional matrices in comparison to the relative paucity of multidimensional arrays of numbers, hypermatrices. Of course multidimensional arrays are useful: every programming language supports them, and I often employ them myself. But these uses treat the arrays primarily as convenient data structures rather than as mathematical objects. When I think of the generalization of polygon to $d$-dimensional polytope, or of two-dimensional surface to $n$-dimensional manifold, I see an increase in mathematical importance and utility; whereas with matrices, the opposite.

One answer to my question that I am prepared to acknowledge is that my perception is clouded by ignorance: hypermatrices are just as important, useful, and prevalent in mathematics as 2D matrices. Perhaps tensors, especially when viewed as multilinear maps, fulfill this role. Certainly they play a crucial role in physics, fluid mechanics, Riemannian geometry, and other areas. Perhaps there is a rich spectral theory of hypermatrices, a rich decomposition (LU, QR, Cholesky, etc.) theory of hypermatrices, a rich theory of random hypermatrices—all analogous to corresponding theories of 2D matrices, all of which I am unaware.

I do know that Cayley explored hyperdeterminants in the 19th century, and that Gelfand, Kapranov, and Zelevinsky wrote a book entitled Discriminants, Resultants and Multidimensional Determinants (Birkhäuser, Boston, 1994) about which I know little.

If, despite my ignorance, indeed hypermatrices have found only relatively rare utility in mathematics, I would be interested to know if there is some high-level reason for this, some reason that 2D matrices are inherently more useful than hypermatrices?

I am aware of how amorphous is this question, and apologize if it is considered inappropriate.

Answer 1

Note that in linear algebra matrices describe at least two different things: linear maps between vector spaces (we consider only finite-dimensional vector spaces here) and bilinear forms. When thinking of matrices as tensors, linear maps between $V$ and $W$ are elements of the space $V^* \otimes W$, whereas bilinear forms between $V$ and $W$ are elements of $V^* \otimes W^* $. Now you can easily generalize the latter case to more than two spaces, but not the former. But it is the former case where several concepts like composition (matrix multiplication), determinants, eigenvalues etc. apply. (Note that eigenvalues and determinants can be defined for bilinear forms on a vector space equipped with an inner product, but not for bilinear forms on plain vector spaces). Of course you can consider spaces like $V^* \otimes W^* \otimes X$, but elements of this space are better thought as linear maps between $V\otimes W$ and $X$ than as three-dimensional hypermatrices. So what is special about the number 2 is that there is a notion of duality for vector spaces, but no "n-ality".

Answer 2

An awfully simplistic answer: we work on two-dimensional paper, so two-dimensional matrices are very convenient to write down and compute with, while higher-dimensional hypermatrices are not.

So while we could represent multilinear forms, tensors, etc. as hypermatrices, we often don’t, because doing so is not nearly as fruitful as representing linear maps, bilinear forms etc. as matrices. Instead, we usually use other notations when working with higher tensors by hand.

In computer algebra, the dimension of the paper is not significant, while some kinds of abstraction are harder, so in this context, higher tensors are much more often represented as hypermatrices.

Answer 3

To such a complex problem, there cannot be a unique answer. I see many, which all justify the tremendous interest that mathematicians have devoted so far to matrices, rather than to hypermatrices.

Ubiquity. Matrices are used by every species of mathematicians, and beyond, by a large fraction of scientists. This is perhaps the only mathematical area to enjoy this versatility. Let me provide a few examples. Matrix exponential is fundamental in differential equations (more generally in dynamical systems) and Lie theory of groups. Symmetric matrices are used in quantum mechanics, statistics, optimisation and numerical analysis; they have deep relations with representation theory and combinatorics (see the solution of Horn's conjecture by Tao & Knutson). Positive matrices are encountered in probability and numerical analysis (discrete maximum principle). Matrix groups are used in representation theory, in number theory (including modular forms), in dynamical systems (because of symmetries). When depending on parameters, matrices enter in PDE theory as symbols.

Simplicity. The concept of matrix is by definition simpler than that of hypermatrices. It is natural that the study of matrices precedes that of HM. This argument will fade as time increases, of course.

Richness. What makes a field particularly attractive is that it involves several apparently unrelated concepts in order to produce unexpected results. This happens in matrix theory, because on the one hand, we may view them as linear maps (where conjugation is relevant) and on the other hand we may see them as bilinear or sesquilinear maps (where congruence is relevant). It becomes especially fruitful when we go back and forth between both points of view. This happens in the remarkable theorem that normal matrices are unitarily diagonalizable, but also in the parametrization of a Lie group by its Lie algebra via the exponential and the Hermitian square root. I am not at all aware of the theory of HM, but if they do not form naturally an algebra, I doubt that their theory could be so rich, or if it is, it will be for completely different mathematical reasons.

To temperate this pledge, let me say that hypermatrices have been studied (although not so deeply) under the name tensors. They are of great importance in differential geometry (Ricci curvature tensor, with the many identities named after Christoffel, Gauss, Codazzi, ...) and in its applications: general relativity, elasticity. These are undoubtedly difficult topics, where even simple problems are not well understood. To mention one of them, there is still no satisfactory description of the twice-symmetric tensors of fourth order ($a_{ijkl}=a_{jikl}=a_{ijlk}$) that satisfy the Legendre-Hadamard condition $$\sum_{i,j,k,l}a_{ijkl}x_ix_j\xi_k\xi_l\ge0,\qquad\forall x\in\mathbb R^n,\xi\in\mathbb R^d.$$ It seems to me that the use of HM is too scattered, and therefore there is no research community specializing on all their aspects. Edit. Likewise, the notion of rank, although correctly defined in the case of tensors, is hard to manipulate and to compute explicitly. This is the reason why the exact algorithmic complexity of the multiplication of matrices is still not known (the operation $(A,B)\mapsto AB$ in $M_n(k)$ may be viewed as a $3$-tensor, and its tensorial rank governs the number of operations needed in an $n\times n$ mulitplication).

Answer 4

I think much of what I'm about to say has been said already, but I wanted to repeat it: I don't agree with the premise of the question at all. A "hypermatrix" is simply another name for a tensor. All aspects of matrix operations that I know (multiplication, determinant, etc.) have direct generalizations to tensors. All of this is best developed and understood in the abstract setting using tensor and exterior algebras over abstract vector spaces, as well as representation theory. Moreover, tensors along with these operations are used in many settings, including but not restricted to algebraic and differential geometry. What is true is that you don't necessarily have the depth of theorems that you have for matrices, but my view is that is because of two reasons: 1) Tensors are more general, so they don't satisfy all of the properties of matrices, and 2) Tensors are more complicated, so it is takes longer to develop them to the same depth.

Answer 5

Bhargava explained Gauss composition of binary quadratic forms using 2x2x2-cubes, which can be identified with 2x2x2-hypermatrices on which $SL_2({\mathbb Z})^3$ acts naturally; the hyperdeterminant of this hypermatrix is the common discriminant of the three associated quadratic forms. Already Cayley realized the connection with composition of binary quadratic forms.

Answer 6

One reason linear algebra is so useful is that the basic notions, like rank, have so many equivalent definitions. Some are better for formulating problems, some for proving theorems, and some for doing computations. The ability to freely move between these is the key to solving many problems.

Some of these definitions of course won't make sense for hypermatrices, but many of them do. The problem is that they don't usually end up being equivalent.

For example: you can define rank one hypermatrices as outer products of vectors ("simple tensors") and define the minimum number of such terms which must be summed to yield a given hypermatrix to be the "hyperrank". But this does not properly classify all hypermatrices up to changes of basis along all the "axes" of the hypermatrix as it does for matrices (in fact the number of equivalence classes is no longer even finite). And except in very simple cases this does not agree with what you'd get by the vanishing of certain "hyperdeterminants" -- indeed, the set of hypermatrices with hyperrank at most $k$ isn't even closed.

So the things you can compute aren't the same as the things you'd like to compute, and everything ends up feeling much more ad hoc.

Of course this complexity makes for a lot of interesting things to study, but not a simple widely applicable tool every undergrad should learn. Though perhaps they should learn why they don't learn it!

Answer 7

Just a small addition to the already nice answers above.

Although the following paper: Most tensor problems are NP Hard by Hillar and Lim does not explain why tensors are not so ubiquitous, it suggests that they might continue remaining non-ubiquitous. There seem to be no easy generalizations of standard notions in the matrix case: eigenvalues, singular value, spectral norm, etc., are shown two be NP-Hard to compute, even for 3-D tensors.

Answer 8

It seems to me is that there are a lot of things in mathematics that one could call hypermatrices if one were so inclined, but which people generally don't (and if they call them anything, they call them tensors). For example, one use of matrices $M$ is that they represent bilinear forms $x^T M y$ and quadratic forms $x^T M x$. Three-dimensional hypermatrices then represent trilinear forms and cubic forms (and so forth for higher dimensions), and these do appear in various places in mathematics, for example in Lie theory. The norm map on a cubic number field is also an example of a cubic form. More generally, alternating multilinear forms appear as differential forms, and the same can be said about more general types of tensors. It could be said that studying a projective hypersurface defined by a homogeneous polynomial $f(x_1, ... x_n) = 0$ is the same as studying a certain $n$-dimensional hypermatrix associated to $f$. It could be said that studying a finite-dimensional algebra or Lie algebra $A$ is the same as studying the hypermatrix giving the structure constants of its multiplication or bracket $m : A \times A \to A$.

So, as I said in the comments, I'm not sure what you mean when you say that hypermatrices are rare. I suppose you are trying to draw a distinction between basis-dependent and basis-independent ideas?

Answer 9

As others have pointed out, higher-order tensors (i.e., hypermatrices) are in fact ubiquitous in mathematics, but they often aren’t discussed in detail as such because there’s not a lot you can say about them in general. Matrices are far more tractable, so there’s more you can deduce from linear algebra once you identify a matrix.

The key issue is how big the groups acting on these objects are. If you have an $n$-by-$n$ matrix, it generally has a roughly $n^2$-dimensional group acting on it. (The group in question depends on what the matrix represents.) This means the group size is comparable to the total number of degrees of freedom in the matrix, so we can expect substantial simplifications via the group action. This is why various canonical forms are so simple in comparison with general matrices. For example, row echelon form, rational canonical form, Jordan canonical form, Sylvester’s law of inertia: in each case a matrix can be described using at most a linear number of free parameters, because the relevant symmetry group is nearly as large as the space of matrices.

By contrast, an $n$-by-$n$-by-$n$ hypermatrix doesn’t get a substantially bigger group: you still end up with $O(n^2)$ dimensions, through matrix groups acting on two coordinates at a time. Because there are $n^3$ degrees of freedom total, there’s no hope of simplification. (For example, if we are describing the multiplication map in an algebra, there are a cubic number of structure constants describing the map, but change of basis lets you control only a quadratic number of degrees of freedom.) No matter how you apply symmetries, you cannot expect to reach any canonical form that’s genuinely simpler than the original hypermatrix.

The upshot is that matrices can often be classified using canonical forms, but hypermatrices almost never can be. That’s why researchers are happy to see matrices but find hypermatrices frustrating.

Answer 10

Since a matrix is just how we write down a linear map $V\to W$ from one vector space to another, it seems to me that the prevalance of matrices over hypermatrices is just a reflection of the fact that we use categories so much more often than multicategories (where a morphism has a list of objects as its domain). And I feel that the large role categories play, with morphisms that just go from one object to another, is due to the way we look at the world in terms of states and processes, of where you are now and how to get where you're going, of being and becoming.

Of course, by using duals of vector spaces we can also use matrices to represent either functionals $V\otimes W\to k$ or elements $k\to V\otimes W$, but I feel that these uses are usually no more special than their generalizations for hypermatrices.

Answer 11

I think that the difference between matrices and hypermatrices is closely related to the difference between graphs and hypergraphs.

1) Many of the miracles for graphs/matrices (characterization, duality, efficient algorithms) do not extend to high dimensions. Gauss elimination and the efficincy of computing determinants is at the roots of some of these miracles.

2) For modeling, in many cases graphs /matrices suffices. You can model a hypergraph easily using a graphs.

Answer 12

This answer is relatively simpleton but I think it gets to the heart of the matter.

Groups are very ubiquitious in mathematics. More so than non binary n-ary groups.

Wherever there are/can be groups used there is a way to use matrices (via representation theory) and vice versa wherever you see matrices acting/relevant in a theory, there is a group action hidden there.

Hypermatrices do NOT lend themselves to a natural binary operation but rather an n-ary operation with some kind of unknown associativity. These are objects we currently ALMOST never use/think in terms of, and thus we almost never find ourselves using hypermatrices/discovering unexpected hypermatrix behavior for the same reasons.

Answer 13

There are many reasons for this. One already mentioned by others is simplicity. The simpler a structure is, the more often you will encounter it. Abelian groups are everywhere ; non-abelian groups already more rare, rings still less frequent, etc. Matrices fit into many basic schemes and structures: Groups, rings, vector spaces, algebras.

The second very important reason is that math is used in science to describe nature, and basic laws in nature are simple, at least in the first approximation which is most often sufficient in applications. Most laws of nature are linear, more so, the relations which are most used. Functions or expressions that depend on more than one variable are so much less frequently used than those which just express proportionality. Matrices correspond to linear relations, proportionality, and yet they even encompass also nearly all (et least the most frequent) cases of bilinear relations (when the result is a scalar). Hypermatrices correspond to bilinear vector-valued, or trilinear and higher order relations, which are much less frequent in nature and use. Do you know any popular laws of that nature? See!

Answer 14

Hypermatrices are as ubiquitous and as important as matrices. Take for example the Riemann tensor, $R^a_{ijk}$ or the Weyl tensor $C^a_{ijk}$. This involves the notion of a manifold and hence that of non-Euclidean geometry, a hugely significant advance in maths and physics - the standard model in both particle and physics involves both.

Now whilst these are hypermatrices - they are also not. What I mean by this is that the definition of a tensor stipulates it must be covariant. Hence a tensor is a family of hypermatrices indexed by the set of all frames and satisfying covariance. A tensor is not a single hypermatrix.

In this sense differential forms are also hypermatrices too. And these are ubiquitous in maths and physics.

Generally, we teach matrices first before we teach their covariant version, linear algebra. This suggests we ought to teach hypermatrices before teach 'hyper' linear algebra, that is linear algebra using tensors. It's well known that a linear transformation, $V \rightarrow V \in Vect$ between a fixed vector space in a fixed basis yields a matrix; likewise, a linear transformation between tensor spaces on a fixed vector space yields a hypermatrix, $\otimes^p V \rightarrow \otimes^q V \in Vect^{\otimes}$. This might help make the step between matrix mathematics and linear algebra to that of tensors significantly easier to understand for students - I certainly recall being confused by the definition of a tensor field as it is explained in GR texts when I first came across it simply because I expected an invariant definition in the way vectors are presented invariantly. Instead, they are presented covariantly.

An invariant definition is manifestly covariant but a covariant definition - despite the name - is usually not.

However an invariant definition of tensor fields is possible, they are sections of vector bundle maps $\otimes^p TM \rightarrow \otimes^q TM \in VBun^{\otimes}M$ and this is manifestly - just look at the definitions - the definition of a tensor lifted to a manifold.

There is also a recent book, whose title and authors I fail to recall bit I do recall their ethnicity - they were either Chinese or had Chinese heritage. They extensively list analogues of matrix operations for hypermatrices. They also came up with an intriguing terminological innovation: they suggest calling a hypermatrix a tentrix in analogy to matrix and keeping the term tensor solely for the invariant notion.

Answer 15

Hi Joseph,

it seems that the basic difference between hypermatrices and matrices is the indexation.

Let me explain what I mean. Consider a finite graph $G=(V,E)$ (the graph structure is not important I mentioning this just to refer to some geometrical interpretation of the index) and a collection of numbers $$(A_{v,w})_{(v,w)\in V\times V}.$$

Despite this list have a "two dimensional" character (bi-indexed) it is in fact a hypermatrice.

The way to import to hypermatrices the results of standard theory of matrix is to define the operations as usual. The product, for instance, is $$ (AB)_{(v,w)}=\sum_{z\in V}A_{v,z}B_{z,w}. $$

The determinant and other important objects can also be defined in this way by replacing the group $\mathbb{S}_n$ by automorphisms of $V$ and so on.

So I guess that do not pop up any interesting feature that justify to give a great attention to this object. Because one can see them just as a replacement of the indexation process.

For the other hand in fields like statistical mechanics and percolation they are much more natural in high dimensional problems than the usual matrices. When we define $p_{uv}$ the probability to see the edge $\{u,v\}$ open or when we are leading with the coupling constants $J_{i,j}$ in the Ising model, all of them are hypermatrices and its linear algebra structure are frequently arise in proofs of important results in correlations inequalities.