Why are tensors a generalization of scalars, vectors, and matrices?

Question

Take two vector spaces $V$ and $W$ over a field $F$. One may form the tensor product $V\otimes W$ and it fulfills an universal property. Elements of $V\otimes W$ are called tensors and they are linear combinations of elementary tensors $v\otimes w$, the elementary tensors generate $V\otimes W$.

People from physics think of a tensor as a generalization of scalars, vectors, and matrices, I think and I have seen them tensoring matrices with matrices as entries with matrices and so on.

What does this mean and what has it to do with the definition from above? What "is" a tensor?

Answer 1

I have heard it said that tensor products are the hardest thing in mathematics. Of course that's not really true, but certainly a fluent understanding of how to work with tensor products is one of the dividing lines in your education from basic to advanced mathematics.

Disclaimer: What I will discuss here are tensor products in the sense of linear algebra, so only tensor products of individual vector spaces rather than tensor fields (which is what the physicists mean by tensor product).

For a long time I could not understand how the physicists could work with tensors by thinking about them as "quantities that transform in a certain way under a change in coordinates". The only way I could come to terms with them is by their characterization as something that satisfies a universal mapping property. Do not think about what tensors (elements of a tensor product space) are but rather what the whole construction of a tensor product space can do for you. It's sort of like quotient groups (only harder), where if you focus all your energy on trying to understand cosets you kind of miss the point of quotient groups. What makes tensor product spaces harder to come to terms with than quotient groups is that most elements of a tensor product space are not elementary tensors $v \otimes w$ but only sums of these things.

The whole (mathematical) point of tensor products of vector spaces is to linearize bilinear maps. A bilinear map is a function $V \times W \rightarrow U$ among $F$-vector spaces $V, W$, and $U$ which is linear in each coordinate when the other one is kept fixed. There are tons of bilinear maps in mathematics, and if we can turn them into linear maps then we can use constructions related to linear maps on them. The tensor product $V \otimes_F W$ of two $F$-vector spaces provides the most extreme space, so to speak, which is a domain for the linearization of all bilinear maps out of $V \times W$ into all vector spaces (over $F$). It is a particular vector space together with a particular bilinear map $V \times W \rightarrow V \otimes_F W$ such that any bilinear map out of $V \times W$ into any vector space naturally (!) gets converted into a linear map out of this new space $V \otimes_F W$. Some notes I wrote on tensor products for an algebra course are at http://www.math.uconn.edu/~kconrad/blurbs/linmultialg/tensorprod.pdf, and in it I address questions like "what does $v \otimes w$ mean?" and "what does it mean to say $$ v_1 \otimes w_1 + \cdots + v_k \otimes w_k = v_1' \otimes w_1' + \cdots + v_k' \otimes w_k'?" $$ Right from the start I allow tensor products of modules over a ring, not just vector spaces over a field. There are some aspects of tensor products which appear in the wider module context that don't show up for vector spaces (particularly since modules need not have bases). So you might want to skip over, say, tensor products involving ${\mathbf Z}/m{\mathbf Z}$ over $\mathbf Z$ on a first pass if you don't know about modules.

As for the question of how tensor products generalize scalars, vector spaces, and matrices, this comes from the natural (!) isomorphisms $$ F \otimes_F F \cong F, \ \ \ F \otimes_F V \cong V, \ \ V \otimes_F V^* \cong {\rm Hom}_F(V,V). $$ On the left side of each isomorphism is a tensor product of $F$-vector spaces, and on the right side are spaces of scalars, vectors, and matrices. In the link I wrote above, see Theorems 4.3, 4.5, and 5.9. You can also tensor two matrices as a particular example in a tensor product of two spaces of linear maps. Spaces of linear maps are vector spaces (with some extra structure to them), so they admit tensor products as well (with some extra features).

Returning to the physicist's definition of tensors as quantities that transform by a rule, what they always forget to say is "transform by a multilinear rule". I discuss the transition between tensor products from the viewpoint of mathematicians and physicists in section 6 of a second set of notes at http://www.math.uconn.edu/~kconrad/blurbs/linmultialg/tensorprod2.pdf.

Answer 2

Suppose you have a pair of elements in vector spaces, $v\in V$ and $w\in W$. Now suppose that at some future point you're going to compute $f(v,w)$ where $f$ is a bilinear function. For example, when $V=W=\mathbb{R}^3$, $f$ might be the familiar dot or cross product. But it might be something else entirely. In fact, suppose you don't know in advance what $f$ is going to be.

The tensor product precisely answers the question "what information do I need about $v$ and $w$ in order to be able to compute $f(v,w)$ at some future point, whatever $f$ turns out to be?" You could say "knowing $v$ and $w$ is enough information". But that that's more information than you need. $v\otimes w$ contains less information than $(v,w)$ and actually contains the least information you can get away with, and still be able to compute $f(v,w)$ for any bilinear $f$.

I don't know if this will be helpful for you, but when I realised this everything suddenly became crystal clear. It's really just a restatement of the universal property.

Answer 3

There is a notion of multiplication of tensors, because the physicist tensors come in covariant, contravariant and mixed flavour. And r-fold covariant and s-fold contravariant tensor is mathematically speaking an element of $V^{\otimes r}\otimes (V^\ast)^{\otimes s}$ (or maybe the other way).

A matrix is a mixed (1,1)-tensor if you interpret it as a linear map, i.e. an element of $V\otimes V^\ast$. A matrix is a pure (0,2)-tensor if you interpret it as a bilinearform. This yields an element of $V^\ast \otimes V^\ast$.

There are two kinds of multiplication of tensors: The tensorproduct itself taking a (r,s) and a (r',s') tensor and returning a (r+r',s+s') tensor. But if you want to multiply a (r,s) and a (s,t) tensor, there is another way: applying the linearform from the $V^\ast$-parts of the first tensor to the vectors from the $V$-parts in the second tensor. For matrices (=(1,1)-tensors) this is $(v\otimes \alpha)\circ (w\otimes \beta) := v\otimes (\alpha(w)\cdot \beta)=\alpha(w)\cdot (v\otimes \beta)$. This second multiplication is a generalization of the matrix multiplication to tensors.

Answer 4

I'd like to borrow a little from several of the answers given so far to give a "practical" perspective. Let $V$ be an $\mathbb{R}$-vector space. A tensor of type $(m, n)$ is an element of $V^{\otimes m} \otimes (V^{*})^{\otimes n}$, which translates for small $m, n$ into the following:

• A scalar is a tensor of type $(0, 0)$,
• A vector (in the strict sense of an element of $V$) is a tensor of type $(1, 0)$,
• A covector is a tensor of type $(0, 1)$,
• A bilinear form is a tensor of type $(0, 2)$, and
• A matrix is a tensor of type $(1, 1)$.

Now, why would we be interested in higher-order tensors? Well, let $f : V \to \mathbb{R}$ be a sufficiently nice function. Then it has a Taylor expansion

$$f(v) = f(0) + v^T \nabla f(0) + v^T H(0) v + ...$$

where the first term is a tensor of type $(0, 0)$, the second is a tensor of type $(0, 1)$ described by the gradient, and the third is a (symmetric) tensor of type $(0, 2)$ described by the Hessian. All this is standard calculus material. But how does the Taylor expansion continue?

As it turns out, the next terms are described by (symmetric) tensors of type $(0, 3)$, type $(0, 4)$, ... etc. So that's one sense in which tensors naturally generalize familiar vector- and matrix-like objects (the gradient and the Hessian).

Answer 5

If you think in terms of bases, then the basis for the tensor product is $\{e_i \otimes e_j\}$. You may think of this as an $n\times m$ matrix with exactly one non-zero entry (1) at the (i,j)th position. The tensor product of $V$ and $W$, then is isomorphic to the space of $n\times m$ matrices. Meanwhile, the tensor product of linear maps can be thought of as an index array with four indices --- i.e. as a hypercubical array.

There is hardly no advantage in trying to multiply hyper-cubical arrays along edges, faces, cubes etc. So a very efficient way of doing so is to begin thinking of things as "abstract tensors." These are boxes with strings emanating from the tops and bottoms. They are notational short hands for the resulting multi-indexed objects such as vectors, and matrices.

An alternative answer can in as I was typing this ...

Answer 6

In old times, a tensor was 'something' whose components transformed in a specific way when you changed coordinates. Nowadays, well, it is the same thing with different language.

Answer 7

I remember being thoroughly confused by this when I first tried to learn this from older math and physics books as a highschool student. From this point of view a tensor was something with many indices which transformed according to specific and rather complicated rules. It was never clear to me what a tensor actually was.

Later on, when I learned the modern viewpoint, things (eventually) made more sense. In modern language, it goes like this. Start with an open set X of Euclidean space or more generally a manifold. A tensor (field) on X of type (m,n) assigns to each $x\in X$ an element of the tensor product $T_x\otimes \ldots T_x\otimes T_x^*\otimes \ldots T_x^*$ where one has m factors of the tangent space $T_x$ and n factors of the cotangent space $T_x^*$. After choosing local coordinates, one can write everything out in bases and compare it to the classical notation. Specializing $(m,n)=(0,0)$ corresponds to a scalar field, and $(m,n)=(1,0)$ to a vector field. Also when $m+n=2$, we can view a tensor (locally) as a matrix valued function.

I hope that helps.

Answer 8

I think many of the answers assume you know what tensor product is...

A particularly illuminating example might help here. Before I give it, I'd like to mention that mathematicians don't acknowledge the origin of the name tensor. The meaning was probably lost in over use, but is preserved in translations in other languages (I'm Chinese). Tensor looks like tension, and (I imagine) was used first to describe tension of a membrane or something. (Then Riemann took it up as something that has more than two indices, for his curvature for example. I could be wrong about the history.)

If you want to tensor a plane with another plane, that's not gonna be particularly illuminating. It'll be a four-dimensional space, but one hardly sees what it has to do with the two planes.

Let $V$ be the vector space of polynomials of degree $\leq n$, and let $W$ be the vector space of polynomials of degree $\leq m$. As you know, the tensor product $V\otimes W$ should consist of all things of the form $v\otimes w$, for each $v\in V$ and each $w\in W$, and all (finite) linear combinations of it. Luckily in this case, we have a good candidate for what $v\otimes w$ is (or is being identified with). For $v=p(x)$ a polynomial in $x$, and $w=q(y)$ a polynomial in (another variable) $y$, then $v\otimes w$ is simply the product $p(x)q(y)$ as a polynomial of two variables. Now you see what the tensor product is: $V\otimes W$ is the vector space of polynomials of two variables, whose degree in $x$ is at most $n$, and degree in $y$ is at most $m$. Obviously $\dim V\otimes W = \dim V\times\dim W$.

It might be worthwhile to think about what happens to $V\otimes V$ when you make a change of basis on $V$.

It would be a sin not to mention the extension in infinite-dimensional case. $k[x]\otimes k[y]=k[x,y]$, as you can check using the universal property.

I hope it helps.

Answer 9

If you take a (finite dimensional) vector space V then there is an isomorphism

$V^* \otimes V \simeq \text{Hom}(V,V)$

So the first tensor product is precisely linear maps from $V$ to itself (viz. matrices, once you pick a basis for $V$).

If you have some extra structure on V (say a non-degenerated inner product or the choice of a basis) then you can identify V with $V^*$, thus having an identification

$V \otimes V \simeq V^* \otimes V \simeq \text{Hom} (V,V)$.

Everything else is just a (straightforward) generalisation of the above.

Answer 10

The following discussion is for finite-dimensional vector spaces only:

The tenor product of two vector spaces $V$ and $W$ arises, because you want to work with bilinear, rather than just linear, maps and functions of $V \times W$. The key observation is that the space of all possible bilinear scalar-valued functions or vector-valued maps is itself a vector space that is spanned by simple bilinear functions constructed by multiplying a linear function on $V$ by a linear function on $W$.

Let $T$ denote the vector space of all scalar-valued bilinear functions on $V \times W$ and observe that there is a natural map $V^* \times W^* \rightarrow T$. Moreover, it is easy to see that $V^* \times W^* $ generates all of $T$ in the sense that its image does not lie in any proper subspace of $T$. So this leads to the notation $T = V^* \otimes W^*$

The next observation is that there is a natural bilinear map $V\times W \rightarrow T^* $ that corresponds to evaluation of the given $(v,w)$ with an element of $T$. Again, the image "spans" all of $T^* $ in the sense that it does not lie in any proper subspace. Moreover, the linear duality between $T$ and $T^* $ corresponds exactly to the evaluation of a bilinear function on an element in $V\times W$. So it is reasonable to denote $T^*$ by $V \otimes W$.

So, morally speaking, $V\otimes W$ is the "smallest" vector space such that there is a bilinear injective map $V \times W \rightarrow V\otimes W$. Of course, this statement can be made precise by defining $V\otimes W$ as a universal object.

Finally, it is not difficult to prove the rather cool (and for me not so obvious) observation that the space of linear maps $V \rightarrow W$ is naturally isomorphic to $W\otimes V^* $. In this sense, tensors generalize the matrices viewed as linear transformations.

Answer 11

More generally, you can form tensor products $V_1\otimes \cdots \otimes V_k$ of an arbitrary number of vector spaces, and a tensor refers to an element of one of these spaces, not just the case $k=2$.

If $k=1$ then obviously a tensor is the same as a vector in $V_k$.

Scott Carter explained how tensors for $k=2$ (i.e. rank 2 tensors) correspond to matrices. Here's another viewpoint on the same thing. If $W^*$ is the dual space of $W$ (i.e. the vector space of linear functionals from $W$ to the base field) then $V\otimes W^*$ can be identified with $L(W,V)$, the space of linear maps from $W$ to $V$, by setting $(v\otimes f)(w) = f(w) v$, for $v\in V$, $f\in W^*$, and $w\in W$. (I'm assuming all vector spaces are finite dimensional for simplicity; also there's some work to be done to show this identification really is well-defined.) Of course, given bases of $W$ and $V$, $L(W,V)$ can also be identified with a space of matrices.

In a similar way, $V^* \otimes V^*$ is identified with bilinear maps on $V$.

I was about to type something about multiplication of tensors, but since Johannes Hahn just wrote an answer about that so I'll refer you there.

Answer 12

Tensor in physical means is kind of linear operator. It acts on tangent/cotangent space of (usually) differential manifold and gives us at the end some invariants in term of scalars which means that such values are independent of certain class of coordinate changes. As main change on the manifold one have to concern is change of the map, such objects in order to have physical meaning ( which is simple independence of coordinate change) should have certain properties, during this change. So basically example here are volume, energy, mass.

The other cause of tonsorial calculus is that evolution operator for many dynamical systems may be prescribed infinitesimally as linear operator on tangent space, and in fact this nation is connected to Lie algebra of group acting on differential manifold of system configurations. So such linear "transports" on manifold configuration space acts on various quantities defined for a system, in infinitesimal as tensors - in fact they are coefficients of multidimensional Taylor expansion of such quantities. Such examples arises mainly in chemical kinetics of reactions for example where usually one concern linear approximations, the stress tensors of various continuous materials, or in General Relativity. It is worth of note that not all physical objects are tensors (although all coordinate independent are, however sometimes usual physical quantity may be for example a part of bigger tensorial object. In this cases such quantity is coordinate dependent, see: see "relativistic mass" or time dilatation discussions).

So basically geometry is behind tensors and it is beautiful one.