In order for such a definition to be multiplicative with respect to composition, the determinants between vector spaces of different dimensions has to be zero. Here is a proof.

Let $f: \mathbb{R}^k \to \mathbb{R}^m$ be a linear map. If $k < m$, then we let $g: \mathbb{R}^m \to \mathbb{R}^m$ to be any projection to the image of $f$. The rank of $g$ is at most $k$, so $\det g$ is zero. Because $g \circ f =f$, we have:
$$0 = (\det g) \cdot (\det f) = \det (g \circ f) = \det f.$$
If $k > m$, there exists a subspace of $\mathbb{R}^k$ on which the restriction of $f$ is an isomorphism to the image of $f$, and we let $h: \mathbb{R}^k \to \mathbb{R}^k$ be the composite of $f$ followed by the inverse of that restriction. We have $f \circ h = f$, and $h$ has rank at most $m$, hence determinant zero. Therefore, $\det f = 0$.

It is conceivable but unlikely that Nash has found a useful nonzero extension of the notion of determinant that does not respect composition.

Incidentally, there is a way to get a meaningful "determinant" from a map $V \to W$, but it involves turning it into a two-term complex of vector spaces. Two-term complexes and maps of vector spaces both happen to look the same, since they are described by the same data, but they admit different natural operations (e.g., we like to compose maps but not complexes). This paper by Knudsen and Mumford describes the determinant as a symmetric monoidal functor from the category $C^\bullet is$, whose objects are bounded chain complexes of finite rank vector spaces and whose morphisms are quasi-isomorphisms, to the category $Pis$, whose objects are graded lines, and whose morphisms are homogeneous isomorphisms. In other words, if you are given a two-term complex $V \to W$, with $W$ in degree zero, determinant produces a graded line canonically isomorphic to $(\bigwedge^{max}V)^\vee \otimes (\bigwedge^{max} W)$ in degree $\dim V$. Note that the determinant in this case is *not* a distinguished element of that line, although a choice of bases of $V$ and $W$ seems to yield a distinguished basis element in the line.