How to prove the following inequality: Let $X$ and $Y$ be $n\times m$ matrices with real entries. Prove that \begin{equation} \det\left(XY^T\right)^2 \leq \det\left(XX^T\right)\det\left(YY^T\right) . \end{equation}
If you replace determinants by traces, then this inequality is just CauchySchwarz for the inner product $(X,Y)=\mathop{\mathrm{tr}}(XY^T)$ on the space of matrices. Now, we just have to recall that determinants are traces: for a $n\times n$matrix $A$ we have $\det(A)=\mathop{\mathrm{tr}}(\Lambda^n(A))$, where $\Lambda^n(A)$ is the $n$th exterior power of the linear operator $A$.
(Now that I took time to look at answers above I believe this is more or less precisely Wadim Zudilin's proof made a tiny bit more invariant.)

1$\begingroup$ The answers are not above any more! :) $\endgroup$ – Wadim Zudilin May 25 '10 at 11:19

$\begingroup$ True  I keep forgetting about the builtin nonlinearity :) $\endgroup$ – Vladimir Dotsenko May 25 '10 at 12:46
If $n>m$ all the determinants are zero.
The CauchyBinet formula allows you to write the determinant of $XY^t$ as the sum $$ \sum_{I\subset\lbrace1,2,\dots,m\rbrace}\det(X_IY_I^t) =\sum_{I\subset\lbrace1,2,\dots,m\rbrace}\det(X_I)\det(Y_I^t) $$ where the sum is over all $n$element subsets of $\lbrace1,2,\dots,m\rbrace$. Here $X_I$ and $Y_I$ are quadratic $n\times n$ matrices with columns of $X$ and $Y$ from the set $I$. The similar formulas are available for $\det(XX^t)$ and $\det(YY^t)$, so the wanted inequality reduces to verification of $$ \left(\sum_{I}x_Iy_I\right)^2\le \sum_{I}x_I^2\sum_Iy_I^2. $$ This a known inequality.

$\begingroup$ Yep, I think this is the easiest answer. Thank you very much! $\endgroup$ – Marine May 28 '10 at 4:12
Denotes vectorsrows of $X$ by $x_1$, $\dots$, $x_n$, of $Y$ by $y_1$, $\dots$, $y_n$, all $x_i$, $y_i$ belong to $\mathbb{R}^m$. Let $y_i=z_i+w_i$, where $z_i$ belongs to linear span of $x_1$,$\dots$,$x_n$, and $w_i$ is orthogonal to this linear span. If we replace $y_i$ to $z_i$, the left hand side does not change, while the right hand side may only decrease ($YY^t=ZZ^t+WW^t$ with natural notations, and all matrices are nonnegative definite, so all eigenvalues of $YY^t$ are at least the same as those of $ZZ^t$, hence $\det(YY^t)\geq \det(ZZ^t)$.) But if all $x_i$, $y_i$ belong to the same subspace of dimension $n$, then equality in inicial inequality occurs, as we may think that $m=n$.