MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).
show/hide this revision's text 3 edited body

As other answers indicated, the answer to yr first two questions is that a $G$-invariant complement to $V\subset U$ is unique iff $V$ and $U/V$ do not contain $G$-isomorphic invariant subspaces. You can prove it quite easily with Schur´s lemma.

The answer to your 3rd question is that for no $n$ there is a unique invariant complement to your $V$. That is, both your $V$ and $U/V$ contain $U(n)$-isomorphic subspaces.

In order to show this it is easier (for me) to complexify your representations. Denote by $W$ the usual representation of $U(n)$ on $\mathbb C^n$ (multiplication of column vectors by unitary matrices on the left). Then it is quite easy to verify the following isomorphisms of $U(n)$-representations:

${\mathbb R}^{2n} \otimes {\mathbb C} = W \oplus W^*$,

$\Lambda^2({\mathbb R}^{2n})\otimes {\mathbb C}=\Lambda^2 W \oplus \Lambda^2 W^* \oplus (W \otimes W^*)$

${\mathfrak u}(n)\otimes {\mathbb C}=W \otimes W^*.$

Also, you can show that your map $A$ is injective (it is in fact injective on all of $Hom({\mathbb R}^{2n},\mathfrak o(2n))$, which is a standard fact in the Cartan theory, sometimes called the $S_3$ lemma).

So you get the following $U(n)$-decomposition of $V$

$V\otimes{\mathbb C}=Hom({\mathbb R}^{2n},\mathfrak u(n))\otimes{\mathbb C}=(W \oplus W^* ) \otimes ( W \otimes W^* )=$

$=W\otimes W \otimes W^* + conj.$

(the $+conj $ means you need to add to previous summands their conjugate, or dual, which in this case is just $W^* \otimes W^* \otimes W$).

Next

$U\otimes{\mathbb C}=Hom(\Lambda^2({\mathbb R}^{2n}), {\mathbb R}^{2n})\otimes{\mathbb C}=(\Lambda^2 W \oplus \Lambda^2 W^* \oplus (W \otimes W^* ))\otimes ( W \oplus W^* ), $

so "substructing" substracting" $V \otimes{\mathbb C}$ you get

$(U/V)\otimes{\mathbb C}=(\Lambda^2 W \oplus \Lambda^2 W^* )\otimes ( W \oplus W^* ) =(\Lambda^2 W \otimes W) \oplus (\Lambda^2 W \otimes W^* )+conj.$

Now since $W\otimes W = \Lambda^2 W \oplus S^2 W$ you can see clearly that both $V$ and $U/V$ (or rather their complexificacions) contain subspaces isomorphic to $\Lambda^2 W\otimes W^* $, so that an invariant complement to $V$ is not unique.

Note that I didn't bother to decompose the spaces into irreducibles since it was not necessary for answering yr question. The common summand $\Lambda^2 W\otimes W^* $ contains in fact a summand isomorphic to $W$ ( it is given by the contraction mapping $\Lambda^2 W\otimes W^*\to W $), which means that $V$ and $U/V$ both have an invariant subspace ismorphic to the standard representation on ${\mathbb R}^{2n}$. Probably one can see it directly without the calculation above but I didn't try.

show/hide this revision's text 2 edited body

As other answers indicated, the answer to yr first two questions is that a $G$-invariant complement to $V\subset U$ is unique iff $V$ and $U/V$ do not contain $G$-isomorphic invariant subspaces. You can prove it quite easily with Schur´s lemma.

The answer to your 3rd question is that for no $n$ there is a unique invariant complement to your $V$. That is, both your $V$ and $U/V$ contain $U(n)$-isomorphic subspaces.

In order to show this it is easier (for me) to complexify your representations. Denote by $W$ the usual representation of $U(n)$ on $\mathbb C^n$ (multiplication of column vectors by unitary matrices on the left). Then it is quite easy to verify the following isomorphisms of $U(n)$-representations:

${\mathbb R}^{2n} \otimes {\mathbb C} = W \oplus W^*$,

$\Lambda^2({\mathbb R}^{2n})\otimes {\mathbb C}=\Lambda^2 W \oplus \Lambda^2 W^* \oplus (W \otimes W^*)$

${\mathfrak u}(n)\otimes {\mathbb C}=W \otimes W^*.$

Also, you can show that your map $A$ is injective (it is in fact injective on all of $Hom({\mathbb R}^{2n},\mathfrak o(2n))$, which is a standard fact in the Cartan theory, sometimes called the $S_3$ lemma).

So you get the following $U(n)$-decomposition of $V$

$V\otimes{\mathbb C}=Hom({\mathbb R}^{2n},\mathfrak u(n))\otimes{\mathbb C}=(W \oplus W^* ) \otimes ( W \otimes W^* )=$

$=W\otimes W \otimes W^* + conj.$

(the $+conj $ means you need to add to previous summands their conjugate, or dual, which in this case is just $W^* \otimes W^* \otimes W$).

Next

$U\otimes{\mathbb C}=Hom(\Lambda^2({\mathbb R}^{2n}), {\mathbb R}^{2n})\otimes{\mathbb C}=(\Lambda^2 W \oplus \Lambda^2 W^* \oplus (W \otimes W^* ))\otimes ( W \oplus W^* )$, $

so "substructing" $V \otimes{\mathbb C}$ you get

$(U/V)\otimes{\mathbb C}=(\Lambda^2 W \oplus \Lambda^2 W^* )\otimes ( W \oplus W^* ) =(\Lambda^2 W \otimes W) \oplus (\Lambda^2 W \otimes W^* )+conj.$

Now since $W\otimes W = \Lambda^2 W \oplus S^2 W$ you can see clearly that both $V$ and $U/V$ (or rather their complexificacions) contain subspaces isomorphic to $\Lambda^2 W\otimes W^* $, so that an invariant complement to $V$ is not unique.

Note that I didn't bother to decompose the spaces into irreducibles since it was not necessary for answering yr question. The common summand $\Lambda^2 W\otimes W^* $ contains in fact a summand isomorphic to $W$ ( it is given by the contraction mapping $\Lambda^2 W\otimes W^*\to W $), which means that $V$ and $U/V$ both have an invariant subspace ismorphic to the standard representation on ${\mathbb R}^{2n}$. Probably one can see it directly without the calculation above but I didn't try.

show/hide this revision's text 1

As other answers indicated, the answer to yr first two questions is that a $G$-invariant complement to $V\subset U$ is unique iff $V$ and $U/V$ do not contain $G$-isomorphic invariant subspaces. You can prove it quite easily with Schur´s lemma.

The answer to your 3rd question is that for no $n$ there is a unique invariant complement to your $V$. That is, both your $V$ and $U/V$ contain $U(n)$-isomorphic subspaces.

In order to show this it is easier (for me) to complexify your representations. Denote by $W$ the usual representation of $U(n)$ on $\mathbb C^n$ (multiplication of column vectors by unitary matrices on the left). Then it is quite easy to verify the following isomorphisms of $U(n)$-representations:

${\mathbb R}^{2n} \otimes {\mathbb C} = W \oplus W^*$,

$\Lambda^2({\mathbb R}^{2n})\otimes {\mathbb C}=\Lambda^2 W \oplus \Lambda^2 W^* \oplus (W \otimes W^*)$

${\mathfrak u}(n)\otimes {\mathbb C}=W \otimes W^*.$

Also, you can show that your map $A$ is injective (it is in fact injective on all of $Hom({\mathbb R}^{2n},\mathfrak o(2n))$, which is a standard fact in the Cartan theory, sometimes called the $S_3$ lemma).

So you get the following $U(n)$-decomposition of $V$

$V\otimes{\mathbb C}=Hom({\mathbb R}^{2n},\mathfrak u(n))\otimes{\mathbb C}=(W \oplus W^* ) \otimes ( W \otimes W^* )=$

$=W\otimes W \otimes W^* + conj.$

(the $+conj $ means you need to add to previous summands their conjugate, or dual, which in this case is just $W^* \otimes W^* \otimes W$).

Next

$U\otimes{\mathbb C}=Hom(\Lambda^2({\mathbb R}^{2n}), {\mathbb R}^{2n})\otimes{\mathbb C}=(\Lambda^2 W \oplus \Lambda^2 W^* \oplus (W \otimes W^* ))\otimes ( W \oplus W^* ) $,

so "substructing" $V \otimes{\mathbb C}$ you get

$(U/V)\otimes{\mathbb C}=(\Lambda^2 W \oplus \Lambda^2 W^* )\otimes ( W \oplus W^* ) =(\Lambda^2 W \otimes W) \oplus (\Lambda^2 W \otimes W^* )+conj.$

Now since $W\otimes W = \Lambda^2 W \oplus S^2 W$ you can see clearly that both $V$ and $U/V$ (or rather their complexificacions) contain subspaces isomorphic to $\Lambda^2 W\otimes W^* $, so that an invariant complement to $V$ is not unique.

Note that I didn't bother to decompose the spaces into irreducibles since it was not necessary for answering yr question. The common summand $\Lambda^2 W\otimes W^* $ contains in fact a summand isomorphic to $W$ ( it is given by the contraction mapping $\Lambda^2 W\otimes W^*\to W $), which means that $V$ and $U/V$ both have an invariant subspace ismorphic to the standard representation on ${\mathbb R}^{2n}$. Probably one can see it directly without the calculation above but I didn't try.