I can just tell you what the space looks like up to the action. The orbit classification of $\text{hom}(V,W) \oplus \text{hom}(W,V)$ looks a lot like the orbit classification of $\text{hom}(V,V)$, which as you know from linear algebra is given by Jordan canonical form. In fact, if you compose the two homs, the classification is clearly at least as complicated as JCF. There is just a modest amount more structure because the nilpotent part is more complicated. Although the formal dimension of the quotient is indeed negative or 0, its geometric dimension is strictly positive when $V$ and $W$ are non-trivial.

Let
$$f:V \to W \qquad g:W \to V$$
be the two linear maps. Then $f$ has a kernel, $g \circ f$ could have a larger kernel, etc. Define the *stable kernel* $V_0$ and the *stable image* $V_1$ of $g \circ f$ to be the direct limits of the kernel and the image of $(g \circ f)^n$. As in one proof of Jordan canonical form, $V = V_0 \oplus V_1$. Similarly $W = W_0 \oplus W_1$. The pair $(f,g)$ canonically splits into two pairs, $(f_0,g_0)$ and $(f_1,g_1)$. The pair $(f_0,g_0)$ is nilpotent, while the pair $(f_1,g_1)$ is invertible and establishes an isomorhism $V_1 \cong W_1$.

Because of the isomorphisms between $V_1$ and $W_1$, the invariant information in the pair $(f_1,g_1)$ is the Jordan canonical form of $g_1 \circ f_1$, which is the same as the JCF of the other composition. In other words, either $f_1$ or $g_1$ can be any isomorphism, and then the other one can be chosen to establish a prescribed Jordan canonical form. Any eigenvalue can appear other than 0.

The nilpotent pair $(f_0,g_0)$ is a little more interesting. It looks like an Ouroboros. An indecomposable nilpotent pair is any finite chain
$$0 \to k \to k \to \cdots \to k \to 0,$$
rolled up from $\mathbb{Z}$-graded to $\mathbb{Z}/2$-graded. (The connecting maps in the middle are all isomorphisms, not differentials.) The chains can have odd length, so that $V_1$ and $W_1$ don't have to have the same dimension. A chain of any length can also descend in two different ways to $V_1$ and $W_1$.

I gather from Ben's comment that this is a right answer to a wrong question. It is a good description of the representation variety of a cyclic quiver; there is nothing special about cycle length 2 in the analysis. But the $A_2$ quiver variety is something else.

twodirected edges in opposite directions? One directed edges would just give you hom(V,W) and not the other one. – Greg Kuperberg Dec 10 '09 at 0:00