By polar decomposition, every continuous linear function $f \colon H \to K$ between Hilbert spaces can be written uniquely as $f = \widehat{f} \circ f$ for a positive operator $f \colon H \to H$ and a partial isometry $\widehat{f} \colon H \to K$ with $\ker(\widehat{f})=\ker(f)$. The binary operation $(f,g) \mapsto g \circ f$ on the set of positive operators is not associative. Is the binary operation $(f,g) \mapsto \widehat{g \circ f}$ on the set of partial isometries associative?
I have not fully checked this idea, but here goes. I prefer the notation $P(T)$ for the partial isometry occurring in the polar decomposition of $T$. I also got lost with three Hilbert spaces in the mix, so this answer is only for the case where $A,B,C$ all operate on the same fixed Hilbert space $H$.
In this notation, I believe the question is whether or not
$$
P(P(AB)C) = P(A P(BC))
$$
holds for all partial isometries $A,B,C$ on $H$.
I think it is easy to see that if $U$ is unitary, then for all $X$ we have $P(UX) = U P(X)$, and that if $T$ is a partial isometry, then $P(T) = T$. From this, it seems to follow that if $A$ is assumed unitary, and $B$ and $C$ are partial isometries, we have $P(P(AB)C) = P(AP(B)C) = P(ABC) = A P(BC)$ and $P(A P(BC)) = A P(P(BC)) = A P(BC)$ so the desired result holds.
If $A$ is not unitary, it still has a unitary dilation. This means there is another Hilbert space $K$ and operators $X: K \to H$ and $Y: K \to K$ with the property that the operator $A'$ on $H \oplus K$ given by the block operator matrix $A' = \begin{pmatrix} A & X \cr 0 & Y \end{pmatrix}$ is unitary. So consider the operators $B' = \begin{pmatrix} B & 0 \cr 0 & 0 \end{pmatrix}$ and $C' = \begin{pmatrix} C & 0 \cr 0 & 0 \end{pmatrix}$ on $H \oplus K$. The operators $B'$ and $C'$ are partial isometries on $H \oplus K$, so by the work above, $$ P(P(A'B')C') = P(A' P(B'C')). $$ Now calculate: $A'B' = \begin{pmatrix} AB & 0 \cr 0 & 0 \end{pmatrix}$ and so a moment's thought ought to show that $P(A'B') = \begin{pmatrix} P(AB) & 0 \cr 0 & 0 \end{pmatrix}$, making the left hand side of the above equal to $$ P(\begin{pmatrix} P(AB) C & 0 \cr 0 & 0 \end{pmatrix}) = \begin{pmatrix} P(P(AB) C) & 0 \cr 0 & 0 \end{pmatrix}. $$ The right hand side is almost the same, but with $P(A P(BC))$ in the upper left corner, and unless I made a ridiculous error, you have what you want. (There is a sightly more complicated unitary dilation theorem for operators between different spaces, so if there were no mistakes in this approach, maybe the same idea works in the general case too.)

$\begingroup$ The first half of your argument in fact shows the desired result to hold when A is an isometry (even when considering functions between different spaces). But unfortunately, in the second half, not every partial isometry can be dilated to an isometry. For example, $\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}$ is a partial isometry on $\mathbb{C}^2$, and no matter what larger space we embed $\mathbb{C}^2$ in, the kernel will stay nontrivial. $\endgroup$ – Chris Heunen Dec 1 '10 at 20:11

$\begingroup$ Ack! I even got the term wrong (my "dilation" is really an extension, and as you point out, things with kernels can't extend to things that don't have them). One can dilate (in the proper sense of the term) any contraction to a unitary, but the ``$A'$'' you get will have to have some junk in the lower left corner. Perhaps it is possible to salvage this idea by understanding what happens with the junk when you do a polar decomposition. But maybe it is no simpler than trying another approach. (I leave my whole answer unedited so that these comments make sense to people who read them.) $\endgroup$ – anon Dec 2 '10 at 0:18
Ah, (quite) some fiddling with Mathematica gave a counterexample. In the notation of anon's answer, take $$ A' = \begin{pmatrix} 1 & 1 & 1 & 1 \\\\ 1 & 1 & 1 & 0 \end{pmatrix}, \qquad B = \begin{pmatrix} 0 & 1 \\\\ 1 & 0 \end{pmatrix}, \qquad C = \begin{pmatrix} 0 & 1/\sqrt{2} \\\\ 1 & 0 \\\\ 0 & 1/\sqrt{2} \end{pmatrix}. $$ Then $A=P(A')$, $B$ and $C$ are partial isometries, but $P(P(CB)A) \neq P(CP(BA))$. Notice that $C$ is a partial isometry that is not an isometry, i.e. has nontrivial kernel.

$\begingroup$ Wow, neat. I wonder if this is a "minimal" example (or if it's not, what a minimal example looks like). My own search was with smaller matrices than this (everything between spaces of dimension at most 3), and nothing seemed to work, so I gave up. $\endgroup$ – anon Dec 9 '10 at 7:57