You can continue to play the same "game" that lead to the probabilities for hitting $0$ and $x+y$ and the expectations for $\tau$ and $X_\tau$. Just construct martingales to which you can apply the optional stopping theorem. Note that $X_n$, $X_n^2-n$, $X_n^3-3nX_n$, $X_n^4-6nX_n^2+n(3n+2)$, are all martingales. Applying the optional stopping theorem to the first two yields
$P(X_\tau=0)=\frac{y}{x+y}$, $P(X_\tau=x+y)=\frac{x}{x+y}$, $E(\tau)=xy$, as you already know.
With the next one you can show that
$E(\tau X_\tau)=\frac{1}{3}xy(2x+y)$.
Conditioning on $X_\tau=0$ and $X_\tau=x+y$ gives
$E(\tau X_\tau)=\frac{y}{x+y}E(\tau X_\tau|X_\tau=0) + \frac{x}{x+y}E(\tau X_\tau|X_\tau=x+y)$,
which tells you that
$E(\tau|X_\tau=x+y) =\frac{1}{3}y(2x+y)$, $E(\tau|X_\tau=0)=\frac{1}{3}x(x+2y)$,
i.e. now you know the expectations of the hitting times conditional on player A winning or losing.
Applying the optional stopping theorem to the fourth martingale will give you the second moment of the hitting time...
After a while this gets a little bit tedious and you might want to use mgf...