I'm not sure I believe your answer.  Perhaps I'm missing something though.

Let $T = X_{ij}Y^{ij} + \lambda (X_{ij}\delta^{ij})$, which is your original function plus a [Lagrange multiplier](http://en.wikipedia.org/wiki/Lagrange_multipliers) for the traceless constraint.

Extremize by setting partial derivatives with respect to the entries $X_{ij}$ to zero:

$0=\frac{\partial T}{\partial X_{ij}}=Y^{ij}+\lambda \delta^{ij}$

For entries where $i=j$, this is $Y^{ii}+\lambda =0$, which yields the condition that all diagonal entries of $Y$ are equal, not that $Y$ is traceless.  For the entries with $i\neq j$, we recover $Y^{ij}=0$ as usual.