I believe this is a very interesting question, that I have been asking myself for quite a long time.
Nevertheless, I have been told by Prof. Beauville that even in the irreducible case one does not have that $$ Ext_X(\mathcal O_X,\mathcal O_X)=Ext_X(T_X,T_X) $$
Namely, consider $X$ being the Hilbert scheme of two points on a $K3$ surface.
Then $Ext_X(\mathcal O_X,\mathcal O_X)=\mathbb{C}\oplus\mathbb{C}[-2]\oplus\mathbb{C}[-4]$.
But $Ext_X(T_X,T_X)=Ext_X(\mathcal O_X,(T^*_X)^{\otimes 2})$ contains $Ext_X(\mathcal O_X,\Omega^2_X)$, which is huge ($h^{2,2}=232$).
Anyway, I must say that this does not kill the question (this just tells we have to reformulate it). I hope to be able to write more about it soon.
EDIT: it seems that the answer to the question is NO. The point is that 232 is also the dimension of $H^1(X,S^3(T_X))$ ($X$ is again a $K3$), therefore $Ext_X^1(S^2(T_X),T_X)=RHom_X(\wedge^2(T_X[-1]),T_X[-1]))$ has dimension $\geq232$.
