Let $X$ be compact Kahler and $E \to X$ a holomorphic vector bundle. Then $E$ has an Atiyah class, $At(E)$, valued in the sheaf cohomology $H^1(\Omega_X \otimes \operatorname{End} E)$. Suppose the topological Chern classes of $E$ vanish rationally. Evaluating $At(E)$ on an invariant polynomial gives a class in $\bigoplus_k H^k(\Omega_X^k)$, which must vanish in non-zero degrees since, by the Hodge decomposition, they correspond to polynomials in the Chern classes of $E$.

My question is:

Is it the case that the full Atiyah class itself, $At(E)$, is necessarily zero in $H^1(\Omega_X \otimes \operatorname{End} E)$?