I think the inconsistency appears because the transpose $X^{\rm t}$ is missing in your answer. Let me try to work this out, along the lines of a similar calculation in the orthogonal (rather than unitary) group.
We need the fourth-order tensor $$\int_{{\rm U}(n)}(U^m)_{ij}(U^m)^*_{kl}\,dU=a_{m}(n)\delta_{ij}\delta_{kl}+b_{m}(n)\delta_{ik}\delta_{jl}+c_{m}(n)\delta_{il}\delta_{jk}$$ so that the required integral takes the form $$\int_{{\rm U}(n)}U^m X(U^\dagger)^m\,dU=a_{m}(n)X+b_{m}(n)\mathbb{1}\,{\rm tr}\,X+c_{m}(n)X^{\rm t} $$ substitution $X=\mathbb{1}$ gives a first relation $$a_m(n)+nb_m(n)+c_m(n)=1$$ one more relation follows from application of theorem 2 of Diaconis and Shahshahani: $$n^2 a_m(n)+nb_m(n)+nc_m(n)=m$$ I need a third relation, which I will try to figure out a bit later today.