No, because your formula does not make sense:

$T\in Hom(E_x,X_x)$ and $\phi\in Hom(E_x,E_y)$ invertible means that
$$\phi^{-1}\circ T\circ \phi$$ is not well-defined unless $x=y.$

If you define $$\psi=\phi\circ T\circ \phi^{-1},$$ then $\psi$ is actually the parallel-transport of the induced connection $\nabla^{End}$ on the endomorphism bundle which is defined by satisfying the equation
$$(\nabla^{End}T)(e)=\nabla T(e)-T(\nabla e)$$
for all  $T\in\Gamma(Hom(E,E))$ and sections $e\in\Gamma(E).$