No, because your formula does not make sense:

$T\in Hom(E_x,X_x)$ and $\phi\in Hom(E_x,E_y)$ invertible means that
$$\phi^{-1}\circ T\circ \phi$$ is not well-defined unless $x=y.$

If you define $$\psi=\phi\circ T\circ \phi^{-1},$$ then $\psi$ is actually the parallel-transport of the induced connection $\nabla^{End}$ on the endomorphism bundle which is defined by satisfying the equation
$$(\nabla^{End}T)(e)=\nabla T(e)-T(\nabla e)$$
for all  $T\in\Gamma(Hom(E,E))$ and sections $e\in\Gamma(E).$ 

For a proof of this property consider $v,w\in E_x$ with $T(v)=w,$ and denote the corresponding parallel sections (along the given curve $\gamma$ from $x$ to $y$) by $v(t)$ and $w(t).$ Then, $\nabla_{\gamma'} v(t)=0$ and $\nabla_{\gamma'}w(t)=0.$ Hence, by uniqueness of solutions of ODE's, the parallel endomorphism field $T(t)$ along $\gamma$ satisfies 
$$T(t)(v(t))=w(t).$$
As $v\in E_x$ is arbitrary, this is equivalent to the equation $\psi=\phi\circ T\circ \phi^{-1}.$

The construction is compatible with metrics, as the same standard arguments for tensor products, dual bundles and corresponding connections carry over to the Riemannian/hermitian situation, i.e., the induced metric on the endomorphism bundle is parallel with respect to $\nabla^{End}.$ 
Note also that $\nabla^{End}$ is the unique connection whose parallel transport satisfies the equation $\psi=\phi\circ T\circ \phi^{-1}.$