Let $x(u,t) = \exp_{\gamma(t)}(u E(t))$. For fixed $t$, as $u$ ranges from 0 to 1, the curve $x(\cdot, t)$ is a geodesic segment from $\gamma(t)$ to $\exp_{\gamma(t)}(E(t))$. Then $\phi'(t) = J(1)$ where $J$ is the Jacobi field along this geodesic segment, with the initial conditions $J(0) = \gamma'(t)$ and $(\nabla_{d x/d u} J)(0) = 0$.

Why? As $t$ varies, $x$ is a variation through geodesics, so for fixed $t$, $\frac{\partial x}{\partial t}(u,t)$ is a Jacobi field (call it $J$) along the geodesic $x(\cdot, t)$. Then $\phi'(t) = \frac{\partial x}{\partial t}(1,t) = J(1)$ while $\gamma'(t) = \frac{\partial x}{\partial t}(0,t) = J(0)$. And using the fact that $\nabla$ is torsion-free and $E$ is parallel we have $(\nabla_{\partial x/\partial u} \frac{\partial x}{\partial t})(0,t) = (\nabla_{\partial x/\partial t} \frac{\partial x}{\partial u})(0,t)= \nabla_{\partial x/\partial t}E(t) = 0.$

(I guess this does not need $\gamma$ to be a geodesic....?)