Nearly length minimizing paths are close to geodesics? Let $M$ be a Riemannian manifold which is geodesically convex.
It's known that length minimizing curves are geodesics (after a possible reparametrization).
Now fix* points $p,q \in M$
Is the following assertion true? 
For any $\epsilon$ there exist a $\delta$ such that: 
If $\alpha$ is a path between $p,q$  such that $L(\alpha) < d(p,q) + \delta$ then $\alpha$ is in $\epsilon$-neighbourhood of some minimizing geodesic $\gamma$ joining $p,q$. (maybe after some reparametrization, what I really want is $d(\operatorname{Image}(\alpha),\operatorname{Image}(\gamma)<\epsilon$). 
*As noted by Sebastian Goette (in an instructive example), the $\delta$ cannot be chosen uniformly for all $p,q$. 
 A: $\newcommand{\Im}{\operatorname{Image}}$
I am trying to construct a complete argument based on Pietro's suggestion:
I assume $M$ is complete. 
Assume by contradiction the claim is false. Then there exists an $\epsilon > 0$, and a sequence of paths $\alpha_n:I \to M$ joining $p,q$ such that, $L(\alpha_n) \le d(p,q) + \frac{1}{n}$, and $\alpha_n$ is not in an $\epsilon$ neighbourhood of any minimizing geodesic.
Since $L(\alpha_n)\to d(p,q)=r$, we can assume $\Im(\alpha_n) \subseteq \bar  B^M(p,2r)$ (the closed ball of radius $2r$ around $p$). By completness of $M$ , $\bar B^M(p,2r)$ is compact.
Hence by Arzela-Ascoli theorem, there exists a subsequence (which we denote also by $\alpha_n$) which converges uniformly to path $\alpha:I \to M$.
Now, by lower semicontinuity of the length functional we deduce:
$L(\alpha) \le \lim_{n \to \infty} L(\alpha_n) = d(p,q) $.
This implies $\alpha$ is length minimizing in $M$, hence it is a geodesic (perhaps after reparametrization).
Now, the (uniform) convergence $\alpha_n \to \alpha$ gives us a contradiction. (There exists $n$ such that $\alpha_n$ is in an $\epsilon$-neighbourhood of a minimizing geodesic).
Is it possible to extend the argument when assuming only $M$ is geodesically convex?
