Lemma 3.24 at page 87 of Alexander Grigor'yan's "Heat Kernel and Analysis on Manifolds" says that (I reformulate it slightly)
For any point $p \in M$ and chart $(U', h)$ around $p$ there exist a $U \subseteq U'$ and a constant $C \ge 1$ such that for all $x,y \in U$ we have
$$\frac 1 C \| h(x) - h(y) \| \le d(x,y) \le C \| h(x) - h(y) \| \ .$$
Taking $h = \exp_p ^{-1}$ and $v \in S_r$ you get
$$\frac 1 C \| \exp_p ^{-1} (u) - \exp_p ^{-1} (v) \| \le d(u,v) \le C \| \exp_p ^{-1} (u) - \exp_p ^{-1} (v) \| \ ,$$
whence by taking $\sup _{v \in S_r}$ one gets
$$\frac 1 C \| \exp_p ^{-1} (u) - \exp_p ^{-1} (S_r) \| \le d(u,S_r) \le C \| \exp_p ^{-1} (u) - \exp_p ^{-1} (S_r) \| \ ,$$
the first inequality being what you are asking for.
By looking at Grigor'yan's proof, it seems to me that for small enough $U$ and $r$ one may choose $C$ independent of $p$ and $r$.
