As others mentioned, you have to remove the diagonal of $M\times M$ or square the distance function. Then, for a complete $M$, the answer is the following.

The distance function is differentiable at $(p,q)\in M\times M$ if and only if there is a unique length-minimizing geodesic from $p$ to $q$. Furthermore, the distance function is $C^\infty$ in a neighborhood of $(p,q)$ if and only if $p$ and $q$ are not conjugate points along this minimizing geodesic.

Thus, the function is smooth everywhere if and only if $M$ is simply connected and the geodesics have no conjugate points. This property has numerous equivalent reformulations, including the following

• for every pair of points, there is a unique minimizing geodesic between them;

• for every pair of points, there is a unique geodesic between them;

• every geodesic is minimizing;

• the exponential map at every point $p\in M$ is a diffeomorphism from $T_pM$ to $M$.

In general, the distance function has one-sided directional derivatives everywhere. This derivative has a nice description in the case when you fix $p\in M$ and study the function $f=d(p,\cdot)$. Namely let $q\in M$, $q\ne p$, and denote by $\overrightarrow{qp}$ the set of initial velocity vectors (in $T_qM$) of unit-speed minimizing geodesics from $q$ to $p$. Then, for a vector $v\in T_qM$, the one-sided derivative $f'_v$ of $f$ in the direction of $v$ is $$ f'_v=\min\{-\langle v,\xi\rangle:\xi\in \overrightarrow{qp}\} . $$ This follows from the first variation formula and holds not only in Riemannian manifolds but also in Alexandrov spaces. It is not hard to derive the above differentiablity properties from this.

I don't have a textbook reference for this precise formulation in the Riemannian case, but any book that covers Berger's lemma about geodesics realizing the diameter probably has directional derivatives as a sublemma. For Alexandrov spaces, the standard reference is Burago-Gromov-Perelman's paper. An intro-level proof (not in a full generality) can be found in (a shameless advertisement follows) "A course in metric geometry" by Burago, Burago and myself, section 4.5.

As others mentioned, you have to remove the diagonal of $M\times M$ or square the distance function. Then, for a complete $M$, the answer is the following.

The distance function is differentiable at $(p,q)\in M\times M$ if and only if there is a unique length-minimizing geodesic from $p$ to $q$. Furthermore, the distance function is $C^\infty$ in a neighborhood of $(p,q)$ if and only if $p$ and $q$ are not conjugate points along this minimizing geodesic.

Thus, the function is smooth everywhere if and only if $M$ is simply connected and the geodesics have no conjugate points. This property has numerous equivalent reformulations, including the following

• for every pair of points, there is a unique minimizing geodesic between them;

• for every pair of points, there is a unique geodesic between them;

• every geodesic is minimizing;

• the exponential map at every point $p\in M$ is a diffeomorphism from $T_pM$ to $M$.

In general, the distance function has one-sided directional derivatives everywhere. This derivative has a nice description in the case when you fix $p\in M$ and study the function $f=d(p,\cdot)$. Namely let $q\in M$, $q\ne p$, and denote by $\vec{qp}$ the set of initial velocity vectors (in $T_qM$) of unit-speed minimizing geodesics from $q$ to $p$. Then, for a vector $v\in T_qM$, the one-sided derivative $f'_v$ of $f$ in the direction of $v$ is $$ f'_v=\min\{-\langle v,\xi\rangle:\xi\in \vec{qp}\} . $$ This follows from the first variation formula and holds not only in Riemannian manifolds but also in Alexandrov spaces. It is not hard to derive the above differentiablity properties from this.

I don't have a textbook reference for this precise formulation in the Riemannian case, but any book that covers Berger's lemma about geodesics realizing the diameter probably has directional derivatives as a sublemma. For Alexandrov spaces, the standard reference is Burago-Gromov-Perelman's paper. An intro-level proof (not in a full generality) can be found in (a shameless advertisement follows) "A course in metric geometry" by Burago, Burago and myself, section 4.5.