Here is a short proof in the case $M$ is complete (if not the statement is false, see the last paragraph) using the idea from Leo Moos' answer of using Rademacher's theorem.
Suppose $\mathcal{B}(p,q)$ has measure $>0$. By Lebesgue's density theorem, almost all points $a\in\mathcal{B}(p,q)$ satisfy that $\mathcal{B}(p,q)$ has density $1$ at $a$. Also, by Rademacher's theorem we can suppose that $d_p:y\mapsto d(p,y)$ and $d_q:y\mapsto d(q,y)$ are both smooth at $a$. Now let $\gamma_p,\gamma_q$ be minimizing geodesics from $p$ and $q$ to $a$, so that $\gamma_p(0)=p,\gamma_q(0)=q$ and letting $k=d(p,a)=d(q,a)$, $\gamma_p(k)=\gamma_q(k)=a$.
Let $v_p,v_q$ be the gradients of $d_p,d_q$ at $a$. Then $|v_p|,|v_q|\leq1$ due to $d_p,d_q$ being $1$-Lipschitz. Combining this with the fact that $d_p(\gamma_p(t))=t$ and $d_q(\gamma_q(t))=t$ for all $t\in[0,k]$, we must have $v_p=\gamma_p'(k),v_q=\gamma_q'(k)$. Also we cannot have $\gamma_p'(k)=\gamma_q'(k)$, because then by uniqueness of geodesics we would have $p=q$. So $v_p\neq v_q$.
Now note that for any vector $v\in T_aM$ with $\langle v_p,v\rangle>\langle v_q,v\rangle$ and some small $\varepsilon_v$ we have that $d(p,exp_a(tv))>d(q,exp_a(tv))$ $\forall t\in[0,\varepsilon_v]$. So for all $v$ with $\langle v_p,v\rangle\neq\langle v_q,v\rangle$ we have that $exp_a(tv)\not\in B(p,q)$ for all $t\in[0,\varepsilon_v]$. This easily contradicts the fact that $B(p,q)$ has density $1$ at $a$.
If $M$ is not complete, the statement is false: consider the manifold $\mathbb{R}^2\setminus\{(x,0);x>0\}$ with the usual metric of $\mathbb{R}^2$ and let $p=(3,-4),q=(0,-5)$. Then all points $(x,y)$ with $x,y>0$ are at the same distance of $p$ and $q$.