For finite dimensional spaces, as distance functions are Lipschitz, they are differentiable almost everywhere. So a stronger form of your conjecture is true, namely the complement of $D\cup A$ has measure zero. In infinite dimensions this fails in general as remarked in the comments. I'm not sure what happens for infinite dimensional Hilbert spaces though. Also, the set of points where the distance function to $A$ is non differentiable is known as the medial axis of the complement of $A$, and a great deal is known about its properties.

For finite dimensional spaces, as distance functions are Lipschitz, they are differentiable almost everywhere. So a stronger form of your conjecture is true, namely the complement of $D\cup A$ has measure zero. In infinite dimensions this fails in general as remarked in the comments. I'm not sure what happens for infinite dimensional Hilbert spaces though. Also, the set of points where the distance function to $A$ is non differentiable is known as the medial axis of the complement of $A$, and a great deal is known about its properties.

Note: http://annals.math.princeton.edu/wp-content/uploads/annals-v157-n1-p05.pdf seems to be a relevant reference for positive results in the infinite dimensional case