(Not so much a response to the OP, but to the question in title.)
One of the motivations for Schwartz's distributions was to clarify the solution to the (free) wave equation (the Cauchy problem), after the earlier works of Hadamard and M. Riesz. Unlike the elliptic and parabolic counterparts, the fundamental solution to the wave equation, for spatial dimension = 3, 5, 7 ... etc., is truly singular; more precisely its support is contained in the lightcone, a phenomenon known as the Huygens' principle.
Riesz essentially constructed the fundamental solution by means of analytic continuation: putting aside some (important) factors, we may start with
$$ R^s(x) = \begin{cases} \frac{1}{\Gamma_n(s)} (x_1^2 - x_2^2 - \cdots - x_n^2)^{(s-n)/2} & x \in C_+ \\ 0 & x \not\in C_+ \end{cases} $$
which defines a distribution for $\operatorname{Re} s>n-2$. Here, $C_+$ is the future cone: $C_+=\{x\in\mathbb R^n | \sqrt{x_2^2+\cdots+x_n^2}\leq x_1\}$. If we let the wave operator $\square = \partial_1^2-\partial_2^2-\cdots-\partial_n^2$ act on $R^s$ (on the $x$ variables), we see by direct calculation that $\square R^s = c R^{s-2}$ for some constant $c$ (depending on $s$). So we may define $\Gamma_n(s)$ appropriately so that it goes away: $\square R^s = R^{s-2}$. This, then, allows one to analytically continue $R^s$ to all $s\in\mathbb C$ (after one notes that $\Gamma_n(s)$, essentially with two factors of the usual Gamma function, is analytic in $s$).
Note what happens to $R^{n-2}$ (which was just shy of being defined by the original expression). It is (by definition) a derivative of $R^n$, which is constant inside (and outside) the cone, so must vanish there upon differentiation: $\operatorname{supp} R^{n-2} \subseteq \partial C_+$. If $n=2$ (i.e., one-dimensional wave equation, and the "lightcone" $\partial C_+$ consists of two rays), one can compute directly that $R^0 = \delta$ (up to some constant; we could define $\Gamma_n(s)$ appropriately to make that go away).
For $n=4$ (our space-time), the distribution $R^{n-2} = R^2$ has a rather simple description: $$ R^2(x) = \frac{\delta(x_1 - r)}{4\pi r}, \qquad r=\sqrt{x_2^2+x_3^2+ x_4^2} $$ and, by applying $\square$ on it again, we magically have $R^0=\delta$. Thus, the fundamental solution is $R^2$. (If I had to pick, that is the one example of distribution I'd keep in mind.)
For higher even dimensions ($n=6, 8,\ldots$), we get succesively $$ R^{n-2}, R^{n-4}, \ldots, R^2, R^0 $$ all but the last one supported on the lightcone -- but more and more "singular" (or "twisted" in some sense) -- and bingo, $R^0=\delta$! This again makes $R^2$ the fundamental solution. (One may of course continue applying $\square$ to obtain $R^{-2}, R^{-4}, \ldots$, all supported at the origin.)
For odd dimensions, $R^{n-2},R^{n-4}, \ldots$ do have the same support structure, but it skips over $R^0$ (which incidentally is still $\delta)$. The true fundamental solution $R^2$ is the repeated derivative of some function that does not vanish inside the cone, so $\operatorname{supp} R^2 = C_+$ (and Huygens' principle fails). The dichotomy (with the parity of $n$) could be made more clear by stating the $\operatorname{supp} R^s$ for all $s\in\mathbb C$.
The crucial fact that $R^0=\delta$ is a surprisingly simple consequence of the fact that the factor $\Gamma_n(s)$ has a pole at $s=0$, or that $\frac{1}{\Gamma_n(s)}$ has a zero. I suspect it was essentially in Riesz, but is perhaps more clear in the language of distributions in Duistermaat [91] and Kolk-Varadarajan [91]. It probably appeared somewhere in Vol. I of Gelfand-Shilov, if not in Schwartz. (I'd welcome opinions on this from experts.)
So, it may be true that any distribution is a derivative of some measure, but it can have some unexpected behavior when the support suddenly jumps -- or rather, shrinks. Of course it happens very rarely, but they are the most interesting cases since all fundamental solutions are just like that.