I suspect the original formula for $\delta(x)$ defined in my question above is not quite correct as the associated derived formula for $\delta'(x)$ has a discontinuity at $x=0$. The definition of $\delta(x)$ in formula (1) below eliminates the piecewise nature of my original formula which resolves this problem and also seems to provide simpler results for formulas derived via the Fourier convolution defined in formula (2) below. The formula for $\delta(x)$ defined in formula (1) below also seems to provide the ability to derive formulas for a wider range of functions via the Fourier convolution defined in formula (2) below. The evaluation limit $f$ in formula (1) below is the evaluation frequency and assumed to be a positive integer. When evaluating formula (1) below (and all formulas derived from it) the evaluation limit $N$ must be selected such that $M(N)=0$ where $M(x)$ is the Mertens function. Formula (1) is illustrated in Figure (1) further below. I believe the series representation of $\delta(x)$ defined in formula (1) below converges in a distributional sense.

(1) $\quad\delta(x)=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\sum\limits_{n=1}^N\frac{\mu(n)}{n}\left(\sum\limits_{k=1}^{f\ n}\left(\cos\left(\frac{2 \pi k (x-1)}{n}\right)+\cos\left(\frac{2 \pi k (x+1)}{n}\right)\right)-\frac{1}{2}\sum\limits_{k=1}^{2\ f\ n}\cos\left(\frac{\pi k x}{n}\right)\right)$

(2) $\quad g(y)=\int\limits_{-\infty}^\infty\delta(x)\,g(y-x)\,dx$

Formula (1) for $\delta(x)$ above leads to formulas (3a) and (3b) for $\theta(x)$ below (illustrated in Figures (2) and (3) further below) and formula (4) for $\delta'(x)$ below (illustrated in Figure (4) further below). Note formula (3b) for $\theta(x)$ below contains a closed form representation of the two nested sums over $k$ in formula (3a) for $\theta(x)$ below.

(3a) $\quad\theta(x)=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\frac{1}{2}+\frac{1}{\pi}\sum\limits_{n=1}^N\mu(n)\left(\sum\limits_{k=1}^{f\ n}\frac{\cos\left(\frac{2 \pi k}{n}\right) \sin\left(\frac{2 \pi k x}{n}\right)}{k}-\frac{1}{2}\sum\limits_{k=1}^{2\ f\ n} \frac{\sin\left(\frac{\pi k x}{n}\right)}{k}\right)$

(3b) $\quad\theta(x)=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\frac{1}{2}+\frac{i}{4 \pi}\sum\limits_{n=1}^N\mu(n) \left(\log\left(1-e^{\frac{2 i \pi (x-1)}{n}}\right)-\log\left(1-e^{\frac{i \pi x}{n}}\right)+\log\left(1-e^{\frac{2 i \pi (x+1)}{n}}\right)-\log\left(1-e^{-\frac{2 i \pi (x-1)}{n}}\right)+\log\left(1-e^{-\frac{i \pi x}{n}}\right)-\log\left(1-e^{-\frac{2 i \pi (x+1)}{n}}\right)\right)$

(4) $\quad\delta'(x)=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\pi\sum\limits_{n=1}^N\frac{\mu(n)}{n^2}\left(\sum\limits_{k=1}^{f\ n} -2 k \left(\sin \left(\frac{2 \pi k (x-1)}{n}\right)+\sin \left(\frac{2 \pi k (x+1)}{n}\right)\right)+\frac{1}{2}\sum\limits_{k=1}^{2\ f\ n} k\ \sin\left(\frac{\pi k x}{n}\right)\right)$

The following formulas are derived from the Fourier convolution defined in formula (2) above using the series representation of $\delta(x)$ defined in formula (1) above. All of the formulas defined below seem to converge for $x\in\mathbb{R}$. Note one of the two nested sums over $k$ in formula (6) below for $e^{-y^2}$ has a closed form representation. Both of the nested sums over $k$ in formulas (5), (8), and (9) below have closed form representations which were not included below because they're fairly long and complex.

(5) $\quad e^{-|y|}=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\sum\limits_{n=1}^N\mu(n)\ n\left(\sum\limits_{k=1}^{f\ n}\frac{2 \left(\cos\left(\frac{2 \pi k (y-1)}{n}\right)+\cos\left(\frac{2 \pi k (y+1)}{n}\right)\right)}{4 \pi^2 k^2+n^2}-\sum\limits_{k=1}^{2\ f\ n}\frac{\cos\left(\frac{\pi k y}{n}\right)}{\pi^2 k^2+n^2}\right)$

(6) $\quad e^{-y^2}=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\sqrt{\pi}\sum\limits_{n=1}^N\frac{\mu(n)}{n}\left(\sum\limits_{k=1}^{f\ n} e^{-\frac{\pi^2 k^2}{n^2}} \left(\cos\left(\frac{2 \pi k (y-1)}{n}\right)+\cos\left(\frac{2 \pi k (y+1)}{n}\right)\right)-\frac{1}{4}\sum\limits_{k=1}^{2\ f\ n} \left(e^{-\frac{\pi k (\pi k+4 i n y)}{4 n^2}}+e^{-\frac{\pi k (\pi k-4 i n y)}{4 n^2}}\right)\right)$

$\qquad\quad=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\sqrt{\pi}\sum\limits_{n=1}^N\frac{\mu (n)}{n}\left(\frac{1}{2} \left(\vartheta_3\left(\frac{\pi (y-1)}{n},e^{-\frac{\pi^2}{n^2}}\right)+\vartheta_3\left(\frac{\pi (y+1)}{n},e^{-\frac{\pi^2}{n^2}}\right)-2\right)-\frac{1}{4} \sum\limits_{k=1}^{2\ f\ n} \left(e^{-\frac{\pi k (\pi k+4 i n y)}{4 n^2}}+e^{-\frac{\pi k (\pi k-4 i n y)}{4 n^2}}\right)\right)$

(7) $\quad\sin(y)\ e^{-y^2}=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\sqrt{\pi } \sum\limits_{n=1}^N\frac{\mu (n)}{n}\left(2 \sum\limits_{k=1}^{f\ n} e^{-\frac{\pi^2 k^2}{n^2}-\frac{1}{4}} \cos\left(\frac{2 \pi k}{n}\right) \sinh\left(\frac{\pi k}{n}\right) \sin\left(\frac{2 \pi k y}{n}\right)-\frac{1}{2}\sum\limits_{k=1}^{2\ f\ n} e^{-\frac{\pi^2 k^2}{4 n^2}-\frac{1}{4}} \sinh\left(\frac{\pi k}{2 n}\right) \sin\left(\frac{\pi k y}{n}\right)\right)$

(8) $\quad\frac{1}{y^2+1}=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\pi\sum\limits_{n=1}^N\frac{\mu (n)}{n}\left(2 \sum\limits_{k=1}^{f\ n} e^{-\frac{2 \pi k}{n}} \cos\left(\frac{2 \pi k}{n}\right) \cos\left(\frac{2 \pi k y}{n}\right)-\frac{1}{2}\sum\limits_{k=1}^{2\ f\ n} e^{-\frac{\pi k}{n}} \cos\left(\frac{\pi k y}{n}\right)\right)$

(9) $\quad\frac{y}{y^2+1}=\underset{\underset{M(N)=0}{N,f\to\infty}}{\text{lim}}\quad\pi\sum\limits_{n=1}^N\frac{\mu(n)}{n}\left(2\sum\limits_{k=1}^{f\ n} e^{-\frac{2 \pi k}{n}} \cos\left(\frac{2 \pi k}{n}\right) \sin\left(\frac{2 \pi k y}{n}\right)-\frac{1}{2}\sum\limits_{k=1}^{2\ f\ n} e^{-\frac{\pi k}{n}} \sin\left(\frac{\pi k y}{n}\right)\right)$

The remainder of this answer illustrates formula (1) for $\delta(x)$ above and some of the other formulas defined above all of which were derived from formula (1). The observational convergence of these derived formulas provides evidence of the validity of formula (1) above.

Figure (1) below illustrates formula (1) for $\delta(x)$ evaluated at $f=4$ and $N=39$. The discrete portion of the plot illustrates formula (1) for $\delta(x)$ evaluates exactly to $2 f$ times the step size of $\theta(x)$ at integer values of $x$ when $|x|\le N$.

Figure (1): Illustration of formula (1) for $\delta(x)$

Figure (2) below illustrates the reference function $\theta(x)$ in blue and formulas (3a) and (3b) for $\theta(x)$ in orange and green respectively where formula (3a) is evaluated at $f=4$ and formulas (3a) and (3b) are both evaluated at $N=39$.

Figure (2): Illustration of formulas (3a) and (3b) for $\theta(x)$ (orange and green)

Figure (3) below illustrates the reference function $\theta(x)$ in blue and formula (3b) for $\theta(x)$ evaluated at $N=39$ and $N=101$ in orange and green respectively.

Figure (3): Illustration of formula (3b) for $\theta(x)$ evaluated at $N=39$ and $N=101$ (orange and green)

Figure (4) below illustrates formula (4) for $\delta'(x)$ above evaluated at $f=4$ and $N=39$. The red discrete portion of the plot illustrates the evaluation of formula (4) for $\delta'(x)$ at integer values of $x$.

Figure (4): Illustration of formula (4) for $\delta'(x)$

Figure (5) below illustrates the reference function $\frac{y}{y^2+1}$ in blue and formula (9) for $\frac{y}{y^2+1}$ above evaluated at $f=4$ and $N=101$.

Figure (5): Illustration of formula (9) for $\frac{y}{y^2+1}$