Let $D_1$ be the answer conditioned on the leftmost point being at 0 and the rightmost point at 1 (colors irrelevant). Let $0 = x_1 \leq \ldots \leq x_{n + m} = 1$ be the ordered coordinates of points. We can see that $s_i = x_{i + 1} - x_i$ are equidistributed subject to $s_1 + \ldots + s_{n + m - 1} = 1$, and coloring is independent of $s_i$. The probability that there are exactly $k$ adjacent color changes in the sequence is $$\frac{{n - 1 \choose \lceil \frac{k - 1}{2} \rceil}{m - 1 \choose \lfloor \frac{k - 1}{2} \rfloor} + {m - 1 \choose \lceil \frac{k - 1}{2} \rceil}{n - 1 \choose \lfloor \frac{k - 1}{2} \rfloor}}{n + m \choose n},$$ since the sequence either starts with a white group, and has $\lceil (k + 1) / 2 \rceil$ white groups and $\lfloor (k + 1) / 2 \rfloor$ black groups, or vice versa.
Conditioned on the colouring, the smallest distance is equal to the minimum of $k$ instances of $s_i$ (it is irrelevant which ones due to symmetry). For $0 \leq t \leq 1 / k$, the probability of $\min(s_1, \ldots, s_k) \leq t$ is $1 - (1 - tk)^{n + m - 2}$. Integrating with density, we have $$E\min(s_1, \ldots, s_k) = \int_0^{1 / k} k(n + m - 2)(1 - tk)^{n + m - 3} tdt =$$ $$(n + m - 2) \int_0^1 z^{n + m - 3}\frac{1 - z}{k} dz = \frac{1}{k(n + m - 1)}$$ (note that the answer is correct in the special case $k = n = m = 1$ where some transformations were illegal). Hence $$D_1 = \frac{1}{(n + m - 1){n + m \choose n}}\sum_{k = 1}^{\infty}\frac{{n - 1 \choose \lceil \frac{k - 1}{2} \rceil}{m - 1 \choose \lfloor \frac{k - 1}{2} \rfloor} + {m - 1 \choose \lceil \frac{k - 1}{2} \rceil}{n - 1 \choose \lfloor \frac{k - 1}{2} \rfloor}}{k}$$
Finally, integrating by the span $s$ between extreme points we obtain the complete answer $$D = \int_0^1 (n + m)(n + m - 1)(1 - s)s^{n + m - 2} \cdot sD_1 ds = \frac{n + m - 1}{n + m + 1}D_1 =$$ $$\frac{1}{(n + m + 1){n + m \choose n}}\sum_{k = 1}^{\infty}\frac{{n - 1 \choose \lceil \frac{k - 1}{2} \rceil}{m - 1 \choose \lfloor \frac{k - 1}{2} \rfloor} + {m - 1 \choose \lceil \frac{k - 1}{2} \rceil}{n - 1 \choose \lfloor \frac{k - 1}{2} \rfloor}}{k}$$
When $n = m = 2$, we have $D = \frac{1}{5 \cdot 6}(\frac{2}{1} + \frac{2}{2} + \frac{2}{3}) = \frac{1}{30} \cdot \frac{11}{3} = \frac{11}{90}$, which matches the computations in the comments. Not sure if the sum can be simplified further, still the answer is feasible to compute.
The probability of the smallest distance being at least $d$ can be found along the same lines by computing $$\frac{(n + m)(n + m - 1)}{n + m \choose n}\sum_{k = 1}^{\infty}\left({n - 1 \choose \lceil \frac{k - 1}{2} \rceil}{m - 1 \choose \lfloor \frac{k - 1}{2} \rfloor} + {m - 1 \choose \lceil \frac{k - 1}{2} \rceil}{n - 1 \choose \lfloor \frac{k - 1}{2} \rfloor}\right) \times \\ \int_0^1 (1 - s)s^{n + m - 2} \cdot \max(0, 1 - \frac{kd}{s})^{n + m - 2}ds$$
NIntegrate[ Min[Abs[x1 - y1], Abs[x1 - y2], Abs[x2 - y1], Abs[x2 - y2]], {x1, 0, 1}, {x2, 0, 1}, {y1, 0, 1}, {y2, 0, 1}]
and got $0.122377$, a bit closer to the theoretical result than the stochastic simulation was. $\endgroup$4 Integrate[ Min[Abs[x1 - y1], Abs[x1 - y2], Abs[x2 - y1], Abs[x2 - y2]], {x1, 0, 1}, {x2, 0, x1}, {y1, 0, 1}, {y2, 0, y1}]
$= 11/90$. $\endgroup$