So there are two parameters $\alpha$ and $\beta$ and a function $V(\alpha,\beta)$, obtained from your first equation by substituting $S=\alpha^2/\beta$ and $\mu=2\beta/\alpha$. With some effort we can make a Taylor series expansion of this function around $\beta=0$, to second order. The result is not pretty:
$$\frac{2}{S}V(\alpha,\beta)=\left(1 + 2 \alpha (\alpha - \sqrt{1 + \alpha^2})\right) + \left(-3 - 4 \alpha^2 + \alpha^{-1}\sqrt{ 1 + \alpha^2}+ 4 \alpha \sqrt{1 + \alpha^2}\right) \beta + \tfrac{1}{3} \left(34 + 3\alpha^{-2} + 16 \alpha^2 - 14\alpha^{-1} \sqrt{1 + \alpha^2}- 2 \alpha (13 + 8 \alpha^2)(1 + \alpha^2)^{-1/2}\right) \beta^2+{\rm order}(\beta^3)$$
I can imagine that the authors of the paper you mention were not happy with this formula, and made one further approximation, even though they did not explicitly say so. They write $|\alpha|>1$, but in fact they assume $|\alpha|\gg 1$, retaining terms of order $\beta^2$ and terms of order $1/\alpha^2$, but neglecting terms of order $(\beta/\alpha)^2$. That gives the desired result,
$$\frac{2}{S}V(\alpha,\beta)=\tfrac{1}{4}\alpha^{-2}+\tfrac{2}{3}\beta^2+\text{higher order terms}.$$
(So this is a Taylor series in $\epsilon$ if you identify $\beta=\beta'\epsilon$, $\alpha=\alpha'/\epsilon$; the "higher order terms" are all of order $\epsilon^3$ or smaller.)