To simplify the notation I denote $\tilde{K}=K\sigma/D$ and $\tilde{\omega}=\omega/D$.
First expand \begin{align} &n(\psi , \omega)= \exp \bigl(-\tilde{K} +\tilde{\omega} \psi+\tilde{K} \cos \psi\bigr) n(0 , \omega)\nonumber\\ &\times\biggl[1+\frac{\bigl(e^{-2 \pi \tilde{\omega}}-1\bigr) \int_{0}^{\phi} e^{(-\tilde{\omega} \phi-\tilde{K} \cos \phi)} d \phi}{\int_{0}^{2 \pi} e^{(-\tilde{\omega} \phi-\tilde{K} \cos \phi)} d \phi} \biggr] \end{align} to fourth order in $K$ and evaluate $$\int_0^{2\pi}n(\psi,\omega)\cos(\psi)\,d\psi=\pi n(0,\omega)\bigl[\frac{\tilde{K}}{1+\tilde{\omega}^2}-\frac{\tilde{K}^2}{(1+\tilde{\omega}^2)^2} +\frac{ \tilde{K}^3 \bigl(3 \tilde{\omega}^4+2 \tilde{\omega}^2+5\bigr)}{2 \bigl(\tilde{1+\omega}^2\bigr)^3 \bigl(4+\tilde{\omega}^2\bigr)}+{\cal O}(K^4)\bigr].$$
The function $n(0,\omega)$ is determined by the normalisation $$\int_0^{2\pi}n(\psi,\omega)\,d\psi=1,$$ which gives to fourth order in $K$ the equation $$n(0,\omega)=\frac{1}{2 \pi }+\frac{\tilde{K}}{2 \pi \bigl(\tilde{\omega}^2+1\bigr)}-\frac{\tilde{K}^2}{4\pi}\frac{\tilde{\omega}^2-2}{(\tilde{\omega}^2+1)(\tilde{\omega}^2+4)}-\frac{\tilde{K}^3}{4\pi}\frac{ \tilde{\omega}^4-17 \tilde{\omega}^2+6}{ \bigl(\tilde{\omega}^2+1\bigr)^2 \bigl(\tilde{\omega}^2+4\bigr) \bigl(\tilde{\omega}^2+9\bigr)}+{\cal O}(K^4).$$ We thus arrive at $$\int_0^{2\pi}n(\psi,\omega)\cos(\psi)\,d\psi=\tfrac{1}{2}\tilde{K}\frac{1}{1+\tilde{\omega}^2}+\tfrac{1}{4}\tilde{K}^3\frac{2\tilde{\omega}^2-1}{(1+\tilde{\omega}^2)^2(4+\tilde{\omega}^2)}+{\cal O}(K^4).$$ This then gives the equation for $\sigma$, $$ \sigma= \tfrac{1}{2} K \sigma\int_{-\infty}^{\infty} g\left(D \omega+\omega_{0}\right) \frac{d \omega}{\omega^{2}+1}$$ $$\qquad-\frac{K^{3} \sigma^{3}}{4 D^{3}} \int_{-\infty}^{\infty} g\left(D \omega+\omega_{0}\right)\left\{\frac{1}{\omega^{2}+4}-\frac{\omega^2}{\left(\omega^{2}+1\right)^{2}}\right\} d \omega+{\cal O}(K^4). $$$$\qquad-\frac{K^{3} \sigma^{3}}{4 D^{3}} \int_{-\infty}^{\infty} g\left(D \omega+\omega_{0}\right)\left\{\frac{1}{\omega^{2}+4}-\frac{\color{red}{\omega^2}}{\left(\omega^{2}+1\right)^{2}}\right\} d \omega+{\cal O}(K^4). $$ The term of order $K$ agrees with the formula in the OP, the term of order $K^3$ is slightly different, the OP has $\omega$ instead of the red $\omega^2$ in the numerator of the second term in curly brackets. I think the whole expression should bethis is a typo (the integrand is an even function of $\omega$, so I think this is a typo).