No, those sets are not open. Indeed, take any non-isolated point $x$ in $X$ and a sequence $(x_n)_{n\in\omega}$ that converges to $x$. For every $n$, consider the sequence of sign measure $\mu_n=\delta_{x}-\delta_{x_n}$, where $\delta_p$ is the Dirac measures at a point $p\in X$. The definition of the Kantorovich-Rubinshtein norm guarantees that $\|\mu_n\|_0$ tends to zero as $n\to\infty$. For $M=\frac 12$ and $K=\{x\}$, the zero measure belongs to the sets $\{\mu\in\mathcal M(X):|\mu|(K)<M\}\subseteq\{\mu\in\mathcal M(X):\mu(K)<M\}$ but for every $n\in\omega$ the measure $\mu_n$ does not belong to those two sets, which implies that these sets are not open in $\mathcal M(X)$.