$\renewcommand{\S}{\mathcal S}\newcommand{\Si}{\Sigma}$Let us assume that you meant to write $|S| = M(n)$ instead of $|S| = M(k)$ (otherwise, the question does not seem to make sense).
For $j\in[n]:=\{1,\dots\}$, let, as usual, $\binom{[n]}j$ stand for the set of all subsets of cardinality $j$ of the set $[n]$, and then let $\nu_{n,j}$ denote the cardinality of the set, say $\S_{n,j}$, of all set $S\in\binom{[n]}j$ without neighbors.
The set $\S_{n,j}$ is in the standard bijective correspondence with the set, say $\Si_{n,j}$, of all sequences in $\{0,1\}^n$ with exactly $j$ $1$'s such that no $1$ is followed by a $1$. So, in each sequence in $\Si_{n,j}$, each $1$ is followed by a $0$, except that a $1$ in the last, $n$th position is not followed by anything. To erase this exception, let us append an additional fixed $0$ in position $n+1$. So, making $10$ a letter, we see that the cardinality $\nu_{n,j}$ of the set $\Si_{n,j}$ is the number of words in the alphabet $\{10,0\}$ of length $n+1-j$ containing exactly $j$ instances of the character $10$ (as the initial $n+1$ positions for $j$ $1$'s and $n+1-j$ positions for $0$'s got reduced to $n+1-j$ positions for $j$ $10$'s and $n+1-2j$ $0$'s).
Thus,
\begin{equation}
\nu_{n,j}=\binom{n+1-j}j.
\end{equation}
So, $M(n)$ is the smallest $j$ such that
\begin{equation}
\binom{n+1-j}j\le\frac12\binom nj.
\end{equation}
For $j\asymp\sqrt n$, we have
\begin{equation}
\binom nj=n^j\,\prod_{i=0}^{j-1}\Big(1-\frac in\Big)=n^j\,\exp-\frac{j^2}{2+o(1)n}
\end{equation}
and, similarly,
\begin{equation}
\binom{n+1-j}j=n^j\,\exp-\frac{3j^2}{2+o(1)n}.
\end{equation}
We conclude that
\begin{equation}
M(n)\sim\sqrt{n\ln 2}.
\end{equation}