I don't think we need the "modulo" anymore: $\delta'_{ij}=1$ if $i=j\in\{0,1,2,\ldots M-1\}$ $\text{modulo}\,(N+M)$. I believe just $\delta'_{ij}=1$ if $i=j\in\{0,1,2,\ldots M-1\}$ will suffice.

I edited the question to add $M<N$, also made a small correction to the conditions for $N_k$ to reflect that. Answer matches perfectly with simulation. Thank you.

$Y=\frac{1}{2}\sigma ^2N\text{tr}\mathbf{AA}^H$ does not match with simulation.

Thank you for the answer. I'm trying to match the analytical solution you gave with simulation, currently they don't match. I'm trying to figure out why.

@CarloBeenakker There are $N$ zeros between $c_0$ and $c_{M-1}$ in the first row. I guess the confusion comes by thinking that $M-1$ is the index of $c_{M-1}$ in the first row of the matrix. I agree it's confusing, but I cannot think of any other way to write it. The way I'm constructing the Toeplitz matrix is I'm taking an $M$ long vector $[c_1,...,c_{M-1}]$ and shifting it through every row of the matrix while filling the remaining $N$ spots in every row with zeros.

@CarloBeenakker Sorry this part was not clear, I updated the question. $[c_0,...,c_{M-1}]$ has length $M$ and the rest of rows and columns are filled with $N$ zeros in the matrix.