First assume $\zeta_i=0$, which conditions guarantee that $w^*$ is a stable fixed point? If it's not a stable fixed point for noise-free case, then the process will not be ergodic with additive noise.

Consider

\begin{equation}
\Sigma=\left(
\begin{matrix}
1 & 0 \\
0 & 2
\end{matrix}
\right)
\end{equation}

One condition on step size $\eta=\alpha$ which guarantees convergence

$$\frac{2}{\alpha}<\text{Tr}(\Sigma)+2\|H\|$$

Which gives that 
$$\alpha<\frac{2}{7}$$

However, this condition is too strict. The necessary and sufficient condition is that

$$\alpha<\frac {4} {9 + \sqrt {17}}$$

This [note](http://yaroslavvb.com/convergence.pdf) derives necessary and sufficient condition above for a general Gaussian. I haven't seen anyone else derives this, please correct me if it occurs in literature.



---
Background for non-Gaussian case:

It can be shown that in the case of 1 dimension and deterministic $x$, the following condition on $\alpha$ is necessary and sufficient for convergence
\begin{equation}\label{supersimple}
\alpha x^4 < 2 x^2
\end{equation}

Since we have $h=x^2$ for Hessian $h$, this reduces to the well known bound on convergent learning rate: $\alpha < 2/h$

In the case of stochastic x, the following is necessary and sufficient

\begin{equation}\label{eq:0}
    \alpha E[x^4] < 2 E[x^2]
\end{equation}

For the case of $x$ being distributed as standard normal, this gives $2/(3h)$ for the largest learning rate, three times smaller than what's allowed in deterministic case 

For the case of $d$ dimensions, the following is a sufficient condition, with $\prec$ indicating Loewner order\footnote{assumption A.6 in Bach [paper](https://www.di.ens.fr/~fbach/aistats_defossez_bach_with_supp.pdf)

\begin{equation}\label{eq:1}
  \alpha E[xx'xx'] \prec E[xx']  
\end{equation}

The right-hand side can be tightened to
\begin{equation}\label{eq:1x}
  \alpha E[xx'xx'] \prec 2 E[xx']  
\end{equation}

Defossez, Bach [showed](https://www.di.ens.fr/~fbach/aistats_defossez_bach_with_supp.pdf) that the following optimization over symmetric matrices gives sufficient condition for convergence, and conjectured it to also be necessary (Lemma 1 of \citep{Defossez2015-fr})

\begin{equation}\label{eq:2}
    \frac{1}{\alpha} < \sup_{A\in \mathcal{S}(R^d)} \frac{E[(x'Ax)^2]}{2 E[x'A^2 x] }
\end{equation}

We can show this to be equivalent to the following positive semi-definite constraint
\begin{equation}\label{eq:3}
    \alpha E[xx' \otimes xx'] \prec E[xx'\otimes I] + E[I\otimes xx']
\end{equation}


Most recently, [Jain](https://arxiv.org/abs/1610.03774) generalized last Eq to batch sizes beyond 1 and formally showed it to be a necessary condition for convergence.