I am minimzing a function $F(x) = \mathbb E(f(x,\Xi))$ where $\Xi$ is some random value, by a stochastic gradient descent that generates a random number $\xi$ from the distribution of $\Xi$ at each iterate and uses the gradient $$g_\xi(x) = \nabla_f(x,\xi).$$

This works well, and there are a lot of theory that essentialy says that if $g(x) = \mathbb E(g_\Xi(x))$ is bouded, if $F$ is strongly convex and smooth, etc. we have some convergence results.

The classical framework deals with the case where $f(.,\xi)$ are all convex functions but with different minimums in $x$. In my special case, all $f(.,\xi)$ are convex, and have minimums that are not a point, but rather a variety (x is multivariate), and the global minimum of $F$, which is a point, lies at the intersection of the minimas of $f(.,\xi)$'s.

This is a huge plus for the convergence of the stochastic gradient descent, however I am not sure how to deal with it. Was it treated somewhere in the literature ?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.