When I try to solve a optimization problem by Riemannian stochastic variance reduced gradient algorithm(RSVRG), the formulation of problem like $\frac{1}{N}\sum_{i=1}^Nf_i(x)$ and $f_i(x)$ is a non-convex function. It can be expressed like $||\mathbf{x}||^2 - \gamma||a\mathbf{x}||^2$.
I found that the value of the function will get bigger and bigger with the iteration process. According to Stochastic Variance Reduction for Nonconvex Optimization's pseudocode, I think the reason for this problem is that the full gradient from the outer loop does not change in the inner loop, causing the x value of the inner loop to keep moving in a wrong direction.
I want to know whether this understanding is correct and whether there is a solution to this problem (I am not familiar with this field, I am afraid that limiting the gradient of the inner loop will change the original convergence).