# Characterization of the behavior of the residuals in conjugate gradient

In conjugate gradient method for solving symmetric positive definite linear system $$Ax=b$$, which can also be regarded as a convex optimization problem $$\dfrac{1}{2} x'Ax - x'b$$, the $$A$$-norm of the error $$\|x_k-x_\star\|_A$$ at iterate $$k$$ is monotonically decreasing and converges linearly and we have the classical bound (or more sophisticated analysis)

$$\|x_k-x_\star\|_A \leq 2 \left(\dfrac{\sqrt{\kappa} - 1}{\sqrt{\kappa} + 1}\right)^k \|x_0-x_\star\|_A$$

where $$x_\star$$ is the solution of the linear system $$Ax_{\star}=b$$ and $$\kappa$$ is the condition number of $$A$$. However, the norm of residuals $$r_k = b-A x_k$$ is not necessarily monotonically decreasing but it still converges to zero.

My question is: Is there a convergence bound or some qualitative characterization of $$\|r_k\|$$?

$$\|r_k\|=\|A(x_*-x_k)\|\le\|A^{1/2}\|\,\|A^{1/2}(x_*-x_k)\| \\ =\|A\|^{1/2}\|x_*-x_k\|_A \le 2\|A\|^{1/2} \left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^k \|x_0-x_*\|_A.$$