I am also interested in if there is any theoretical analysis to the stability of computing inverse ill-conditioned matrices.

Thanks Brian, very intuitive examples. The stability is exactly what I am concerned. The high level picture is that I want to solve a document clustering problem which can be finally converted to solve a set of least square problems. Then I get the potentially ill-conditioned matrix M. And now I need to find a good way to solve this problem.

M is guaranteed to be a positive semi-definite (PSD) matrix. There is no guarantee that all eigenvalues are positive. Then, there is a potential problem with my paper because of my careless during the formulation (dmml.asu.edu/users/xufei/Papers/ICDM2011.pdf page 3). So I want to find a ``good'' solution in terms of stability for this matrix inversion problem.

I mean stability. Will these two approaches differ significantly. My application uses the inverse matrix of the PSD matrix to solve the minimization problem, $\min ~ \| M^{1/2} L - M^{-1/2} N \|_F^{2} $ Here matrices M and N are known, and L is to be computed. I minimize the Frobenious norm, which can be solved by a set of least square problems. So different inverse matrices would affect the final solution of L. I would like to see any references discussing the effect of different transformation approaches (from PSD to PD).