Just use a backward stable algorithmYou are essentially using normal equations to solve the least-squares problem $\min \|W^{1/2}(Ax-b)\|_2$ resulting from IRLS. For instanceNormal equations are known not to be a backward stable algorithm. Use other standard algorithms for LS problems instead, uselike the QR factorization or the SVD of $W^{1/2}A$, instead of forming normal equations like you. Those are doing hereguaranteed to be backward stable.