This is probably going to lead nowhere, but maybe it be possible to use the matrix logarithm to invert matrices? For positive definite matrices, we have that the logarithm exists and $$ \log(A^{-1})= - \log(A) $$ So very brutally applying the first taylor approximation everywhere (assuming this works for matrices) $$ A^{-1} = \exp(-\log(A)) \approx \exp(- (A- \mathbb{I}) \approx \mathbb{I} + (\mathbb{I} - A) = 2\mathbb{I} -A $$ This is likely to only work well for matrices where the eigenvalues are all close to $1$, which doesn't really fit my use case so I don't really want to pursue this further. But maybe this is useful somewhere?
Has this concept come up before? Are there more intelligent approximations than the first taylor approximation?