Let $A$ be a $N \times N$ symmetric positive semidefinite matrix with $N \geq 2$. Let $D$ be a diagonal matrix of dimension $N$. I would like to measure how much $A$ "is far" from $D$, i.e. I am trying to find a way to quantify how $A$ differs from a diagonal matrix. More generally, the aim is to come up with a measure that captures the overall "level of orthogonality" of $A$, such that if I have another positive semidefinite matrix $B$, with the same dimension, I can argue that $A$ is more or less close to being orthogonal than $B$, which means I am interested in a relative statement. To be more explicit: I would like to quantify how $A$ is close to being an orthogonal matrix, and how $B$ is close to being an orthogonal matrix, not how $A$ and $B$ are orthogonal to each other. Any suggestion (even partial) is welcome

$\begingroup$ Could you clarify what you mean by "I can argue that $A$ is more or less close to being orthogonal than $B$"? Do you want to measure how close $A$ and $B$ are to being orthogonal with respect to some inner product? Orthogonal as positive semidefinite matrices (in the sense that $\operatorname{im}(A)$ and $\operatorname{im}(B)$ are orthogonal)? Or do you want to measure how close each of $A$ and $B$ is to being an orthogonal matrix? I assumed the latter, but you accepted an answer which assumes the former. $\endgroup$– J. van Dobben de BruynCommented Jul 19, 2023 at 0:54

5$\begingroup$ Also, are you aware that you asked this question on the forum for professional mathematicians to discuss researchlevel mathematics? So far the question seems to be well received here, but in general I think questions like this might be more appropriate for Mathematics Stack Exchange. In any case, welcome to MathOverflow! $\endgroup$– J. van Dobben de BruynCommented Jul 19, 2023 at 0:57

1$\begingroup$ @J.vanDobbendeBruyn  sometimes there's more to a question than initially meets the eye. Initially, I didn't pay much attention to this one  until the answer by Ryan C came along pointing to a nice construction that I hadn't been aware of (of course, this might mean I have no business on MO). Then I upvoted the question, for having elicited the answer (along, of course, with the answer). $\endgroup$– Michael EngelhardtCommented Jul 19, 2023 at 2:07

1$\begingroup$ You start by speaking about distance from diagonal, but then you seem to be mentioning distance from orthogonal. Is that a mistake? Otherwise, how are those concept related? Do you measure the distance from diagonal of $A^*A$? $\endgroup$– Federico PoloniCommented Jul 19, 2023 at 11:50

1$\begingroup$ It is confusing to mention both A and B if the goal is to measure how close a matrix is from being an orthogonal matrix. If you have defined such a measure, say orthog(X) for any matrix X, then you can apply orthog( ) to any matrices that you wish. $\endgroup$– Daniel AsimovCommented Jul 19, 2023 at 16:47
4 Answers
Q: Is there a measure that captures the overall "level of orthogonality" of two matrices $A$ and $B$.
You can collect the $N^2$ elements of $A$ and $B$ into a pair of vectors $a$, $b$, and then take the inner product $(a,b)=\sum_{ij}\bar{A}_{ij}B_{ij}={\rm tr}\,A^\ast B$. This would generalize the usual measure of orthogonality of pairs of vectors to pairs of matrices.
I understand this to be the motivation for your question of the "distance of a matrix $A$ from a diagonal matrix $D$"; for that distance you could just use the Frobenius norm, ${\rm tr}\,(AD)(AD)^\ast$.

$\begingroup$ this is a great way to tackle the problem, thanks a lot! $\endgroup$ Commented Jul 18, 2023 at 10:54
I read the question differently, so at least one of us must have misunderstood your question. 😅
Q: Is there a measure that captures how close a matrix $A \in \mathbb{R}^{n \times n}$ is to being orthogonal?
A: A real square matrix $A$ is orthogonal if and only if $A^TA = AA^T = I$. So to measure how close a matrix is to being orthogonal, one usually measures something like $\lVert A^TA  I \rVert$ or $\lVert AA^T  I \rVert$ or the maximum of the two, for some matrix norm $\lVert \:\cdot\: \rVert$.
Suppose now that $A$ is positive semidefinite, and we want to measure how close $A$ is to being a positive semidefinite orthogonal matrix. Then we note that the only positive semidefinite orthogonal matrix is $I$, so we could simply compute the value of $\lVert A  I \rVert$ for some matrix norm $\lVert \:\cdot\: \rVert$.
Examples of matrix norms include the Frobenius norm $\lVert A \rVert_F = \sqrt{\operatorname{tr}(A^TA)}$ (already mentioned in Carlo Beenakker's answer), the operator norm $\lVert A \rVert_{\operatorname{op}} = \sup_{\lVert x \rVert = 1} \lVert Ax \rVert$, or the Schatten $p$norms $\lVert A \rVert_p = \left(\sum_{i=1}^n \sigma_i^p\right)^{1/p}$, where $\sigma_1,\ldots,\sigma_n$ are the singular values of $A$. The choice of norm depends on the context.

$\begingroup$ I see, I might have been sloppy in my notation. Initially I was thinking about how to quantify how a group of random variable are at being orthogonal among each other. Hence, focusing on the covariance matrix, I thought about a way to quantify how this is close to being orthogonal, which would represent the case where the random variables are also orthogonal to each other. Since a covariance matrix is symmetric positive semidefinite, but not the identity matrix even with orthogonal random variables due to the variances on the main diagonal, I was referring to a generic diagonal matrix $D$ $\endgroup$ Commented Jul 19, 2023 at 9:56

$\begingroup$ Such matrix $D$ would indeed represent the covariance matrix of orthogonal random variables. I hope now this is clear enough. If the consensus is I should adjust the question giving this detail, I will be happy to do it $\endgroup$ Commented Jul 19, 2023 at 9:57

$\begingroup$ @Matteo thanks, I'm starting to get the picture. 🙂 In the comments below the question, you spoke about normalizing the random variables so that they have mean $0$ and standard deviation $1$. In that case the matrix $D$ would simply be the identity, so you could just compute $d(A, I)$ for some appropriate distance measure $d$, depending on your application. When the random variables are not normalized, the problem becomes trickier, and I'm not sure how to deal with that. I guess you could measure the difference between $A$ and the diagonal matrix whose entries are given by the diagonal of $A$? $\endgroup$ Commented Jul 19, 2023 at 20:54
A good tool for real symmetric positivedefinite matrices is to let $d(A,B)=$ the square root of the sum of the squares of the natural logarithms of the generalized eigenvalues of $A$ and $B$; that is, $\sqrt{\sum \ln^2(\lambda)}$ of the $N$ solutions $\lambda$ of $\det(A\lambda B)=0$.
This $d$ has a number of useful properties, such as $d(A,B) = d(A^{1},B^{1})$, and $d(A,B) = d(XAX^T, XBX^T)$ for any $X$ in $GL(N,\mathbb{R})$.
I first learned about it from from Förstner W., Moonen B. A Metric for Covariance Matrices. In: Grafarend E.W., Krumm F.W., Schwarze V.S. (eds) GeodesyThe Challenge of the 3rd Millennium. Springer, Berlin, Heidelberg. https://doi.org/10.1007/9783662052969_31, but the core of the proof (that this formula matches the natural torsionfree connection on the space) is in Kobayashi & Nomizu.

1$\begingroup$ Those are interesting references. Going by the description in your first sentence, could it be that there is a logarithm missing under your square root? $\endgroup$ Commented Jul 19, 2023 at 1:44

1$\begingroup$ @MichaelEngelhardt yes, thank you, i have fixed it. The squared logarithm has the useful property that multiplying or dividing by an overall scale factor makes equal change to the distance. $\endgroup$– Ryan CCommented Jul 19, 2023 at 2:52

1$\begingroup$ this is also a very interesting and clever way to think about it, with the addition of having the properties you mention. $\endgroup$ Commented Jul 19, 2023 at 9:43
Another measure of "how far one positive definite matrix is from another" is given by the KullbackLeibler divergence. This approach might be especially suitable in a probability context  you mentioned covariances matrices in a comment, so this might be helpful for you.
You can think of a positive definite matrix as parametrising a Gaussian (with zero mean), and to calculate the divergence between the two Gaussians that they quantify. This works out to be
$$ D(P\parallel Q) = \operatorname{Tr}(PQ^{1})  \log(PQ^{1})  n, $$ where $n$ is the dimensionality of the matrices and $\cdot$ is the determinant. (This is equation 4.151 in Amari's book 'Information Theory and its Applications'.)
This will be 0 if the matrices are equal, and in your case it probably makes most sense to have $Q$ as the diagonal one.
It may still make sense to use this formula even if you're not working with Gaussians, since if you want to you can forget the probability interpretation and just regard it as a Bregman divergence in the space of positive definite matrices.