Return to Answer

Here is a proof that holds under a slight additional hypothesis, namely, suppose that we also have $\|A^*B^*-I\| < \epsilon$. The conclusion is also stronger than the one requested, as under this additional hypothesis, the claim holds for all unitarily invariant norms, not just Schatten-$p$ norms.

Theorem ([Thm. IX.7.2, Bha97]). Let $A=UP$ be the polar decomposition of $A$. Then, for every unitarily invariant norm $\|\cdot\|$ and every unitary matrix $V$ \begin{equation*} \|A-U\| \le \|A-V\|. \end{equation*}

Applying this theorem to the direct sum $A\oplus (-A^*)$ we have \begin{equation*} \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - \begin{pmatrix} U & 0\\ 0 & -U^* \end{pmatrix}\right\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - V^*\right\| \end{equation*} for any unitary matrix $V$. Since $B$ is a contraction, we can dilate it to the unitary matrix \begin{equation*} V := \begin{pmatrix} B & (I-BB^*)^{1/2}\\ (I-B^*B)^{1/2} & -B^* \end{pmatrix}. \end{equation*}

Since the norms involved are unitarily invariant, we therefore obtain \begin{equation*} \|(A-U)\oplus(U-A)^*\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix}V - I\right\| = \|(AB-I)\oplus (A^*B^*-I)\|. \end{equation*} But notice that $s_j(A-U)=s_j(U^*-A^*)$, for each singular value $s_j$ for $1\le j \le n$. Therefore, we can conclude the following weak-majorization between the singular values: \begin{equation*} s(A-U) \prec_w \max\{s_j(AB-I),s_j(A^*B^*-I)\}_{j=1}^n. \end{equation*} From this under our hypotheses $\|AB-I\|<\epsilon$ and $\|A^*B^*-I\| < \epsilon$, it follows that $\|A-U\| < \epsilon$.

Note: It should be possible to get rid of the extra assumption, but am not sure as of now.

EDIT The claim below is false (stupid error, as noted in the comments). I will leave the answer below as a potential approach; am checking if the idea can be fixed!

Here is a proof that holds under a slight additional hypothesis, namely, suppose that we also have $\|A^*B^*-I\| < \epsilon$. The conclusion is also stronger than the one requested, as under this additional hypothesis, the claim holds for all unitarily invariant norms, not just Schatten-$p$ norms.

Theorem ([Thm. IX.7.2, Bha97]). Let $A=UP$ be the polar decomposition of $A$. Then, for every unitarily invariant norm $\|\cdot\|$ and every unitary matrix $V$ \begin{equation*} \|A-U\| \le \|A-V\|. \end{equation*}

Applying this theorem to the direct sum $A\oplus (-A^*)$ we have \begin{equation*} \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - \begin{pmatrix} U & 0\\ 0 & -U^* \end{pmatrix}\right\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - V^*\right\| \end{equation*} for any unitary matrix $V$. Since $B$ is a contraction, we can dilate it to the unitary matrix \begin{equation*} V := \begin{pmatrix} B & (I-BB^*)^{1/2}\\ (I-B^*B)^{1/2} & -B^* \end{pmatrix}. \end{equation*}

Since the norms involved are unitarily invariant, we therefore obtain \begin{equation*} \|(A-U)\oplus(U-A)^*\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix}V - I\right\| = \|(AB-I)\oplus (A^*B^*-I)\|. \end{equation*} But notice that $s_j(A-U)=s_j(U^*-A^*)$, for each singular value $s_j$ for $1\le j \le n$. Therefore, we can conclude the following weak-majorization between the singular values: \begin{equation*} s(A-U) \prec_w \max\{s_j(AB-I),s_j(A^*B^*-I)\}_{j=1}^n. \end{equation*} From this under our hypotheses $\|AB-I\|<\epsilon$ and $\|A^*B^*-I\| < \epsilon$, it follows that $\|A-U\| < \epsilon$.

Note: It should be possible to get rid of the extra assumption, but am not sure as of now.

Here is a proof that holds under a slight additional hypothesis, namely, suppose that we also have $\|A^*B^*-I\| < \epsilon$. The conclusion is also stronger than the one requested, as under this additional hypothesis, the claim holds for all unitarily invariant norms, not just Schatten-$p$ norms.

Let $A=UP$ be the polar decomposition of $A$. Then, we know that for unitarily invariant norms \begin{equation*} \|A-U\| \le \|A-V\| \le \|A+U\|, \end{equation*} for any unitary matrix $V$. It follows therefore that \begin{equation*} \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - \begin{pmatrix} U & 0\\ 0 & -U^* \end{pmatrix}\right\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - V^*\right\| \end{equation*} for any unitary matrix $V$. Since $B$ is a contraction, we can dilate it to a unitary matrix, e.g., \begin{equation*} V := \begin{pmatrix} B & *\\ * & -B^* \end{pmatrix}, \end{equation*} where the $*$ denotes unspecified entries.

Since the norms involved are unitarily invariant, we have \begin{equation*} \|(A-U)\oplus(U-A)^*\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix}V - I\right\| = \|(AB-I)\oplus (A^*B^*-I)\|. \end{equation*} But notice that $s_j(A-U)=s_j(U^*-A^*)$, for each singular value $s_j$ for $1\le j \le n$. Therefore, we can conclude the following weak-majorization between the singular values: \begin{equation*} s(A-U) \prec_w \max\{s_j(AB-I),s_j(A^*B^*-I)\}_{j=1}^n. \end{equation*} From this under our hypotheses $\|AB-I\|<\epsilon$ and $\|A^*B^*-I\| < \epsilon$, it follows that $\|A-U\| < \epsilon$.

Note: It should be possible to get rid of the extra assumption, but am not sure as of now.

Here is a proof that holds under a slight additional hypothesis, namely, suppose that we also have $\|A^*B^*-I\| < \epsilon$. The conclusion is also stronger than the one requested, as under this additional hypothesis, the claim holds for all unitarily invariant norms, not just Schatten-$p$ norms.

Theorem ([Thm. IX.7.2, Bha97]). Let $A=UP$ be the polar decomposition of $A$. Then, for every unitarily invariant norm $\|\cdot\|$ and every unitary matrix $V$ \begin{equation*} \|A-U\| \le \|A-V\|. \end{equation*}

Applying this theorem to the direct sum $A\oplus (-A^*)$ we have \begin{equation*} \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - \begin{pmatrix} U & 0\\ 0 & -U^* \end{pmatrix}\right\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - V^*\right\| \end{equation*} for any unitary matrix $V$. Since $B$ is a contraction, we can dilate it to the unitary matrix \begin{equation*} V := \begin{pmatrix} B & (I-BB^*)^{1/2}\\ (I-B^*B)^{1/2} & -B^* \end{pmatrix}. \end{equation*}

Since the norms involved are unitarily invariant, we therefore obtain \begin{equation*} \|(A-U)\oplus(U-A)^*\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix}V - I\right\| = \|(AB-I)\oplus (A^*B^*-I)\|. \end{equation*} But notice that $s_j(A-U)=s_j(U^*-A^*)$, for each singular value $s_j$ for $1\le j \le n$. Therefore, we can conclude the following weak-majorization between the singular values: \begin{equation*} s(A-U) \prec_w \max\{s_j(AB-I),s_j(A^*B^*-I)\}_{j=1}^n. \end{equation*} From this under our hypotheses $\|AB-I\|<\epsilon$ and $\|A^*B^*-I\| < \epsilon$, it follows that $\|A-U\| < \epsilon$.

Note: It should be possible to get rid of the extra assumption, but am not sure as of now.

Here is a proof that holds under a slight addition hypothesis, namely, suppose in addition we also have $\|A^*B^*-I\| \le \epsilon$. The conclusion is also stronger than the one requested in question, as under this additional hypothesis, the claim holds for all unitarily invariant norms, not just Schatten-$p$ norms.

Let $A=UP$ be the polar decomposition of $A$. Then, we know that for unitarily invariant norms \begin{equation*} \|A-U\| \le \|A-V\| \le \|A+U\|, \end{equation*} for any unitary matrix $V$. It follows therefore that \begin{equation*} \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - \begin{pmatrix} U & 0\\ 0 & -U^* \end{pmatrix}\right\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - V^*\right\| \end{equation*} for any unitary matrix $V$. Since $B$ is a contraction, we can dilate it to a unitary matrix, e.g., \begin{equation*} V := \begin{pmatrix} B & *\\ * & -B^* \end{pmatrix}, \end{equation*} where the $*$ denotes unspecified entries. Since the norms involved are unitarily invariant, we have \begin{equation*} \|(A-U)\oplus(U-A)^*\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix}V - I\right\| = \|(AB-I)\oplus (A^*B^*-I)\|. \end{equation*} But notice that $s_j(A-U)=s_j(U^*-A^*)$. And we assumed that $\|AB-I\|<\epsilon$ and $|A^*B^*-I\| < \epsilon$. It thus follows that $\|A-U\| < \epsilon$.

Note: Perhaps the proof can be simplified and made more elegant while getting rid of the extra assumption.

Here is a proof that holds under a slight additional hypothesis, namely, suppose that we also have $\|A^*B^*-I\| < \epsilon$. The conclusion is also stronger than the one requested, as under this additional hypothesis, the claim holds for all unitarily invariant norms, not just Schatten-$p$ norms.

Let $A=UP$ be the polar decomposition of $A$. Then, we know that for unitarily invariant norms \begin{equation*} \|A-U\| \le \|A-V\| \le \|A+U\|, \end{equation*} for any unitary matrix $V$. It follows therefore that \begin{equation*} \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - \begin{pmatrix} U & 0\\ 0 & -U^* \end{pmatrix}\right\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix} - V^*\right\| \end{equation*} for any unitary matrix $V$. Since $B$ is a contraction, we can dilate it to a unitary matrix, e.g., \begin{equation*} V := \begin{pmatrix} B & *\\ * & -B^* \end{pmatrix}, \end{equation*} where the $*$ denotes unspecified entries. 

Since the norms involved are unitarily invariant, we have \begin{equation*} \|(A-U)\oplus(U-A)^*\| \le \left\|\begin{pmatrix} A & 0\\ 0 & -A^* \end{pmatrix}V - I\right\| = \|(AB-I)\oplus (A^*B^*-I)\|. \end{equation*} But notice that $s_j(A-U)=s_j(U^*-A^*)$, for each singular value $s_j$ for $1\le j \le n$. Therefore, we can conclude the following weak-majorization between the singular values: \begin{equation*} s(A-U) \prec_w \max\{s_j(AB-I),s_j(A^*B^*-I)\}_{j=1}^n. \end{equation*} From this under our hypotheses $\|AB-I\|<\epsilon$ and $\|A^*B^*-I\| < \epsilon$, it follows that $\|A-U\| < \epsilon$.

Note: It should be possible to get rid of the extra assumption, but am not sure as of now.