Is it impossible for determinants of these matrices to both be negative?

Question

Suppose $A,B \in M_{n}(\Bbb{R})$ such that $A = \left[C_{1}\middle|\frac{I}{0\dots0}\right], B= \left[C_{2}\middle|\frac{I}{0\dots0}\right]$ , where $A$ and $B$ have different first columns (represented as $C_{1}, C_{2}$).

Thus we have $B = A+ \xi e_{1}^T$, where $\xi$ is a $n \times 1$ column vector and $e_{1}^{T} = [1, 0,\ldots,0]$

$\textbf{Assumptions}:$

Let $\lambda_{i}, i=1, \ldots, n$ denote the eigenvalues of $AB^2$. Suppose we have the condition that $|\lambda_{i}|<1 \, \forall i$.

Let $\beta_{i}, i=1, \ldots, n$ denote the eigenvalues of $A^2B$. Suppose we have the condition that $|\beta_{i}|<1 \, \forall i$.

$\textbf{Claim}:$

Then I have an intuition that $\textbf{ $\det(AB+A+I) < 0$ and $\det(BA+B+I)<0$ is not possible.}$ That is both of the determinants cannot be negative. I am not sure how to prove it?

$\textit{Some thoughts:}$

$(a) \textbf{Using $A = B+ \xi e_{1}^T$, we have }$ \begin{align} AB+A+I &= A^2 + A + I + A \xi e_{1}^{T},\\ BA+B+I &= A^2 + A + I + \xi e_{1}^{T}(I+A) \end{align}

Since the absolute values of the eigenvalues of $AB^2,$ and $A^2B$ are less than one, that means we have $|\det(AB^2)| < 1$ and $|\det(A^2B)|<1$.

\begin{align} A^2B &= A^3 + A^2 \xi e_{1}^{T},\\ AB^2 &= A^3 + (\xi e_{1}^{T})^2 + 2A^{2} \xi e_{1}^{T} \end{align}

$\textbf{(b)}$ If we proceed via the method of contradiction. if $\det(AB+A+I)<0$ and $\det(BA+B+I)<0$, then some of the eigenvalues of $AB^2$ or $A^2B$ will be greater than one in absolute value. THis would then violate the assumption of $|\lambda_{i}|<1 \forall i$,, $|\sigma_{i}|<1, \forall i$ .

$\textbf{(c)}$ Another thought is to use the perturbation argument: Fix $A, B$. Define $B(\epsilon):= A + \epsilon (B-A)$. For $\epsilon = 0$, we get $\det(A^2+A+I) \geq 0$ and hence the statement holds. For $\epsilon=1$, we have $B(1) = B$. If the statement fails in this case then there should be a minimal $\epsilon$ for which the statement is false. There might be a contradiction for $\epsilon < 1$?

Answer 1

This is a partial answer, now with added material.

Assume that $n$ is a multiple of 3, the other two cases should be similar. I'll denote $C_1\mapsto a$ and $C_2\mapsto b$, such that for, e.g., $n=6$, $$\tag{1}\label{1} A = \begin{pmatrix} a_1 & 1 & 0 & 0 & 0 & 0 \\ a_2 & 0 & 1 & 0 & 0 & 0 \\ a_3 & 0 & 0 & 1 & 0 & 0 \\ a_4 & 0 & 0 & 0 & 1 & 0 \\ a_5 & 0 & 0 & 0 & 0 & 1 \\ a_n & 0 & 0 & 0 & 0 & 0 \end{pmatrix},\quad A^{-1} = \begin{pmatrix} 0 & 0 & 0 & 0 & 0 & 1/a_n \\ 1 & 0 & 0 & 0 & 0 & -a_1/a_n \\ 0 & 1 & 0 & 0 & 0 & -a_2/a_n \\ 0 & 0 & 1 & 0 & 0 & -a_3/a_n \\ 0 & 0 & 0 & 1 & 0 & -a_4/a_n \\ 0 & 0 & 0 & 0 & 1 & -a_5/a_n \end{pmatrix}. $$ Note that $\det A = (-1)^{n-1}a_n$. The inverse of $A$ is easily calculated, see \eqref{1}, such that the determinant of $AB+A+I$ (and also of the other case, exchange $a_i$ and $b_i$) can be expressed via a Schur complement of the matrix $A^{-1}(AB+A+I) = B+I+A^{-1}$: We take the Schur complement w.r.t. the first and last row/column and get $$\tag{2}\label{2} \det(AB+A+I)=\det \begin{pmatrix} \alpha_1-\alpha_3 & \beta_1-\beta_2\\ \alpha_2-\alpha_3 & \beta_1-\beta_3 \end{pmatrix} =\det \begin{pmatrix} \alpha_1 &1& \beta_2\\ \alpha_2 &1& \beta_3\\ \alpha_3 &1& \beta_1 \end{pmatrix}, $$ where in the last step we used basic determinant rules. The $\alpha_j$ and $\beta_j$, with $j\in\{1,2,3\}$, are simply given by $$\tag{3}\label{3} \alpha_j = -\delta_{3,j} + \sum_{k=0}^{n/3-1} a_{3k+j}, \qquad \beta_j = -\delta_{3,j} + \sum_{k=0}^{n/3-1} b_{3k+j}, $$ with Kronecker's $\delta$. I’ve renamed OPs $\beta\mapsto\nu$.

For the eigenvalue assumption $|\lambda_i|, |\nu_i| <1$ it should be sufficient to consider the characteristic polynomials \begin{align}\tag{4a}\label{4a} P_a(\lambda)&=\det(A^2 B - \lambda I),\\ P_b(\nu )&=\det(A B^2 - \nu I),\tag{4b}\label{4b} \end{align} which can be calculated in a similar fashion: now we use $A^{-1}(A^2 B - \lambda I) = A B - \lambda A^{-1}$ and build the Schur complement w.r.t. the first, second and last row/column.

If the eigenvalues fulfill $|\lambda_i|<1$, the (real) zeroes of $P_a(\lambda)$ are between $\lambda=\pm 1$. At this point, I am not sure how to handle the complex zeroes (but see edit below). However, as $P_a(\lambda) \sim (-\lambda)^n$ for large $|\lambda|$, it should fulfill $P(-1)>0$ and $(-1)^{n}P(1)>0$. Let's evaluate $P_a(1)$ as one example, $$\tag{5}\label{5} P_a(1)=\det(A^2B-I)= \det \begin{pmatrix} \alpha_1 & \alpha_3 & \beta_2\\ \alpha_2 & \alpha_1 & \beta_3\\ \alpha_3 & \alpha_2 & \beta_1 \end{pmatrix}. $$ Note the similarity of \eqref{2} and \eqref{5}, and that \eqref{5} can be generalised to $P_a(\lambda)$ using polynomials $\alpha_j(\lambda),\beta_j(\lambda)$. So, the problem for arbitrary $n$ can be reduced to a discussion of the sign of the determinants of related $3\times3$ matrices.

Disclaimer: there might be sign errors due to even/odd $n$ and row/column permutations, please check.

Edit 12.08.22, 09:00 CEST:

From now on, we only consider even $n$, such that $n\mod 6 \equiv 0$, to get rid of the $(-1)^n$ terms.

As noted in my comments, $P_a(1)>0$ and $P_a(-1)>0$ are necessary conditions for $|\lambda_i|<1$, because complex $\lambda_i$ appear in complex conjugate pairs $\lambda_{i'}=\lambda_i^*$, such that $$\tag{6}\label{6} |\lambda_i|<1 \Rightarrow (\lambda_i \pm 1)(\lambda_i^* \pm 1)>0, $$ and all factors in $$\tag{7}\label{7} P_a(\lambda) = \prod_{i=1}^{n/2} (\lambda_i - \lambda)(\lambda_{i'} - \lambda) $$ are positive for $\lambda=\pm 1$. Here, we grouped the real eigenvalues in arbitrary pairs $(i,i')$.

As shown above, $P_{a,b}(1)$ have the simple representation \eqref{5}. Hence, we consider the matrix $$\tag{8}\label{8} D(\alpha,\gamma,\beta)=\begin{pmatrix} \alpha_1 &\gamma_3& \beta_2\\ \alpha_2 &\gamma_1& \beta_3\\ \alpha_3 &\gamma_2& \beta_1 \end{pmatrix} $$ with 3d vectors $\alpha,\beta,\gamma$, and formulate a geometric version of the problem. Define $\delta=(1,1,1)^T$, then the OPs conjecture holds, if $$\tag{9}\label{9} D(\alpha,\alpha,\beta)>0 \land D(\alpha,\beta,\beta)>0 \Rightarrow D(\alpha,\delta,\beta)>0 \lor D(\beta,\delta,\alpha)>0. $$ Note (a) that the determinant in 3D is known as triple product, $$\tag{10}\label{10} \det(a,b,c) = a \cdot (b \times c) = b \cdot (c \times a) = c \cdot (a \times b), $$ and (b) that the cyclic index permutations of in \eqref{2}, \eqref{5} and \eqref{8} are rotations by $120^\circ$ around $\delta$. I guess that the OP question can now be answered through a discussion of the (rotated) directions of $\alpha$, $\beta$ and $\delta$ in 3D.

Answer 2

OK, second try, equation references are pointing to my other answer:

Using the definitions (3) for $\alpha_j$ and $\beta_j$, we can proof the OPs conjecture in the following way: As the determinants $$ \begin{align} D_a = \det(AB+A+I) \tag{11a}\label{11a}\\ D_b = \det(BA+B+I) \tag{11b}\label{11b} \end{align} $$ are continuous functions of the $a_i$ and $b_i$ (and $\alpha_j$ and $\beta_j$), and both shall be negative, w.l.o.g. we can assume that, say, either $D_a=0$ crosses zero while the other is still positive, $D_b>0$, or, both are zero simultaneously. This case is handled first.

case $D_a=D_b=0$

First we show that $D_a$ and $D_b$ cannot simultaneously vanish under the OPs eigenvalue condition $|\lambda_i|<1$.

As can be seen from (2), w.l.o.g. the condition $D_a = D_b = 0$ is equivalent to $\alpha_1=\alpha_2=\alpha_3$. (Note that this condition can be solved for, e.g., $a_1$ and $a_2$.) Inserting this solution into (5), we immediately see that the first two columns of (5) become linearly dependent, such that the characteristic polynomial (4a) reads $P_a(1)=0$. Therefore, we have identified one eigenvalue $\lambda_1=1$, contradicting the assumption that all eigenvalues $|\lambda_i|<1$. The other possible case $\beta_1=\beta_2=\beta_3$ would give $P_b(1)=0$.

case $D_a=0 \land D_b>0$

Now we turn to the case $D_a=0 \land D_b>0$: Then, from (2) we see that $$\tag{12}\label{12} \alpha_1=\beta_2 \land \alpha_2=\beta_3 \land \alpha_3=\beta_1 $$ must hold. Inserting this into (2) for the determinant $D_b$, we get the inequality $$ \begin{align} D_b &= \det \begin{pmatrix} \beta_1 &1& \alpha_2\\ \beta_2 &1& \alpha_3\\ \beta_3 &1& \alpha_1 \end{pmatrix} = \det\begin{pmatrix} \alpha_3 &1& \alpha_2\\ \alpha_1 &1& \alpha_3\\ \alpha_2 &1& \alpha_1 \end{pmatrix} \tag{13a}\label{13a} \\ \\ &= -\frac{(\alpha_1-\alpha_2)^2 + (\alpha_2-\alpha_3)^2 + (\alpha_3-\alpha_1)^2}{2} < 0 \tag{13b}\label{13b} \end{align} $$ which contradicts the assumption. Therefore, the only continuous path to negative $D_a$ and $D_b$ goes through $D_a=D_b=0$, at which point $\lambda_1=1$.