Maximal number of couples of vectors $(r,n)$ such that the product $\langle n_i , n_j\rangle+\langle r_i \times n_i , r_j \times n_j\rangle<0$?

Question

Let $\langle v,w\rangle$ and $v\times w$ stand for the dot product and the cross product of vectors $v,w\in\mathbb R^3$.

Do there exist $2k$ vectors $$ r_1, \dots, r_k, \; n_1, \dots, n_k\in \mathbb{R}^3, \qquad\text{with $k>4$,} $$ satisfying the inequality $$ \langle n_i , n_j\rangle + \langle r_i \times n_i, r_j \times n_j\rangle <0 $$ for all $i<j$? If yes, what is an upper bound for such $k$?

Notice that $$ \langle n_i , n_j\rangle + \langle r_i \times n_i , r_j \times n_j\rangle \\ = \langle n_i , n_j\rangle + \langle r_i , r_j\rangle \langle n_i , n_j\rangle - \langle r_i , n_j\rangle \langle r_j , n_i\rangle \\ = \langle (r_i \times n_i) + n_j , (r_j \times n_j) + n_i \rangle. $$

Answer 1

$\newcommand\rk{\mathop{\mathrm{rank}}}$I will shjow that this is possible for $k=6$, and impossible for $k>7$.

Prelude. First of all, set $k_i=r_i\times n_i$. The only condition the vectors $n_i$ and $k_i$ must obey is $k_i\perp n_i$.

Now, set $a_i=n_i+k_i$, $b_i=n_i-k_i$; because of orthogonality, we have $|a_i|=|b_i|$. Conversely, for any such pair we have that $n_i=(a_i+b_i)/2$ and $k_i=(a_i-b_i)/2$ are orthogonal. Moreover, the $a_i$ and the $b_i$ satisfy the same relations, as $$ \langle a_i,a_j\rangle +\langle b_i,b_j\rangle =2(\langle n_i,n_j\rangle+\langle k_i,k_j\rangle). $$

1. We show that $k\leq 7$ even without the restriction on the lengths of $a_i$ and $b_i$. By a small perturabtion, we reach that all the $a_i$ and $b_i$ are nonzero.

Consider the vectors $(a_i,b_i)\in\mathbb R^3\oplus \mathbb R^3=\mathbb R^6$. Our condition says that all of their angles are obtuse. It is well known that the number of such vectors may exceed the dimension by no more than $1$ (see, e.g., this or this posts); hence $k\leq 7$.

2. To present an example of 12 such vectors, choose a small $\epsilon >0$, let $a_1,a_2,\dots,a_6$ be the normalized columns of the matrix $$ \begin{pmatrix} 1& \epsilon & -\epsilon & -1& -\epsilon & -\epsilon \\ 0 & 1& -\epsilon ^2& \epsilon & -1& -3\epsilon ^2\\ 0 & -\epsilon ^2& 1& \epsilon & -3\epsilon ^2& -1 \end{pmatrix}; $$ only the products in pairs $(1,2)$, $(3,4)$, $(5,6)$ are slightly positive, all others are negative, and the products in pairs $(1,4)$, $(2,5)$, $(3,6)$ are strongly negative.

Now it remains to assign to the same normalized columns the vectors $b_6,b_3,b_2,b_5,b_4,b_1$. The roles of pairs swap, so the result is what we want.