Let $M$ be an invertible $4 \times 4$ real matrix, and let $\mathbf{u}, \mathbf{v} \in \mathbb{R}^4$. Consider the matrix $$\displaystyle \mathcal{M} = \mathcal{M}(\mathbf{u}, \mathbf{v}) = [\mathbf{u} | M \mathbf{u} | \mathbf{v} | M \mathbf{v} ].$$ Here the notation is that the columns of $\mathcal{M}$ are $\mathbf{u} , M \mathbf{u} , \mathbf{v} , M \mathbf{v}$ respectively.

Generically, $\mathcal{M}$ will be invertible and so the only vector orthogonal to all four columns will be the zero vector. If $\mathbf{u}$ and $\mathbf{v}$ are proportional then $\mathcal{M}$ will have rank at most two, and there is no way of determining a unique direction which is orthogonal to all of the columns. However, there exist pairs $\mathbf{u}, \mathbf{v}$ such that $\mathcal{M}$ has rank three, and hence $\ker \mathcal{M}^T$ is 1-dimensional and there should be an algebraic formula which gives a generator in terms of $\mathbf{u}, \mathbf{v}$ and $M$.

The motivation here is a generalization of the cross-product in $\mathbb{R}^3$. There the goal is to find an algebraic expression which gives the generator of the kernel of $[\mathbf{u} | \mathbf{v}]^T$, namely the cross product $\mathbf{u} \times \mathbf{v}$.

One way to do this is to note that the condition that $\ker \mathcal{M}^T$ is non-trivial is equivalent to the condition that $\det(\mathcal{M}) = 0$. Expanding, we find that

$$\displaystyle \det(\mathcal{M}) = Q_M(\mathbf{u}, \mathbf{v})$$

which is a bi-quadratic form in $\mathbf{u}, \mathbf{v}$. Fixing $\mathbf{u}$, we then obtain a quadratic form $\mathcal{Q}_{M, \mathbf{u}}(\mathbf{v})$ in $\mathbf{v}$, which has an obvious solution $\mathbf{v} = \mathbf{u}$. Using the general theory of quadratic forms one can then parametrize all solutions to the equation $\mathcal{Q}_{M, \mathbf{u}}(\mathbf{v}) = 0$. From here one can then find the kernel by taking a Laplace expansion, just like in the 3-dimensional case with the cross product.

Is there a more "linear" method of describing the set, and perhaps in a way which is more symmetric in $\mathbf{u}, \mathbf{v}$?

Let $M$ be an invertible $4 \times 4$ real matrix, and let $\mathbf{u}, \mathbf{v} \in \mathbb{R}^4$. Consider the matrix $$\displaystyle \mathcal{M} = \mathcal{M}(\mathbf{u}, \mathbf{v}) = [\mathbf{u} | M \mathbf{u} | \mathbf{v} | M \mathbf{v} ].$$ Here the notation is that the columns of $\mathcal{M}$ are $\mathbf{u} , M \mathbf{u} , \mathbf{v} , M \mathbf{v}$ respectively.

For generic $M$ and for generic pairs $\mathbf{u}, \mathbf{v}$, $\mathcal{M}$ will be invertible and so the only vector orthogonal to all four columns will be the zero vector. If $\mathbf{u}$ and $\mathbf{v}$ are proportional then $\mathcal{M}$ will have rank at most two, and there is no way of determining a unique direction which is orthogonal to all of the columns. However, there exist pairs $\mathbf{u}, \mathbf{v}$ such that $\mathcal{M}$ has rank three, and hence $\ker \mathcal{M}^T$ is 1-dimensional and there should be an algebraic formula which gives a generator in terms of $\mathbf{u}, \mathbf{v}$ and $M$. Our $M$ will always be chosen so that generically $\mathcal{M}$ is invertible and there exist $\mathbf{u}, \mathbf{v}$ such that $\mathcal{M}$ has rank three.

The motivation here is a generalization of the cross-product in $\mathbb{R}^3$. There the goal is to find an algebraic expression which gives the generator of the kernel of $[\mathbf{u} | \mathbf{v}]^T$, namely the cross product $\mathbf{u} \times \mathbf{v}$.

One way to do this is to note that the condition that $\ker \mathcal{M}^T$ is non-trivial is equivalent to the condition that $\det(\mathcal{M}) = 0$. Expanding, we find that

$$\displaystyle \det(\mathcal{M}) = Q_M(\mathbf{u}, \mathbf{v})$$

which is a bi-quadratic form in $\mathbf{u}, \mathbf{v}$. Fixing $\mathbf{u}$, we then obtain a quadratic form $\mathcal{Q}_{M, \mathbf{u}}(\mathbf{v})$ in $\mathbf{v}$, which has an obvious solution $\mathbf{v} = \mathbf{u}$. Using the general theory of quadratic forms one can then parametrize all solutions to the equation $\mathcal{Q}_{M, \mathbf{u}}(\mathbf{v}) = 0$. From here one can then find the kernel by taking a Laplace expansion, just like in the 3-dimensional case with the cross product.

Is there a more "linear" method of describing the set, and perhaps in a way which is more symmetric in $\mathbf{u}, \mathbf{v}$?