Added Wed 13 Apr 2022
I have written a short note with experimental data, which shows not all pseudo keys are good keys.
Public key cryptography based on non-invertible matrices
We got public key cryptography scheme based on non-invertible matrices and would like to know how easy it is to break.
Working over $\mathbb{F}_p$ with $p$ large. All matrices are square $n \times n$.
Alice chooses matrices $M_A,X_A$ and makes $P_A=M_A X_A$ public.
Bob chooses matrices $M_B,X_B$ and makes $P_B=M_B X_B$ public.
Assume $M_A,M_B$ are not invertible and satisfy $M_A M_B=M_B M_A$.
To ensure this, take the field to be finite and Alice and Bob agree on a singular matrix $M_0$ and Alice chooses large integer $N_A$ and set $M_A=M_0^{N_A}$ and Bob chooses large integer $N_B$ and set $M_B=M_0^{N_B}$. Since powers of matrices commute, $M_A,M_B$ commute. Observe that this is different from the discrete logarithm since $M_A,M_B$ are unknown.
Assume that $X_A,X_B$ are invertible.
To exchange shared secret, Bob makes public $S_B=M_B P_A=M_B M_A X_A$ and Alice computes $S_B X_A^{-1}=M_B M_A$
Likewise, Alice makes public $S_A=M_A P_B=M_A M_B X_B$ and Bob computes $S_A X_B^{-1}=M_A M_B$.
At this point, Alice and Bob have the shared secret $M_A M_B=M_B M_A$, which is expected to be hard to find by adversary.
Also, everyone, including an adversary know
$P_A=M_A X_A,P_B=M_B X_B,S_A=M_A P_B,S_B=M_B P_A$.
If $P_B$ were invertible, adversary could break the scheme by computing $S_A P_B^{-1}=M_A$.
Q1 What is complexity of breaking this crypto scheme, i.e. given $P_A,P_B,S_A,S_B$, find $M_A M_B$?
Q2 Can we replace matrices by some object and make a crypto scheme based on non-invertibility?
We are interested in choices of the matrices and the field for which breaking the scheme is hard.
Treating the matrices as variables, we get $4n^2$ equations with $4n^2$ variables and $2n^2$ of the equations are linear and the other equations are quadratic.
We got partial results about hardness.
The problem is given $P_A,P_B,S_A,S_B$, find $M_A M_B$.
Experimentally with a toy implementations, we found many solutions which satisfy the construction, but don't give the shared secret. If the number of potential solutions is large, the scheme will be hard to break. For $p=3,n=3$, we got $27$ total solutions and for $p=2,n=4$ we got $256$ total solutions.
In addition, there is algebraic attack. We have four unknown matrices, set all their entries to variables.
We have four equations over matrices, two of which linear. From the linear equation eliminate variables using gaussian elimination, which leaves $2n^2$ quadratic equations.
sagemath toy implementation with many solutions: