Neural networks over gadgets other than $\mathbb{R}$

Question

Recently, I learned that neural networks (NN) can be defined over fields other than $\mathbb{R}$: for example, Khrennikov and Tirozzi wrote a paper in 1999 (!) on $p$-adic neural networks, or neural networks over $p$-adic fields. It seems that there are some applications towards $p$-adic dynamical systems.

Are there similar lines of work, building NNs over gadgets other than $\mathbb{R}$? For example, some sort of arithmetic neural network?

In categorical terms, consider the category $\mathbf{CartSp}$ of cartesian spaces ($\mathbb{R}^n$); one can say that a neural network learns morphisms in this category. My question could be formulated as: over which categories could NNs be used to learn morphisms?

Answer 1

Studies of neural networks that are more general than $\mathbb{R}^n\mapsto\mathbb{R}^k$ include

• Deep Complex Networks (2017)
• Deep Quaternion Networks (2017)
• P-Adic Neural Networks (2004)
• Deep Learning on Lie Groups (2017)

A general overview is provided in Neural Networks on Groups (2019).

One general thing to keep in mind in this context is that the training of the network by steepest descent will rely on continuous differentiability, which not all implementations provide.

Answer 2

Deep neural networks with ReLU activation should not be thought of as neural networks that build upon the field $(\mathbb{R},+,\cdot)$, but they should instead be thought of as neural networks over the tropical semiring $(\mathbb{R},\oplus,\otimes)$.

Recall that the ReLU activation function is the function $\sigma:\mathbb{R}\rightarrow\mathbb{R}$ defined by $\sigma(x)=\max(x,0)$. Right now, ReLU seems to be the most popular activation function for deep neural networks, and it also seems to be the most mathematical activation function since continuous piecewise linear functions and tropical rational functions are essentially deep neural networks with ReLU activation.

Define operations $\oplus,\otimes,\oslash$ on $\mathbb{R}$ by setting $x\oplus y=\max(x,y),x\otimes y=x+y,x\oslash y=x-y$. Recall that the (max) tropical semiring is the semiring $(\mathbb{R},\oplus,\otimes)$.

A tropical monomial function is a function $F:\mathbb{R}^n\rightarrow\mathbb{R}$ of the form $F(x_1,\dots,x_n)=a\otimes x_{i_1}\otimes\dots\otimes x_{i_n}$. A tropical polynomial is a function $F:\mathbb{R}^n\rightarrow\mathbb{R}$ where $F(x_1,\dots,x_n)=F_1(x_1,\dots,x_n)\oplus\dots\oplus F_s(x_1,\dots,x_n)$ for some tropical monomials $F_1,\dots,F_s$.

A tropical rational function is a function $F:\mathbb{R}^n\rightarrow\mathbb{R}$ of the form $F(x_1,\dots,x_n)=G(x_1,\dots,x_n)\oslash H(x_1,\dots,x_n)$ where $G,H$ are tropical polynomials.

Theorem: Let $F:\mathbb{R}^n\rightarrow\mathbb{R}$ be a function. Then the following are equivalent.

1. $F$ is a tropical rational function.

2. $F$ is a continuous piecewise linear function with integer coefficients.

3. $F$ is the function computed by a deep neural network with activation functions $\sigma_1,\dots,\sigma_r$ such that $\sigma_k(x)=\max(x,a_k)$ for some constants $a_1,\dots,a_r$, real-valued biases, and integer-valued weights.

The restriction to integer coefficients is not a terribly

Corollary: Let $F:\mathbb{R}^n\rightarrow\mathbb{R}$ be a function. Then the following are equivalent.

1. $N\cdot F$ is a tropical rational function for some non-zero integer $N$.

2. $F$ is a continuous piecewise linear function with rational coefficients.

3. $F$ is the function computed by a deep neural network with activation functions $\sigma_1,\dots,\sigma_r$ such that $\sigma_k(x)=\max(x,a_k)$ for some constants $a_1,\dots,a_r$, real-valued biases, and rational-valued weights.

The correspondence between tropical rational functions and deep neural networks with ReLU activation has been documented in the 2018 paper Tropical Geometry of Deep Neural Networks by Liwen Zhang, Gregory Naitzat, and Lek-Heng Lim.

Answer 3

In cryptography, substitution-permutation networks (a type of block cipher) can be thought of as hand-crafted deep neural networks over finite fields. The AES encryption function is the symmetric encryption function that we all use to encrypt large amounts of data, and the AES encryption function is a substitution-permutation network. In a substitution-permutation network, the substitution boxes can be thought of as the activation functions while the affine layers correspond to the weight matrices and bias vectors.

Both symmetric cryptography and machine learning require universality, and the need for universality drives both symmetric cryptography and neural networks to have the same kind of structure but over different kinds of algebras. In symmetric cryptography, one needs universality since the composition of round permutations needs to approximate a generic permutation in few rounds while in machine learning, one needs for the composition of layers to be able to efficiently approximate an arbitrary continuous function.

Suppose that $K$ is a finite field and $V_1,\dots,V_n$ are vector spaces over the field $K$. Then a substitution permutation network round function is a function $f:(V_1\oplus\dots\oplus V_n)^2\rightarrow V_1\oplus\dots\oplus V_n$ of the form $f_k(x)=f(k,x)=k+P(S(x))$ where $P$ is an affine mapping and where $S(x_1,\dots,x_n)=(S_1(x_1),\dots,S_n(x_n))$ for permutations $S_j:V_j\rightarrow V_j$. The permutations $S_j$ are called S-boxes and these S-boxes correspond to the non-linear portion of a neural network. The round permutation $f_k$ should be considered as a single layered neural network, while the entire encryption function with expanded key $k_1,\dots,k_s$ is the composition $f_{k_1}\circ\dots\circ f_{k_s}$. The affine mapping $P$ does not change after each round, so the encryption permutation $f_{k_1}\circ\dots\circ f_{k_s}$ is a recurrent neural network.

The AES encryption function is a convolutional neural network (CNN) over $F_{256}$ where the mixcolumns step is the convolutional layer. For the AES, the S-box non-linear activation function is the composition of the inversion mapping $x\mapsto x^{-1}$ with an $F_2$-linear layer. The CNN architecture for the AES allows the AES to be more easily investigated mathematically without compromising security and the CNN architecture allows for a easier and more efficient implementation of the cipher and it allows less room for a back door.