Studies of neural networks that are more general than $\mathbb{R}^n\mapsto\mathbb{R}^k$ include
 

 - <A HREF="https://arxiv.org/abs/1705.09792">Deep Complex Networks</A> (2017)
 - <A HREF="https://arxiv.org/abs/1712.04604">Deep Quaternion
   Networks</A> (2017)
 - <A HREF="https://link.springer.com/chapter/10.1007/978-1-4020-2660-7_8">P-Adic
   Neural Networks</A> (2004)
 - <A HREF="https://openaccess.thecvf.com/content_cvpr_2017/html/Huang_Deep_Learning_on_CVPR_2017_paper.html">Deep
   Learning on Lie Groups</A> (2017)

A general overview is provided in <A HREF="https://arxiv.org/abs/1907.03742v2">Neural Networks on Groups</A> (2019).
 
One general thing to keep in mind in this context is that the training of the network by steepest descent will rely on continuous differentiability, which not all implementations provide.