Deep Learning on Lie Groups for Skeleton-Based Action Recognition (2017) studies a neural network structure where the data space as well as the weight space of each layer correspond to a Lie group.
One general thing to keep in mind in this context is that the training of the network by steepest descent will rely on continuous differentiability. This training requirement is not a complication of the Lie group based networks, but may become an obstacle for other implementations.