I would like to provide you with a non-trivial example (and a reference) of a Lie algebra of symmetries
which is not a Lie algebra of a Lie group within the framework of conventional quantum mechanics.
This example relies on the notion of "dynamical groups" (which you can find a lot of literature about in the net).
I think that the most precise definition of a dynamical group of a quantum system would be a Lie group
which the system's phase space is a coadjoint orbit of. In the cases known to me
the system's Hamiltonian belongs to the Lie algebra of the dynamical group, but I don't think that
this is an essential requirement (The Hamiltonian can be a member of the universal enveloping algebra).
The main application of dynamical groups is to provide algebraic solution to the quantum mechanical problem.
The spectrum of the systems can be obtained from the representation of the dynamical group associated
with the coadjoint orbit instead of solving the Shroedinger equation.
The Lie algebra of the dynamical group generally consists of the usual space time symmetries
such as the angular momentum su(2) in addition to more generators which originally required
a lot of ingenuity to come up with, such as the Runge-lenz generators of the Hydrogen atom problem, sometimes
referred to as the Kepler problem,which close together with the angular momentum generators to o(4) for elliptical motion and o(3,1) for hyperbolic motion.
Another known example in which a dynamical group formulation is used is the harmonic oscillator
with the dynamical group SU(1,1).
Returning to the Kepler problem. The treatment described above considers only a fixed energy subspaces. In the following article , by C. Dabul, J. Dabul, P. Slodowy,
a twisted Kac-Moody dynamical algebra is constructed which is a simultaneous dynamical Lie algebra
of the full problem (corresponding to elliptic, parabolic and hyperbolic trajectories).
This is an example of a dynamical Lie algebra of symmetries which is not a Lie algebra of
a Lie group.
Regrettably, I didn't see a followup of this work in terms of algebraic solution of the
full Kepler problem in terms of representations of this algebra, nor a treatment of this problem
in terms of coadjoint orbits of (the non-Lie) Kac-Moody groups which correspond to this algebra and are
subjects of active research. I think that these would be interesting research problems.