As I mentioned in my comment, I recall this concept as is the "time-ordered exponential" (as I remember from Quantum Field Theory lectures, long ago). Alternatively, the "path-ordered exponential"exponential. I'm no expert here, so I can't really point you to any definitive sources that you couldn't find with a bit of searching yourself anyway.
The function f(t) does not have to be particularly 'nice', just locally integrable should be enough. You can go even further and replace f(t)dt by dF(t) for a continuous finite-variation function F. In fact, F can be any continuous semimartingale, as long as you use Stratonovich integration in the associated SDE (as Ito integration is not coordinate independent). This is the method used by Rogers & Williams (Diffusions, Markov Processes and Martingales) to construct Brownian motions on lie Lie groups and referred to there as the product integral product-integral injection. Actually, it's a bijection from the continuous semimartingales X in the Lie algebra starting at $X_0=0$ to the continuous semimartingales Y in the Lie group starting at the point $Y_0=1$ satisfying the Stratonovich SDE $$ \partial Y = Y\,\partial X. $$
Also, why restrict to lie Lie groups/algebras? Any manifold with an affine connection will do, where f maps to the tangent space at some base point, and is moved along the generated curve by parallel transport. There is the possibility of the solution exploding in finite time though. In the case of Lie Groupsgroups, there is a standard invariant connection which gives you the time-ordered exponential.