Where did Sophus Lie write the group commutator for two one parameter groups If $X,Y$ are vector fields and $\def\Fl{\operatorname{Fl}}\Fl^X_t$ and $\Fl^Y_t$ their local flows, let $[\Fl^X_t,\Fl^Y_t]:= \Fl^Y_{-t}\Fl^X_{-t}\Fl^Y_t\Fl^X_t$ denote the group commutator of the flows. Then
$$
\partial_t|_0 [\Fl^X_{t},\Fl^Y_{t}] =0, \qquad
\frac12 \partial_t^2|_0 [\Fl^X_{t},\Fl^Y_{t}] = [X,Y].
$$
See


*

*Markus Mauhart, Peter W. Michor: Commutators of flows and fields. Archivum Mathematicum (Brno) 28,3-4 (1992), 228–236, (pdf)
for an extension of this to formal bracket expressions of arbitrary length.
Where did Sophus Lie write this? And where did he compute something like
$$
\lim_n \frac1n [\Fl^X_{1/n},\Fl^Y_{1/n}]^n\quad ?
$$
 A: Looking around on the internet, I found an English translation of Lie's 1891 paper Die Grundlagen für die Theorie der unendlichen kontinuierlichen Transformationsgruppen. I.  (I.e., The foundations of the theory of infinite continuous transformation groups - I) at the location 
http://neo-classical-physics.info/uploads/3/0/6/5/3065888/lie-_infinite_continuous_groups_-_i.pdf
In this English translation (I haven't gone to look for the original German), Lie gives the formula for the commutator of the flows of two vector fields in the form that you require.  Look at Paragraphs 35-37, noting especially the displayed equation (45) and the first displayed equation in Paragraph 36, which is exactly the first formula you asked about.
It's quite possible that these formulae appeared even earlier in Lie's papers.  He mentions, in the introduction, a paper of 1883 that might well also have these formulae. I'll check when I get the chance.
Concerning the Trotter formula (your second formula), I have no idea.
A: I believe you're not going to find exactly what you want in Lie, because he never formalized flows (or finite transformations) and their commutation as you do. Maybe the closest would be this, from Über Differentialinvarianten, Math. Ann. 24 (1884) 537-578:

... erhalten wir folgenden Fundamentalsatz, den ich 1872 entdeckt habe:
Satz 3. Enthält eine kontinuierliche Gruppe die beiden infinitesimalen Transformationen:
  $$
Bf=\sum\xi_\varkappa\frac{\partial f}{\partial x_\varkappa}
\quad\textit{und:}\quad
Cf=\sum\eta_\varkappa\frac{\partial f}{\partial x_\varkappa},
$$
  so enthält sie ebenfalls die infinitesimale Transformation:
  $$
\sum_i(B\eta_i-C\xi_i)\frac{\partial f}{\partial x_i},
$$
  deren Symbol bekanntlich auf die beiden äquivalenten Formen:
  $$
B(C(f)) - C(B(f)) = (B, C)
$$
  gebracht werden kann. 

As you can see, his definition of the bracket of vector fields is always as the commutator of the derivations they define on functions (something that goes back to Jacobi). What this Satz states, then, is that the finite transformations (or flow) generated by the infinitesimal commutator $(B,C)$ belong to the group generated by (the flows of) $B$ and $C$. Not surprisingly, Lie's proof is by expanding the flows to second order.
Lie may or may not have stated this Satz elsewhere before 1884, but I doubt he ever wrote  a formula for, much less definition of, the bracket as limit of commutators of finite transformations.
Correction Robert Bryant has now found an 1891 reference where Lie (or at least Engel) indeed commutes finite transformations. See his reply and the comments there.
Update As to your question of who (esp. first) expressed the bracket as a derivative of commutators of flows: I don't know (my impression is that these things developed slowly in a sort of consensus). As a data point though, one might argue that the formula
$$
[V,T]=\frac{d}{ds}\frac{d}{dt}e^{-sV}e^{tT}e^{sV}\Bigr|_{s=t=0}
$$
is on p. 240 of Poincaré, Sur les groupes continus, Trans. Cambridge Philos. Soc. 18 (1900) 220-255.
Further update Trotter's formula that you also mention now is indeed called "Lie-Trotter" by e.g. Chernoff [1968,1974] or Chorin et al. [1978]. The latter write (sic):

... the equation $dx/dt=Ax+Bx$ leads to the 1875 formula of S. Lie [38]:
  $$
\exp\{A+B\} = \lim_{n\to\infty}(\exp\{A/n\}\exp\{B/n\})^n.\tag{$*$}
$$
  This and the related formula
  $$
\exp\{[A,B]\} = \lim_{n\to\infty}(
\exp\bigl\{\frac{-B}{\sqrt n}\bigr\}
\exp\bigl\{\frac{-A}{\sqrt n}\bigr\}
\exp\bigl\{\frac{B}{\sqrt n}\bigr\}
\exp\bigl\{\frac{A}{\sqrt n}\bigr\})\tag{$**$}
$$
  occur in the theory of Lie groups.
...
[38] Lie, S., and Engel, F., Theorie der Transformationsgruppen, 3 Vols., Teubner, Leipzig, 1888.

The problem is that [38] is not from 1875, nor does it contain anything remotely like formula ($*$) (I am ready to bet a lot of money). I may be wrong but until someone finds that elusive 1875 paper, I would tend to date ($*$) and ($**$) from around von Neumann [1929, p. 19].
