If I understand correctly, you mean the definition of an object’s **{symmetries}** as its stabilizer under some group action on an ambient space. While a clearcut “first” might not exist, I’d say **Jordan** has early approximations to that, provided you are willing to replace “function” by “transformation”, “substitution”, or “motion”. E.g. in his [*Traité des substitutions* (1870, p. 50)](https://archive.org/stream/traitdessubsti00jorduoft#page/50):
> **§V. — Symmetry of rational functions.**
> *The link between groups and functions.*
> 60. Let $\mathrm F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $\mathrm F_1, \mathrm F_\alpha, \mathrm F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don’t alter the function $\mathrm F_1$ obviously form a group, which one can call the *function’s group*.
(Earlier Serret’s [*Cours d’algèbre supérieure* (1866, p. 387)](//archive.org/details/coursdalgbresup11serrgoog/page/n405) wrote that the “substitutions admises” by a function form a “système conjugué” ($=$ Cauchy’s term for *group*, p. 251). I’d have thought crystallographic groups were defined as stabilizers before this, but [it seems not](http://www.clarku.edu/~djoyce/wallpaper/history.html).)
-----
**EDIT:** To answer the question as recast in your comment (“first published book or article where symmetries ***in geometry*** are defined as transformations”), I think a problem is that early sources won’t call any maps *symmetries*: instead you’ll find expressions like *mouvements qui superposent à lui-même* ([Jordan 1867, p. 230](http://gallica.bnf.fr/ark:/12148/bpt6k3022r/f231)), *Transformationen in sich* ([Klein & Lie 1871](https://archive.org/stream/gesammeltemath01kleirich#page/n441); [Klein 1893, p. 326](https://archive.org/stream/eininhohere01kleirich#page/n9)), *Deckbewegungen* ([Sohncke 1875](https://www.emis.de/cgi-bin/JFM-item?08.0634.01), [p. 115](https://books.google.com/books?id=mYE7AQAAIAAJ&pg=PA113&dq=Deckbewegung)), *Deckoperationen* ([Schoenflies 1891, p. 13](https://books.google.com/books?id=HWlUAAAAMAAJ&pg=PA13); [Curie 1894, p. 395](https://babel.hathitrust.org/cgi/pt?u=1&id=umn.319510006112153&num=393); [Love 1906, p. 147](https://archive.org/stream/treatiseonmathem00loveuoft#page/147)), or at best *symmetry-operations* ([Hilton 1903, p. 32](https://archive.org/stream/mathematicalcry02hiltgoog#page/n48)).
While these references all contain the main idea, the exact terminology you want does not seem to appear until **A. Speiser**, [*Die Theorie der Gruppen von endlicher Ordnung* (2nd ed., 1927, p. 78)](http://dx.doi.org/10.1007/978-3-662-38551-7_7):
>Eine kongruente Abbildung des Gitters auf sich selber nennen wir eine ***Symmetrie*** des Gitters.
Interestingly, this sentence is absent in the first edition [(1923, p. 53)](http://dx.doi.org/10.1007/978-3-662-42031-7_6). In English, the earliest such definition I can find is by H. S. M. Coxeter in [*Regular skew polyhedra in three and four dimensions and their topological analogues* (1937, p. 35)](http://dx.doi.org/10.1112/plms/s2-43.1.33):
>We define a ***symmetry*** (or "symmetry operation") of any figure as a congruent transformation of the figure into itself (i.e., a combination of translations, rotations, and reflections).
Coxeter later repeats it in e.g. [*Mathematical Recreations and Essays* (1939, p. 130)](http://dx.doi.org/10.1090/S0002-9904-1940-07170-8) or [*Regular complex polytopes* (1974, p. 2)](https://books.google.com/books?id=9BY9AAAAIAAJ&pg=PA2).