If I understand correctly, you mean the definition of an object’s {symmetries} as its stabilizer under some group action on an ambient space. While a clearcut “first” might not exist, I’d say Jordan has early approximations to that, provided you are willing to replace “function” by “transformation”, “substitution”, or “motion”. E.g. in his Traité des substitutions (1870, p. 50):
§V. — Symmetry of rational functions.
The link between groups and functions.
60. Let $\mathrm F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $\mathrm F_1, \mathrm F_\alpha, \mathrm F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don’t alter the function $\mathrm F_1$ obviously form a group, which one can call the function’s group.
(Earlier Serret’s Cours d’algèbre supérieure (1866, p. 387) wrote that the “substitutions admises” by a function form a “système conjugué” ($=$ Cauchy’s term for group, p. 251). I’d have thought crystallographic groups were defined as stabilizers before this, but it seems not.)
EDIT: To answer the question as recast in your comment (“first published book or article where symmetries in geometry are defined as transformations”), I think a problem is that early sources won’t call any maps symmetries: instead you’ll find expressions like mouvements qui superposent à lui-même (Jordan 1867, p. 230), Transformationen in sich (Klein & Lie 1871; Klein 1893, p. 326), Deckbewegungen (Sohncke 1875, p. 115), Deckoperationen (Schoenflies 1891, p. 13; Curie 1894, p. 395; Love 1906, p. 147), or at best symmetry-operations (Hilton 1903, p. 32).
While these references all contain the main idea, the exact terminology you want does not seem to appear until A. Speiser, Die Theorie der Gruppen von endlicher Ordnung (2nd ed., 1927, p. 78):
Eine kongruente Abbildung des Gitters auf sich selber nennen wir eine Symmetrie des Gitters.
Interestingly, this sentence is absent in the first edition (1923, p. 53). In English, the earliest such definition I can find is by H. S. M. Coxeter in Regular skew polyhedra in three and four dimensions and their topological analogues (1937, p. 35):
We define a symmetry (or "symmetry operation") of any figure as a congruent transformation of the figure into itself (i.e., a combination of translations, rotations, and reflections).
Coxeter later repeats it in e.g. Mathematical Recreations and Essays (1939, p. 130) or Regular complex polytopes (1974, p. 2).
