IIUC you have in mind the definition of an object's **{symmetries}** as its stabilizer under some group action on an ambient space. While a clearcut origin may not exist (Grattan-Guinness [warns](http://www.ams.org/mathscinet-getitem?mr=533433): "One of the pitfalls in doing the history of mathematics is "priorities": of accepting, as valid, questions of the form, "Who was the first to...?""), I'd say **Jordan** has early approximations to what you want, provided you are willing to replace "function" by "transformation", "substitution", or "motion".

E.g. his [*Traité des substitutions* (1870, p. 50)](https://archive.org/stream/traitdessubsti00jorduoft#page/50) has:

> **§V. -- SYMMETRY OF RATIONAL FUNCTIONS.**

> *The link between groups and functions.*

> 60 . Let $F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $F_1, F_\alpha, F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don't alter the function $F_1$ obviously form a group, which one can call the *function's group*.

(I would have thought crystallographic groups were defined as stabilizers before Jordan, but [it seems not](http://www.clarku.edu/~djoyce/wallpaper/history.html).)

**EDIT:** To your recast question ("first published book or article where symmetries are defined as transformations"), I think a problem is that early sources won't have the word **symmetries**: instead you'll find expressions like **mouvements qui superposent à lui-même** ([Jordan 1867, p. 230](http://gallica.bnf.fr/ark:/12148/bpt6k3022r/f231)), **Transformationen in sich** ([Klein-Lie 1871](https://archive.org/stream/gesammeltemath01kleirich#page/n441); [Klein 1893, p. 326](https://archive.org/stream/eininhohere01kleirich#page/n9)), **Deckoperationen** ([Schoenflies 1891, p. 13](https://books.google.com/books?id=HWlUAAAAMAAJ&pg=PA13); [Love 1906, p. 147](https://archive.org/stream/treatiseonmathem00loveuoft#page/147)), or at best **symmetry-operations** ([Hilton 1903, p. 32](https://archive.org/stream/mathematicalcry02hiltgoog#page/n48)).

While the above references all contain the main idea, the exact terminology you want does not seem to appear until A. Speiser, [*Die Theorie der Gruppen von endlicher Ordnung* (2nd edition, 1927, p. 78)](http://dx.doi.org/10.1007/978-3-662-38551-7_7):

>Eine kongruente Abbildung des Gitters auf sich selber nennen wir eine ***Symmetrie*** des Gitters.

Interestingly, this sentence is absent in the first edition [(1923, p. 53)](http://dx.doi.org/10.1007/978-3-662-42031-7_6). In English, the earliest version I can find of this definition is by H. S. M. Coxeter in [*Regular skew polyhedra in three and four dimensions and their topological analogues* (1937, p. 35)](http://dx.doi.org/10.1112/plms/s2-43.1.33):

>We define a ***symmetry*** (or "symmetry operation") of any figure as a congruent transformation of the figure into itself (i.e., a combination of translations, rotations, and reflections).

Coxeter later repeats it, e.g. in [*Mathematical Recreations and Essays* (1939, p. 130)](http://dx.doi.org/10.1090/S0002-9904-1940-07170-8) or [*Regular complex polytopes* (1974, p. 2)](https://books.google.com/books?id=9BY9AAAAIAAJ&pg=PA2).