IIUC you have in mind the definition of an object's **{symmetries}** as its stabilizer under some group action on an ambient space. While a clearcut origin may not exist (Grattan-Guinness [warns](http://www.ams.org/mathscinet-getitem?mr=533433): "One of the pitfalls in doing the history of mathematics is "priorities": of accepting, as valid, questions of the form, "Who was the first to...?""), I'd say **Jordan** has early approximations to what you want, provided you are willing to replace "function" by "transformation", "substitution", or "motion".

E.g. his [*Traité des substitutions* (1870, p. 50)](https://archive.org/stream/traitdessubsti00jorduoft#page/50) has:

> **§V. -- SYMMETRY OF RATIONAL FUNCTIONS.**

> *The link between groups and functions.*

> 60 . Let $F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $F_1, F_\alpha, F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don't alter the function $F_1$ obviously form a group, which one can call the *function's group*.

(I would have thought crystallographic groups were defined as stabilizers before Jordan, but [it seems not](http://www.clarku.edu/~djoyce/wallpaper/history.html).)

**EDIT:** To your recast question ("first published book or article where symmetries are defined as transformations"), I think a problem is that early sources won't have the word **symmetries**: instead you'll find expressions like **transformations qui laissent invariant**, **Transformationen in sich** ([Klein-Lie 1871](https://archive.org/stream/gesammeltemath01kleirich#page/n441); [Klein 1893, p. 326](https://archive.org/stream/eininhohere01kleirich#page/n9)), **Deckoperationen** ([Schoenflies 1891, p. 13](https://books.google.com/books?id=HWlUAAAAMAAJ&pg=PA13); [Love 1906, p. 147](https://archive.org/stream/treatiseonmathem00loveuoft#page/147)), or at best **symmetry-operations** ([Hilton 1903, p. 32](https://archive.org/stream/mathematicalcry02hiltgoog#page/n48)).

The earliest I can find who uses "**a** symmetry" to mean **a** transformation is J. A. Todd, [*The groups of symmetries of the regular polytopes* (1931, p. 214)](http://dx.doi.org/10.1017/S030500410001032X), but he doesn't frame this as a definition. The earliest such *definition* I can find is in H. S. M. Coxeter, [*Regular skew polyhedra in three and four dimensions and their topological analogues* (1937, p. 35)](http://dx.doi.org/10.1112/plms/s2-43.1.33):

>We define a *symmetry* (or "symmetry operation") of any figure as a congruent transformation of the figure into itself (i.e., a combination of translations, rotations, and reflections).

Coxeter makes the same definition in [*Mathematical Recreations and Essays* (1939, p. 130)](http://dx.doi.org/10.1090/S0002-9904-1940-07170-8):

>The vague statement that a figure has a certain amount of "symmetry" can be made precise by saying that the figure has a certain number of *symmetries*, a symmetry* being defined as any combination of motions and reflections which leaves the figure unchanged as a whole.

> *Or "symmetry operation."

and again in [*Regular complex polytopes* (1974, p. 2)](https://books.google.com/books?id=9BY9AAAAIAAJ&pg=PA2):

>Any isometry that leaves a figure invariant as a whole (while possibly
permuting parts of the figure) is called a *symmetry operation* (or simply,
a 'symmetry').