IIUC you have in mind the definition of an object's {symmetries} as its stabilizer under some group action on an ambient space. While a clearcut origin may not exist (Grattan-Guinness warns: "One of the pitfalls in doing the history of mathematics is "priorities": of accepting, as valid, questions of the form, "Who was the first to...?""), I'd say Jordan has early approximations to what you want, provided you are willing to replace "function" by "transformation", "substitution", or "motion".
E.g. his Traité des substitutions (1870, p. 50) has:
§V. -- SYMMETRY OF RATIONAL FUNCTIONS.
The link between groups and functions.
60 . Let $F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $F_1, F_\alpha, F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don't alter the function $F_1$ obviously form a group, which one can call the function's group.
(I would have thought crystallographic groups were defined as stabilizers before Jordan, but it seems not.)
EDIT: To your recast question ("first published book or article where symmetries are defined as transformations"), I think a problem is that early sources won't have the word symmetries: instead you'll find expressions like mouvements qui superposent à lui-même (Jordan 1868, p. 180), Transformationen in sich (Klein-Lie 1871; Klein 1893, p. 326), Deckoperationen (Schoenflies 1891, p. 13; Love 1906, p. 147), or at best symmetry-operations (Hilton 1903, p. 32).
The earliest I can find who uses "a symmetry" to mean a transformation is J. A. Todd, The groups of symmetries of the regular polytopes (1931, p. 214), but he doesn't frame this as a definition. The earliest such definition I can find is in H. S. M. Coxeter, Regular skew polyhedra in three and four dimensions and their topological analogues (1937, p. 35):
We define a symmetry (or "symmetry operation") of any figure as a congruent transformation of the figure into itself (i.e., a combination of translations, rotations, and reflections).
Coxeter makes the same definition in Mathematical Recreations and Essays (1939, p. 130):
The vague statement that a figure has a certain amount of "symmetry" can be made precise by saying that the figure has a certain number of symmetries, a symmetry* being defined as any combination of motions and reflections which leaves the figure unchanged as a whole.
*Or "symmetry operation."
and again in Regular complex polytopes (1974, p. 2):
Any isometry that leaves a figure invariant as a whole (while possibly permuting parts of the figure) is called a symmetry operation (or simply, a 'symmetry').