IIUC you have in mind the definition of an object's {symmetries} as its stabilizer under some group action on its ambient space. While a clearcut origin may not exist (Grattan-Guinness warns: "One of the pitfalls in doing the history of mathematics is "priorities": of accepting, as valid, questions of the form, "Who was the first to...?""), I'd say Jordan has early approximations to what you want, provided you are willing to replace "function" by "transformation", "substitution", or "motion".
E.g. his Traité des substitutions (1870, p. 50) has:
§V. -- SYMMETRY OF RATIONAL FUNCTIONS.
The link between groups and functions.
60 . Let $F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $F_1, F_\alpha, F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don't alter the function $F_1$ obviously form a group, which one can call the function's group.
(I would have thought crystallographic groups were defined as stabilizers before Jordan, but it seems not.)
Edit: To your added question ("first published book or article where symmetries are defined as transformations?"), I think a problem is that early sources won't have the word symmetries: instead you'll find expressions like transformations qui laissent invariant, or Transformationen in sich (e.g. Klein and Lie here or here (p. 326)). For all I know, the definition you want might not appear before Hermann Weyl, Symmetry (1952, p. 45):
What has all this to do with symmetry? It provides the adequate mathematical language to define it. Given a spatial configuration $\mathfrak F$, those automorphisms of space which leave $\mathfrak F$ unchanged form a group $\Gamma$, and this group describes exactly the symmetry possessed by $\mathfrak F$.