IIUC you have in mind the definition of an object's {symmetries} as its stabilizer under some group action on an ambient space. While a clearcut origin may not exist (Grattan-Guinness warns: "One of the pitfalls in doing the history of mathematics is "priorities": of accepting, as valid, questions of the form, "Who was the first to...?""), I'd say Jordan has early approximations to what you want, provided you are willing to replace "function" by "transformation", "substitution", or "motion".
E.g. his Traité des substitutions (1870, p. 50) has:
§V. -- SYMMETRY OF RATIONAL FUNCTIONS.
The link between groups and functions.
60 . Let $F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $F_1, F_\alpha, F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don't alter the function $F_1$ obviously form a group, which one can call the function's group.
(I would have thought crystallographic groups were defined as stabilizers before Jordan, but it seems not.)
Edit: To your recast question ("first published book or article"), I think a problem is that early sources won't have the word symmetries: instead you'll find expressions like transformations qui laissent invariant, Transformationen in sich (Klein-Lie 1871; Klein 1893, p. 326), Deckoperationen (Schoenflies 1891, p. 13), covering operations (Love 1906, p. 147). The closest I can find is symmetry-operations in Harold Hilton, Mathematical crystallography and the theory of groups of movements (1903, p. 32):
If $U$, $U'$ be any two congruent (...) figures then $U$ can be brought into coincidence with $U'$ by a rotation, a translation, a combination of rotation and translation, or a rotatory-reflexion.
Now if $U'$ is identical with $U$, $U$ is brought into coincidence with itself by some operation of the kind just described. In that case $U$ is said to have symmetry, and the operation which brings $U$ to self-coincidence is called a symmetry-operation of $U$.
So early authors didn't define a symmetry as a transformation, but rather the symmetry as a group thereof. Such is still the case with Hermann Weyl in 1928 (Gruppentheorie und Quantenmechanik, p. 99, translation, p. 112):
Nach Kleins Erlanger Programm beruht jede Art von Geometrie in einem Punktfeld auf einer bestimmten Transformationsgruppe $\mathfrak G$ des Punktfeldes (...) Die Symmetrie einer speziellen Figur in einem solchen Raum wird beschrieben durch eine Untergruppe von $\mathfrak G$, bestehend aus allen Transformationen von $\mathfrak G$, welche die Figur in sich überführen.
and even in 1952 (Symmetry, p. 45):
What has all this to do with symmetry? It provides the adequate mathematical language to define it. Given a spatial configuration $\mathfrak F$, those automorphisms of space which leave $\mathfrak F$ unchanged form a group $\Gamma$, and this group describes exactly the symmetry possessed by $\mathfrak F$.
So let me return the question: where do you find a symmetry defined as a transformation?