IIUC you have in mind the definition of an object's **{symmetries}** as its stabilizer under some group action on an ambient space. While a clearcut origin may not exist (Grattan-Guinness [warns](http://www.ams.org/mathscinet-getitem?mr=533433): "One of the pitfalls in doing the history of mathematics is "priorities": of accepting, as valid, questions of the form, "Who was the first to...?""), I'd say **Jordan** has early approximations to what you want, provided you are willing to replace "function" by "transformation", "substitution", or "motion".

E.g. his [*Traité des substitutions* (1870, p. 50)](https://archive.org/stream/traitdessubsti00jorduoft#page/50) has:

> **§V. -- SYMMETRY OF RATIONAL FUNCTIONS.**

> *The link between groups and functions.*

> 60 . Let $F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $F_1, F_\alpha, F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don't alter the function $F_1$ obviously form a group, which one can call the *function's group*.

(I would have thought crystallographic groups were defined as stabilizers before Jordan, but [it seems not](http://www.clarku.edu/~djoyce/wallpaper/history.html).)

**Edit:** To your recast question ("first published book or article"), I think a problem is that early sources won't have the word **symmetries**: instead you'll find expressions like **transformations qui laissent invariant**, **Transformationen in sich** ([Klein-Lie 1871](https://archive.org/stream/gesammeltemath01kleirich#page/n441); [Klein 1893, p. 326](https://archive.org/stream/eininhohere01kleirich#page/n9)), **Deckoperationen** ([Schoenflies 1891, p. 13](https://books.google.com/books?id=HWlUAAAAMAAJ&pg=PA13)), **covering operations** ([Love 1906, p. 147](https://archive.org/stream/treatiseonmathem00loveuoft#page/147)). The closest I can find is **symmetry-operations** in Harold Hilton, [*Mathematical crystallography and the theory of groups of movements* (1903, p. 32)](https://archive.org/stream/mathematicalcry02hiltgoog#page/n48):

> If $U$, $U'$ be any two congruent (...) figures then $U$ can be brought into coincidence with $U'$ by a rotation, a translation, a combination of rotation and translation, or a rotatory-reflexion.

>Now if *$U'$ is identical with $U$*, $U$ is brought into coincidence with itself by some operation of the kind just described. In that case $U$ is said to have *symmetry*, and the operation which brings $U$ to self-coincidence is called a *symmetry-operation* of $U$. 

So early authors didn't define **a** symmetry as **a** transformation, but rather **the** symmetry as a group thereof. Such is still the case with Hermann Weyl in 1928 (*Gruppentheorie und Quantenmechanik*, p. 99, [translation, p. 112](https://archive.org/stream/quantumtheoryofa029235mbp#page/n137)):
> Nach *Kleins* Erlanger Programm beruht jede Art von Geometrie in einem Punktfeld auf einer bestimmten Transformationsgruppe $\mathfrak G$ des Punktfeldes (...) Die Symmetrie einer speziellen Figur in einem solchen Raum wird beschrieben durch eine Untergruppe von $\mathfrak G$, bestehend aus allen Transformationen von $\mathfrak G$, welche die Figur in sich überführen.

and even in 1952 ([*Symmetry*, p. 45](https://archive.org/stream/Symmetry_482/Weyl-Symmetry#page/n29)):

> What has all this to do with symmetry? It provides the adequate mathematical language to define it. Given a spatial configuration $\mathfrak F$, those automorphisms of space which leave $\mathfrak F$ unchanged form a group $\Gamma$, and *this group describes exactly the symmetry possessed by $\mathfrak F$.*

and even today in the [Online Dictionary of Crystallography](http://reference.iucr.org/dictionary/Symmetry_operation). So let me return the question: where do *you* find **a** symmetry defined as **a** transformation?