IIUC you have in mind the definition of an object's **{symmetries}** as its stabilizer under some group action on its ambient space. While a clearcut origin may not exist (Grattan-Guinness [warns](http://www.ams.org/mathscinet-getitem?mr=533433): "One of the pitfalls in doing the history of mathematics is "priorities": of accepting, as valid, questions of the form, "Who was the first to...?""), I'd say **Jordan** has early approximations to what you want, provided you are willing to replace "function" by "transformation", "substitution", or "motion".

E.g. his [*Traité des substitutions* (1870, p. 50)](https://archive.org/stream/traitdessubsti00jorduoft#page/50) has:

> **§V. -- SYMMETRY OF RATIONAL FUNCTIONS.**

> *The link between groups and functions.*

> 60 . Let $F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $F_1, F_\alpha, F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don't alter the function $F_1$ obviously form a group, which one can call the *function's group*.

(I would have thought crystallographic groups were defined as stabilizers before Jordan, but [it seems not](http://www.clarku.edu/~djoyce/wallpaper/history.html).)

**Edit:** To your added question ("first published book or article where symmetries are defined as transformations?"), I think a problem is that early sources won't have the word **symmetries**: instead you'll find expressions like **transformations qui laissent invariant**, or **Transformationen in sich** (e.g. Klein and Lie [here](https://archive.org/stream/gesammeltemath01kleirich#page/n441) or [here (p. 326)](https://archive.org/stream/eininhohere01kleirich#page/n9)). For all I know, the definition you want might not appear before Hermann Weyl, [*Symmetry* (1952, p. 45)](https://archive.org/stream/Symmetry_482/Weyl-Symmetry#page/n29):

> What has all this to do with symmetry? It provides the adequate mathematical language to define it. Given a spatial configuration $\mathfrak F$, those automorphisms of space which leave $\mathfrak F$ unchanged form a group $\Gamma$, and *this group describes exactly the symmetry possessed by $\mathfrak F$.*