IIUC you have in mind the definition of an object's **{symmetries}** as its stabilizer under some group action on its ambient space. While a clearcut origin may not exist (Grattan-Guinness [warns](http://www.ams.org/mathscinet-getitem?mr=533433): "One of the pitfalls in doing the history of mathematics is "priorities": of accepting, as valid, questions of the form, "Who was the first to...?""), I'd say **Jordan** has early approximations to what you want, provided you are willing to replace "function" by "transformation", "substitution", or "motion".

E.g. his [Traité des substitutions (1870, p. 50)](https://archive.org/stream/traitdessubsti00jorduoft#page/50) has:

> **§V. -- SYMMETRY OF RATIONAL FUNCTIONS.**

> *The link between groups and functions.*

> 60 . Let $F_1$ be an arbitrary rational function of $k$ letters $a, b, c,\dots$; $F_1, F_\alpha, F_\beta$ the $1.2.\dots k$ functions obtained by letting the $1.2.\dots k$ substitutions $1, \alpha, \beta,\dots$ operate on these letters (...) The $M$ substitutions $1,\alpha,\dots$ which don't alter the function $F_1$ obviously form a group, which one can call the *function's group*.

(I would have thought crystallographic groups were defined as stabilizers before Jordan, but [it seems not](http://www.clarku.edu/~djoyce/wallpaper/history.html).)