For not necessarily abelian groups, we can interpret the division in (1) as the right or left division.
Let $G$ be any group. The answer to the question is given by
Theorem: The following three conditions are equivalent to one another:
(right): there is a function $f\colon G\to G$ such that
\begin{equation*}
f(xf(y))=f(f(x))y^{-1} \label{r}\tag{$r$}
\end{equation*}
for all $x$ and $y$ (in $G$);
(left): there is a function $f\colon G\to G$ such that
\begin{equation*}
f(xf(y))=y^{-1}f(f(x)) \label{l}\tag{$l$}
\end{equation*}
for all $x$ and $y$;
(involutive) $x^{-1}=x$ for all $x$ (and hence $G$ is abelian).
Proof of implication ($x^{-1}=x$ for all $x$)$\implies$ $G$ is abelian: For all $x$ and $y$ we have $xy=(xy)^{-1}=y^{-1}x^{-1}=yx$, so that $xy=yx$, as claimed. $\Box$
Proof of implications (involutive)$\implies$(right) and (involutive)$\implies$(left). Suppose the (involutive) property holds, so that $G$ is abelian. Let $f(x)=x$ for all $x$. Then $f(xf(y))=xy=f(f(x))y^{-1}$ for all $x,y$, so that condition \eqref{r} holds. Since $G$ is abelian, condition \eqref{l} holds as well. $\Box$
Proof of implication (right)$\implies$ (involutive): Substituting $x=1$ in \eqref{r}, we get $f(f(y))=f(b)y^{-1}$ (for all $y$), where
\begin{equation*}
b:=f(1).
\end{equation*}
So, $f(f(x))=f(b)x^{-1}$, and now \eqref{r} yields $f(xf(y))=f(b)x^{-1}y^{-1}$. Substituting here $y=1$, we get $f(xb)=f(b)x^{-1}$ or, equivalently,
\begin{equation*}
f(z)=cz^{-1} \label{2}\tag{2}
\end{equation*}
for all $z$, where $c:=f(b)b$. Now \eqref{r} can be rewritten as
\begin{equation*}
yc^{-1}x^{-1}=xc^{-1}y^{-1}. \label{r'}\tag{$r'$}
\end{equation*}
Substituting here $y=1$ and $x=c$, we get $c^{-2}=1$, that is, $c^{-1}=c$. Taking now any $z\in G$ and letting $y=zxc$, we rewrite \eqref{r'} as $z=z^{-1}$, which means $G$ has the (involutive) property.
$\Box$
Proof of implication (left)$\implies$ (involutive): Substituting $x=1$ in \eqref{l}, we get $f(f(y))=y^{-1}f(b)$, where $b=f(1)$, as before.
So, $f(f(x))=x^{-1}f(b)$, and now \eqref{l} yields $f(xf(y))=y^{-1}x^{-1}f(b)$. Substituting here $y=1$, we get $f(xb)=x^{-1}f(b)$ or, equivalently,
\begin{equation*}
f(z)=bz^{-1}d
\end{equation*}
for all $z$, where $d:=f(b)$. Now \eqref{l} can be rewritten as
\begin{equation*}
bd^{-1}yb^{-1}x^{-1}=y^{-1}bd^{-1}xb^{-1}. \label{l'}\tag{$l'$}
\end{equation*}
Substituting here $y=1$ and $x=b$, we get $b^{-2}=1$, that is, $b^{-1}=b$.
Substituting in \eqref{l'} $y=b$ and $x=d$, we get $bd^{-2}=b^{-1}=b$ and hence $d^{-2}=1$, that is, $d^{-1}=d$.
Now \eqref{l'} becomes
\begin{equation*}
bdybx^{-1}=y^{-1}bdxb. \label{l2}\tag{$l''$}
\end{equation*}
Substituting here $y=b$ and $x=1$, we get $bd=db$.
Taking now any $z\in G$ and letting $y=bdz$, so that $z=bdy$, we rewrite \eqref{l2} as
\begin{equation*}
zbx^{-1}=z^{-1}xb.
\end{equation*}
Substituting here $x=b$, we get $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$
The theorem is now completely proved.
Remark: As seen from \eqref{2}, if any one of the three equivalent conditions -- (right), (left), or (involutive) -- holds, then the solutions of equation \eqref{r} and/or, equivalently, \eqref{l} are precisely those of the form $f(x)=cx$ for some $c$ and all $x$.
Corollary: Neither the (right) nor the (left) property can hold for any non-abelian group $G$.
