For not necessarily abelian groups, we can interpret the division in (1) as the right or left division.

Let $G$ be any group. The answer to the question is given by

Theorem: The following three conditions are equivalent to one another:

(right): there is a function $f\colon G\to G$ such that \begin{equation*} f(xf(y))=f(f(x))y^{-1} \label{r}\tag{$r$} \end{equation*} for all $x$ and $y$ (in $G$);

(left): there is a function $f\colon G\to G$ such that \begin{equation*} f(xf(y))=y^{-1}f(f(x)) \label{l}\tag{$l$} \end{equation*} for all $x$ and $y$;

(involutive) $x^{-1}=x$ for all $x$ (and hence $G$ is abelian).

Proof of implication ($x^{-1}=x$ for all $x$)$\implies$ $G$ is abelian: For all $x$ and $y$ we have $xy=(xy)^{-1}=y^{-1}x^{-1}=yx$, so that $xy=yx$, as claimed. $\Box$

Proof of implications (involutive)$\implies$(right) and (involutive)$\implies$(left). Suppose the (involutive) property holds, so that $G$ is abelian. Let $f(x)=x$ for all $x$. Then $f(xf(y))=xy=f(f(x))y^{-1}$ for all $x,y$, so that condition \eqref{r} holds. Since $G$ is abelian, condition \eqref{l} holds as well. $\Box$

Proof of implication (right)$\implies$ (involutive): Substituting $x=1$ in \eqref{r}, we get $f(f(y))=f(b)y^{-1}$ (for all $y$), where \begin{equation*} b:=f(1). \end{equation*} So, $f(f(x))=f(b)x^{-1}$, and now \eqref{r} yields $f(xf(y))=f(b)x^{-1}y^{-1}$. Substituting here $y=1$, we get $f(xb)=f(b)x^{-1}$ or, equivalently, \begin{equation*} f(z)=cz^{-1} \label{2}\tag{2} \end{equation*} for all $z$, where $c:=f(b)b$. Now \eqref{r} can be rewritten as \begin{equation*} yc^{-1}x^{-1}=xc^{-1}y^{-1}. \label{r'}\tag{$r'$} \end{equation*} Substituting here $y=1$ and $x=c$, we get $c^{-2}=1$, that is, $c^{-1}=c$. Taking now any $z\in G$ and letting $y=zxc$, we rewrite \eqref{r'} as $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$

Proof of implication (left)$\implies$ (involutive): Substituting $x=1$ in \eqref{l}, we get $f(f(y))=y^{-1}f(b)$, where $b=f(1)$, as before. So, $f(f(x))=x^{-1}f(b)$, and now \eqref{l} yields $f(xf(y))=y^{-1}x^{-1}f(b)$. Substituting here $y=1$, we get $f(xb)=x^{-1}f(b)$ or, equivalently, \begin{equation*} f(z)=bz^{-1}d \end{equation*} for all $z$, where $d:=f(b)$. Now \eqref{l} can be rewritten as \begin{equation*} bd^{-1}yb^{-1}x^{-1}=y^{-1}bd^{-1}xb^{-1}. \label{l'}\tag{$l'$} \end{equation*} Substituting here $y=1$ and $x=b$, we get $b^{-2}=1$, that is, $b^{-1}=b$.
Substituting in \eqref{l'} $y=b$ and $x=d$, we get $bd^{-2}=b^{-1}=b$ and hence $d^{-2}=1$, that is, $d^{-1}=d$. Now \eqref{l'} becomes \begin{equation*} bdybx^{-1}=y^{-1}bdxb. \label{l2}\tag{$l''$} \end{equation*} Substituting here $y=b$ and $x=1$, we get $bd=db$.

Taking now any $z\in G$ and letting $y=bdz$, so that $z=bdy$, we rewrite \eqref{l2} as \begin{equation*} zbx^{-1}=z^{-1}xb. \end{equation*} Substituting here $x=b$, we get $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$

The theorem is now completely proved.

Remark: As seen from \eqref{2}, if any one of the three equivalent conditions -- (right), (left), or (involutive) -- holds, then the solutions of equation \eqref{r} and/or, equivalently, \eqref{l} are precisely those of the form $f(x)=cx$ for some $c$ and all $x$.

For not necessarily abelian groups, we can interpret the division in (1) as the right or left division.

Let $G$ be any group. The answer to the question is given by

Theorem: The following three conditions are equivalent to one another:

(right): there is a function $f\colon G\to G$ such that \begin{equation*} f(xf(y))=f(f(x))y^{-1} \label{r}\tag{$r$} \end{equation*} for all $x$ and $y$ (in $G$);

(left): there is a function $f\colon G\to G$ such that \begin{equation*} f(xf(y))=y^{-1}f(f(x)) \label{l}\tag{$l$} \end{equation*} for all $x$ and $y$;

(involutive) $x^{-1}=x$ for all $x$ (and hence $G$ is abelian).

Proof of implication ($x^{-1}=x$ for all $x$)$\implies$ $G$ is abelian: For all $x$ and $y$ we have $xy=(xy)^{-1}=y^{-1}x^{-1}=yx$, so that $xy=yx$, as claimed. $\Box$

Proof of implications (involutive)$\implies$(right) and (involutive)$\implies$(left). Suppose the (involutive) property holds, so that $G$ is abelian. Let $f(x)=x$ for all $x$. Then $f(xf(y))=xy=f(f(x))y^{-1}$ for all $x,y$, so that condition \eqref{r} holds. Since $G$ is abelian, condition \eqref{l} holds as well. $\Box$

Proof of implication (right)$\implies$ (involutive): Substituting $x=1$ in \eqref{r}, we get $f(f(y))=f(b)y^{-1}$ (for all $y$), where \begin{equation*} b:=f(1). \end{equation*} So, $f(f(x))=f(b)x^{-1}$, and now \eqref{r} yields $f(xf(y))=f(b)x^{-1}y^{-1}$. Substituting here $y=1$, we get $f(xb)=f(b)x^{-1}$ or, equivalently, \begin{equation*} f(z)=cz^{-1} \label{2}\tag{2} \end{equation*} for all $z$, where $c:=f(b)b$. Now \eqref{r} can be rewritten as \begin{equation*} yc^{-1}x^{-1}=xc^{-1}y^{-1}. \label{r'}\tag{$r'$} \end{equation*} Substituting here $y=1$ and $x=c$, we get $c^{-2}=1$, that is, $c^{-1}=c$. Taking now any $z\in G$ and letting $y=zxc$, we rewrite \eqref{r'} as $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$

Proof of implication (left)$\implies$ (involutive): Substituting $x=1$ in \eqref{l}, we get $f(f(y))=y^{-1}f(b)$, where $b=f(1)$, as before. So, $f(f(x))=x^{-1}f(b)$, and now \eqref{l} yields $f(xf(y))=y^{-1}x^{-1}f(b)$. Substituting here $y=1$, we get $f(xb)=x^{-1}f(b)$ or, equivalently, \begin{equation*} f(z)=bz^{-1}d \end{equation*} for all $z$, where $d:=f(b)$. Now \eqref{l} can be rewritten as \begin{equation*} bd^{-1}yb^{-1}x^{-1}=y^{-1}bd^{-1}xb^{-1}. \label{l'}\tag{$l'$} \end{equation*} Substituting here $y=1$ and $x=b$, we get $b^{-2}=1$, that is, $b^{-1}=b$.
Substituting in \eqref{l'} $y=b$ and $x=d$, we get $bd^{-2}=b^{-1}=b$ and hence $d^{-2}=1$, that is, $d^{-1}=d$. Now \eqref{l'} becomes \begin{equation*} bdybx^{-1}=y^{-1}bdxb. \label{l2}\tag{$l''$} \end{equation*} Substituting here $y=b$ and $x=1$, we get $bd=db$.

Taking now any $z\in G$ and letting $y=bdz$, so that $z=bdy$, we rewrite \eqref{l2} as \begin{equation*} zbx^{-1}=z^{-1}xb. \end{equation*} Substituting here $x=b$, we get $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$

The theorem is now completely proved.

Remark: As seen from \eqref{2}, if any one of the three equivalent conditions -- (right), (left), or (involutive) -- holds, then the solutions of equation \eqref{r} and/or, equivalently, \eqref{l} are precisely those of the form $f(x)=cx$ for some $c$ and all $x$.

Corollary: Neither the (right) nor the (left) property can hold for any non-abelian group $G$.

For not necessarily abelian groups, we can interpret the division in (1) as the right or left division.

Let $G$ be any group. The answer to the question is given by

Theorem: The following three conditions are equivalent to one another:

(right): there is a function $f\colon G\to G$ such that \begin{equation*} f(xf(y))=f(f(x))y^{-1} \label{r}\tag{$r$} \end{equation*} for all $x$ and $y$ (in $G$);

(left): there is a function $f\colon G\to G$ such that \begin{equation*} f(xf(y))=y^{-1}f(f(x)) \label{l}\tag{$l$} \end{equation*} for all $x$ and $y$;

(involutive) $x^{-1}=x$ for all $x$ (and hence $G$ is abelian).

Proof of implication ($x^{-1}=x$ for all $x$)$\implies$ $G$ is abelian: For all $x$ and $y$ we have $xy=(xy)^{-1}=y^{-1}x^{-1}=yx$, so that $xy=yx$, as claimed. $\Box$

Proof of implications (involutive)$\implies$(right) and (involutive)$\implies$(left). Suppose the (involutive) property holds, so that $G$ is abelian. Let $f(x)=x$ for all $x$. Then $f(xf(y))=xy=f(f(x))y^{-1}$ for all $x,y$, so that condition \eqref{r} holds. Since $G$ is abelian, condition \eqref{l} holds as well. $\Box$

Proof of implication (right)$\implies$ (involutive): Substituting $x=1$ in \eqref{r}, we get $f(f(y))=f(b)y^{-1}$ (for all $y$), where \begin{equation*} b:=f(1). \end{equation*} So, $f(f(x))=f(b)x^{-1}$, and now \eqref{r} yields $f(xf(y))=f(b)x^{-1}y^{-1}$. Substituting here $y=1$, we get $f(xb)=f(b)x^{-1}$ or, equivalently, \begin{equation*} f(z)=cz^{-1} \label{2}\tag{2} \end{equation*} for all $z$, where $c:=f(b)b$. Now \eqref{r} can be rewritten as \begin{equation*} yc^{-1}x^{-1}=xc^{-1}y^{-1}. \label{r'}\tag{$r'$} \end{equation*} Substituting here $y=1$ and $x=c$, we get $c^{-2}=1$, that is, $c^{-1}=c$. Taking now any $z\in G$ and letting $y=zxc$, we rewrite \eqref{r'} as $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$

Proof of implication (left)$\implies$ (involutive): Substituting $x=1$ in \eqref{l}, we get $f(f(y))=y^{-1}f(b)$, where $b=f(1)$, as before. So, $f(f(x))=x^{-1}f(b)$, and now \eqref{l} yields $f(xf(y))=y^{-1}x^{-1}f(b)$. Substituting here $y=1$, we get $f(xb)=x^{-1}f(b)$ or, equivalently, \begin{equation*} f(z)=bz^{-1}d \end{equation*} for all $z$, where $d:=f(b)$. Now \eqref{l} can be rewritten as \begin{equation*} bd^{-1}yb^{-1}x^{-1}=y^{-1}bd^{-1}xb^{-1}. \label{l'}\tag{$l'$} \end{equation*} Substituting here $y=1$ and $x=b$, we get $b^{-2}=1$, that is, $b^{-1}=b$.
Substituting in \eqref{l'} $y=b$ and $x=d$, we get $bd^{-2}=b^{-1}=b$ and hence $d^{-2}=1$, that is, $d^{-1}=d$. Now \eqref{l'} becomes \begin{equation*} bdybx^{-1}=y^{-1}bdxb. \label{l2}\tag{$l''$} \end{equation*} Substituting here $y=b$ and $x=1$, we get $bd=db$.

Taking now any $z\in G$ and letting $y=bdz$, so that $z=bdy$, we rewrite \eqref{l2} as \begin{equation*} zbx^{-1}=z^{-1}xb. \end{equation*} Substituting here $x=b$, we get $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$

The theorem is now completely proved.

*Remark: * As seen from \eqref{2}, if any one of the three equivalent conditions -- (right), (left), or (involutive) -- holds, then the solutions of equation \eqref{r} and/or, equivalently, \eqref{l} are precisely those of the form $f(x)=cx$ for some $c$ and all $x$.

For not necessarily abelian groups, we can interpret the division in (1) as the right or left division.

Let $G$ be any group. The answer to the question is given by

Theorem: The following three conditions are equivalent to one another:

(right): there is a function $f\colon G\to G$ such that \begin{equation*} f(xf(y))=f(f(x))y^{-1} \label{r}\tag{$r$} \end{equation*} for all $x$ and $y$ (in $G$);

(left): there is a function $f\colon G\to G$ such that \begin{equation*} f(xf(y))=y^{-1}f(f(x)) \label{l}\tag{$l$} \end{equation*} for all $x$ and $y$;

(involutive) $x^{-1}=x$ for all $x$ (and hence $G$ is abelian).

Proof of implication ($x^{-1}=x$ for all $x$)$\implies$ $G$ is abelian: For all $x$ and $y$ we have $xy=(xy)^{-1}=y^{-1}x^{-1}=yx$, so that $xy=yx$, as claimed. $\Box$

Proof of implications (involutive)$\implies$(right) and (involutive)$\implies$(left). Suppose the (involutive) property holds, so that $G$ is abelian. Let $f(x)=x$ for all $x$. Then $f(xf(y))=xy=f(f(x))y^{-1}$ for all $x,y$, so that condition \eqref{r} holds. Since $G$ is abelian, condition \eqref{l} holds as well. $\Box$

Proof of implication (right)$\implies$ (involutive): Substituting $x=1$ in \eqref{r}, we get $f(f(y))=f(b)y^{-1}$ (for all $y$), where \begin{equation*} b:=f(1). \end{equation*} So, $f(f(x))=f(b)x^{-1}$, and now \eqref{r} yields $f(xf(y))=f(b)x^{-1}y^{-1}$. Substituting here $y=1$, we get $f(xb)=f(b)x^{-1}$ or, equivalently, \begin{equation*} f(z)=cz^{-1} \label{2}\tag{2} \end{equation*} for all $z$, where $c:=f(b)b$. Now \eqref{r} can be rewritten as \begin{equation*} yc^{-1}x^{-1}=xc^{-1}y^{-1}. \label{r'}\tag{$r'$} \end{equation*} Substituting here $y=1$ and $x=c$, we get $c^{-2}=1$, that is, $c^{-1}=c$. Taking now any $z\in G$ and letting $y=zxc$, we rewrite \eqref{r'} as $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$

Proof of implication (left)$\implies$ (involutive): Substituting $x=1$ in \eqref{l}, we get $f(f(y))=y^{-1}f(b)$, where $b=f(1)$, as before. So, $f(f(x))=x^{-1}f(b)$, and now \eqref{l} yields $f(xf(y))=y^{-1}x^{-1}f(b)$. Substituting here $y=1$, we get $f(xb)=x^{-1}f(b)$ or, equivalently, \begin{equation*} f(z)=bz^{-1}d \end{equation*} for all $z$, where $d:=f(b)$. Now \eqref{l} can be rewritten as \begin{equation*} bd^{-1}yb^{-1}x^{-1}=y^{-1}bd^{-1}xb^{-1}. \label{l'}\tag{$l'$} \end{equation*} Substituting here $y=1$ and $x=b$, we get $b^{-2}=1$, that is, $b^{-1}=b$.
Substituting in \eqref{l'} $y=b$ and $x=d$, we get $bd^{-2}=b^{-1}=b$ and hence $d^{-2}=1$, that is, $d^{-1}=d$. Now \eqref{l'} becomes \begin{equation*} bdybx^{-1}=y^{-1}bdxb. \label{l2}\tag{$l''$} \end{equation*} Substituting here $y=b$ and $x=1$, we get $bd=db$.

Taking now any $z\in G$ and letting $y=bdz$, so that $z=bdy$, we rewrite \eqref{l2} as \begin{equation*} zbx^{-1}=z^{-1}xb. \end{equation*} Substituting here $x=b$, we get $z=z^{-1}$, which means $G$ has the (involutive) property. $\Box$

The theorem is now completely proved.

Remark: As seen from \eqref{2}, if any one of the three equivalent conditions -- (right), (left), or (involutive) -- holds, then the solutions of equation \eqref{r} and/or, equivalently, \eqref{l} are precisely those of the form $f(x)=cx$ for some $c$ and all $x$.