I am trying to compute the first variation of the functional

$$\mathcal F(\rho) = \int_{\Omega} R(x;\rho) d\rho(x)$$

where $R$ is some function of $x$ that also depends on $\rho$. Here $\rho$ is a probability measure, i.e. $\rho \in \mathcal P(\Omega)$. I'm using the Definition 7.12 from Optimal Transport for Applied Mathematicians by Santambrogio, which defines $\frac{\delta \mathcal F}{\delta \rho}(\rho)$ as a measurable function satisfying

$$\frac{d}{d\epsilon} \mathcal F_1(\rho+\epsilon\chi) \bigg|_{\epsilon=0} = \int \frac{\delta\mathcal F_1}{\delta\rho}(\rho) d\chi$$

for all perturbations $\chi=\tilde\rho-\rho$ with $\tilde\rho\in L_c^\infty(\Omega)\cap\mathcal P(\Omega)$. Since $\mathcal F$ didn't seem to fit into the types of functionals listed in Santambrogio's book, I resorted to computing this directly, which gave me

$$\frac{d}{d\epsilon} \mathcal F_1(\rho+\epsilon\chi) = \int R(x;\rho) d\chi(x) + \frac{d}{d\epsilon} \int R(x;\rho+\epsilon\chi) d\rho(x) + \epsilon \frac{d}{d\epsilon} \int R(x;\rho+\epsilon\chi) d\chi(x).$$

Then I wrote $f(x,\delta) = R(x;\rho+\delta\chi)$ and substituted in the Taylor series $R(x;\rho+\epsilon\chi) = f(x,0) + \epsilon f_\delta(x,\xi_\epsilon)$ for some $\xi_\epsilon \in[0,\epsilon]$. This gave me

$$\frac{d}{d\epsilon} \mathcal F_1(\rho+\epsilon\chi) = \int R(x;\rho) d\chi(x) + \int f_\delta(x,\xi_\epsilon) + \epsilon f_{\delta,\delta}(x,\xi_\epsilon) d(\rho +\epsilon\chi)(x).$$

I then set $\epsilon=0$ so that

$$\frac{d}{d\epsilon} \mathcal F_1(\rho+\epsilon\chi) \bigg|_{\epsilon=0} = \int R(x;\rho) d\chi(x) + \int f_\delta(x,0) d\rho(x).$$

As per the definition, I need to write the remainder as an integral against $\chi$. This is simple for some special cases. For example, if $R(x;\rho)=\log(\rho(x))$ then $\int f_\delta(x,0)d\rho(x) = \int d\chi(x)$ and so $\frac{\delta \mathcal F}{\delta \rho}(\rho)(x) = \log(\rho(x))+1$.

However, this seems to not work as well in general. I had the idea to use the Radon-Nikodym derivative, which would give $\frac{\delta \mathcal F}{\delta \rho}(\rho)(x) = R(x;\rho) + f_\delta(x,0)\frac{d\rho}{d\chi}(x)$, but I don't think $\chi$ should appear in the first variation.

I have two questions:

1. Does this just mean that $\mathcal F$ does not have a first variation for many choices of $R$? Even for sufficiently regular $R$ where the remainder just can't be written as an integral against $\chi$?
2. Are there some references dealing with first variations of functionals like this one? Perhaps with a more general definition to handle the remainder above?

Some more context for my particular setting: I generally don't know the form of $R$, but I know $\rho(x)\nabla_x R(x,\rho) = \mathscr F^{-1}(g\cdot\mathscr F(\nabla\rho))(x)$ where $\mathscr F$ denotes the Fourier transform and $g$ is a known function, which is usually very smooth. In particular, $g=1$ gives the special case $R(x;\rho)=\log(\rho(x))$ mentioned above.