Functional derivative of the square of an integral I have the following functional 
$$F(y)=\left[\int \frac{1}{y(x)+A}\cos(x)dx\right]^2$$
How do I find the functional derivative $dF$?
(I never encountered the square of an integral before when I did functional derivatives)
 A: Replace $y$ by $y+t\delta y$ and then compute
$$\lim_{t\to 0} \Bigl(\; F(y+t\delta y)-F(y)\;\Bigr). $$
Here are the details. Observe first that
$$\frac{1}{y+ A+ t\delta y}-\frac{1}{y+A} = -\frac{1}{(y+A)^2} t\delta y+ O(t^2),$$
so that
$$F(y+t\delta y)= \left(\int\;\Bigl(\; \frac{1}{y+A} -\frac{1}{(y+A)^2} t\delta y+O(t^2)\;\Bigr) \cos(x) dx\right)^2 $$
$$= F(y) -2t\left(\int\frac{\cos (x)}{y+A}\right)\left(\int\frac{(\delta y)\cos(x)}{\bigl(y+A)^2} dx\right) +O(t^2). $$
From here you can read that
$$\frac{\delta F}{\delta y} =-2 \left(\int\frac{\cos (x)}{y+A}dx\right)\frac{\cos(x)}{\bigl(\;y(x)+A \;\bigr)^2} $$
Update 1. Here are some rules that will help you solve the problem you mentioned in your comment.
Suppose that $I$ is an interval.  If
$$F(y) =\int_I f(y(x)) w(x) dx.$$
Then
$$\frac{\delta F}{\delta y}= f'(y(x)) w(x). \tag{1}  $$
If   
$$ F(y)=\left| \int_I  f(y)  w(x) dx\right|^2, $$
$w$ complex valued, then $\DeclareMathOperator{\re}{\boldsymbol{Re}}$
$$\frac{\delta F}{\delta y}=  2\re\left(\; f'(y(x)) \overline{w(x)} \int_I f(y) w(x) dx \;\right) \tag{2} $$
Let me set 
$$ F_1(y)=\int_I \frac{A}{y(x)+ A} dx,\;\;I=(-\kappa \pi,\kappa\pi), $$
$$ F_2(y) =\left|\int_I \frac{A}{y(x)+ A} e^{- i x} dx\right|^2, $$
$$ E(y) = F_1-\frac{F_2}{F_1} $$
Then
$$ \frac{\delta E}{\delta y} =\frac{\delta F_1}{\delta y} -\frac{ \frac{\delta F_2}{\delta y} F_1- F_2\frac{\delta F_1}{\delta y} }{ F_1(y)^2 }. $$
Now compute the various derivatives using (1) and (2).   
Your problem reduces via Lagrange multipliers to the system
$$\frac{\delta E}{\delta y} =\lambda,\;\;\int_I y dx=1. $$
Update 2.   Introduce a new  variable
$$ y=\frac{A}{G+A},\;\;G= \frac{A}{y}-A. $$
The constraint 
$$ \int G = const $$ 
becomes 
$$C(y)= \int\frac{1}{y} =const $$
and the energy functional becomes
$$E(y)=\int y -\frac{\left|\int ye^{-i x}\right|^2}{\int y }. $$
The constrained Euler-Lagrange equations have the form
$$\frac{\delta E}{\delta y}=\lambda\frac{\delta C}{\delta y}=-\frac{\lambda}{y^2} \tag{3} $$
Which   translates  into an equality of the form
$$C_1(y) y^2+ C_2(y) \cos x +C_3(y) \sin x= C_4(y)-\lambda C_1(y)^2, $$
where $C_i(y)$ are constants that depend explicitly on $y$, e.g.,
$$C_1(y)=\left(\int y\right)^2. $$
Now at least you know  that $y$ must be of a rather special form
$$ y =\sqrt{ A_1+A_2\cos x+ A_3\sin x}. $$  If you play with (3) some more I bet that you can  extract more precise info.
