I have a (given) realvalued function defined over an area, $b(x)$, with $x\in \Omega \subset \mathbb{R}^2$. I would like to find a smooth realvalued function $a(x)$ that maximises $J = \int_{x \in \Omega} a^3(x) + a(x)b(x)~dx$ with the constraint $\int_{\Omega} a(x) ~dx=0$. If it helps prevent unbounded or irregular solutions I am happy to introduce further constraints or a higherorder penalty, eg, $J = \int_{x\in\Omega} a^3(x) + a(x)b(x)  \epsilon a^4(x)~dx$ (with the same constraint). How would you solve this? Do I need to specify boundary conditions on $a$?
If we choose as a domain of the functional e.g. the space $L^3(\Omega)$ (with the constraint), and we assume $b\in L^{3/2}(\Omega),$ then $J$ is smooth, and if $\sup b(x)<\infty$ it has a bunch of critical points, all of the form described in the first answer, that is, all the $a(x)$ satisfying $3a(x)^2+b(x)=c$, where the constant $c\geq\sup b$ is arbitrary while the sign of $a(x)$ only has to satisfy the constraint condition $\int_\Omega a(x)\ dx=0$. The second variation of $J$ at a critical point $a(x)$ is $D^2J(a)[h]^2= 6\int_\Omega a(x)h^2(x)\ dx$. Thus, any critical point is neither a local minimizer nor a local maximizer (unless $a=0$, that only happens if $b$ is a constant and the problem is 1 dimensional). Moreover, $J$ is unbounded (choose a function $a(x)$ with $\int_\Omega a(x)dx=0$ and $\int_\Omega a^3(x)dx\neq 0$, then for real $t$, $J(ta)$ is a third degree polynomial in $t$, thus unbounded both from above and from below). If you take the perturbation with $\varepsilon>0$ the situation becomes more delicate . The natural domain is $L^4(\Omega)$ (with the constraint); assuming $b\in L^{4/3}(\Omega)$, $J$ is again smooth, unbounded from below, but now it is bounded from above; again it has a bunch of critical points, characterized by $3a(x)^24\varepsilon a^3(x)+b(x)=c$. There are no local minimizers (unless $b$ is constant) for the second variation $6\int_\Omega \left( a(x)4\varepsilon a^2(x)\right) h^2(x)\ dx$ is not negative. It's not immediate to decide whether there are local or global maxima; certainly they can't be continuous since they have both to change sign and to verify $a(x)\leq 4\varepsilon a^2(x)$. 


It is easy to compute the critical points of $J$. If you formally derive in $a$, you get $\int_\Omega 3a^2+b$, so that the minimizer $a_0$ should satisfy $3a_0^2+b=\mathrm{const}$ (apply $J$ to $a_0+\varepsilon h$ with $\varepsilon$ small and $h$ an arbitrary function of null integral, if you prefer). There are two choices left (the constant and the sign function of $a$). 

