I am having difficulty formulating a problem, which involves optimizing a contour shape, into a well-posed variational form that would give a reasonable answer.

Within a bounded region on the $xy$ plane, say $x\in[-x_{0},x_{0}], y\in[-y_{0},y_{0}]$, we have a continuous scalar field $H=H(x,y)$. Both the field and the geometry of the problem exhibit no variations in the $z$ direction (i.e. $\partial/\partial z=0)$. On the plane within the specified region, there exists a closed loop (contour) $g(x,y)=0$ that encloses and defines a planar area $A$ that is penetrated by the field $H$. It is known from the physics of the problem that varying the shape of the contour $g$ can result in extremising the functional

$$ J(y):=\iint_{A} H(x,y)dA $$

and I need to find the optimal shape $g$ of the contour, for a given $H$ function that is nontrivial ($\neq 0)$ and captured area $A$ that is nonzero. In writing $J$ here I assumed that $x$ is the independent variable and $y$ is dependent on it, to draw the contour shape.

Anticipating that the closed contour function will be most likely expressible in parameteric form $(x(t),y(t))$, and since classical variational formulations I am familiar with usually deal with paths rather than areas, I tried to write the functional in terms of the contour (instead of the area) as follows, using Green's theorem:

$$ J=\iint_{A} H(x,y)dA=\iint_{A} \left(\frac{\partial F_{y}}{\partial x} -\frac{\partial F_{x}}{\partial y}\right)dA=\oint_{g}(F_{x}dx+F_{y}dy) =\int_{t=0}^{2\pi}(F_{x}\dot{x}+F_{y}\dot{y})dt $$

where $\boldsymbol{F}=F_{x}\hat{i}+F_{y}\hat{j}$ is some vector field whose curl may be defined to give $H$ (assuming we can find such field), dotted symbols like $\dot{x}$ denoting derivate in parameter $t\in[0,2\pi]$, and $\oint_{g}$ denoting integral around closed contour $g$. So, we can think of the Lagrangian of this problem as $L(x,y,\dot{x},\dot{y}):=F_{x}\dot{x}+F_{y}\dot{y}$.

The problem now is, if I don't impose any further contraints, the two Euler-Lagrange equation here (in $t$ now as independent variable and both $x$ and $y$ as dependents) give the same result (instead of two independent answers), which says that $H=0$. I plugged in different test fields $H$, and this is always the answer.

If I try to improve the formulation by imposing a constraint that $\iint_{A}dA=A_{0}$ to make the area nonzero constant, thus:

$$\frac{1}{2}\int_{0}^{2\pi}(x\dot{y}-y\dot{x})dt=A_{0} \Rightarrow \int_{0}^{2\pi}\left[ \frac{x\dot{y}}{2}-\frac{y\dot{x}}{2}-\frac{A_{0}}{2\pi} \right]dt=0,$$

giving a new (constrained) Lagrangian as $L(x,y,\dot{x},\dot{y},\lambda):=F_{x}\dot{x}+F_{y}\dot{y}+\frac{\lambda}{2}(x\dot{y}-y\dot{x}-\frac{A_{0}}{2\pi}),$

then, again, the two Euler-Lagrange equations in $x$ and $y$ give the same answer, basically that $H=\lambda$, where $\lambda$ is Lagrange's multiplier for this contraint.

What is wrong with my formulation, and how do I make it well posed for this problem so I can proceed?

vector-valued function? (Or is $H$ actually real-valued despite you call it a "field"?). From what you wrote in comments, it seems like you actually have a scalar function $H$ and just want a domain over which the integral is $0$ but then you start talking about "eigenfunctions that do not change" and that makes me totally confused... $\endgroup$ – fedja Sep 18 '17 at 8:12