I posted this question on Math.SE, but I could not get any help.
The eigenvalue $\lambda(t)$ is characterised as the minimum of the Rayleigh quotient (where $t$ is a scalar variable) $$R(u,\Omega_t)= \frac{\int_{\Omega_t} |\nabla u|^2 dy }{\int_{\Omega_t} u^2 dy}$$ i.e. $\lambda(\Omega_t) = \mathbf{min} \{R(v, \Omega_t) : v \in H^{1,2}(\Omega_t)\}$ The minimiser $u^*$ of the Rayleigh quotient satisfies : $$\left. \frac{d}{ds} R(u^* + s\varphi) \right \vert_{s=0} =0 $$ Solving it, the conditions I get are
$$\Delta u^* + \lambda (\Omega_t) u^* =0 \quad \textrm{in} \quad \Omega_t $$
$$\partial_{\nu} u^* = 0 \quad \textrm{in} \quad \partial \Omega_t$$
Similarly, I want to find the conditions that are fulfilled by the minimizer $u^*$ for the Rayleigh quotient defined as follows.
$$R(u,\Omega_t)= \frac{\int_{\Omega_t} |\nabla u|^2 dy + \alpha \oint_{\partial \Omega_t} u^2 dS}{\left(\int_{\Omega_t} u^q dy \right)^{2/q}}$$
My attempt: we have $$\lambda(x, \Omega_t) = \frac{\int_{\Omega_t} {\mid \nabla u(x) \mid }^2 \, dx + \alpha \oint_{\partial \Omega_t} u^2\, ds} { \left( \int_{\Omega_t} u^q \, dx\right )^{2/q}} $$
The corresponding eigenvalue equation is given by the following computation.
Let $t \in \mathbb{R} , \varphi \in H^{1,2}\Omega$. If $\lambda(x, \Omega_t) $ is the eigenvalue then the following holds
$\frac{d}{ds} R(u_t + s \varphi , \Omega_t) \rvert_{s=0}$
$$\frac{d}{ds} \left(\frac {\int_{\Omega_t} {\mid \nabla (u+s\varphi) \mid}^2\,dx}{\left(\int_{\Omega_t} {\mid u +s\varphi \mid}^q \, dx \right)^{2/q}} \right) = \frac{d}{ds} \left (\frac{\int_{\Omega_t} {\mid \nabla u \mid}^2 + 2 s \nabla u \nabla \varphi + s^2 {\mid \nabla \varphi \mid}^2 \, dx} {\left( \int_{\Omega_t} (u+s \varphi )^q \, dx\right )^{2/q}} \right)_{s=0} $$
$$=\frac{2 \int_{\Omega_t} \nabla u \nabla \varphi\,dx }{ \left( \int_{\Omega_t} u^q \, dx\right )^{2/q}} - 2 \left( \int_{\Omega_t} {\mid u \mid }^q\right)^{\frac{-2}{q} -1 } \int_{\Omega_t} u^{q-1} \varphi \, dx \int_{\Omega_t} {\mid \nabla u \mid}^2 dx ... (1*) $$
Evaluating next term, $$\frac{d}{ds} \left( \frac{\alpha \oint_{\partial \Omega_t} (u+s \varphi)^2 \, ds}{ \left(\int_{\Omega_t} (u+s\varphi)^q \, dx \right )^{2/q} }\right )_{s= 0} $$
$$= \frac{2 \alpha \int_{\partial \Omega_t} u \varphi \, ds - 2 \alpha \oint_{\partial \Omega_t} u^2 \, ds \left(\int_{\Omega_t} u^q \, dx \right )^{-1} \int_{\Omega_t} u^{q-1} \varphi \, dx }{ \left( \int_{\Omega_t} u^q \, dx \right )^{2/q} } ... (2*) $$
Adding $(1)$ and $(2)$ gives :
$$\frac{2 \int_{\Omega_t} \nabla u \nabla \varphi dx - \int_{\Omega_t} {\mid \nabla u \mid}^2 dx \left( \int_{\Omega_t} u^q \, dx \right )^{-1} \int_{\Omega_t} u^{q-1} \varphi \, dx + \alpha \int_{\partial \Omega_t} u \varphi \, ds - \alpha \int_{\partial \Omega_t } u^2 \, ds \left( \int_{\Omega_t} u^q dx \right)^{-1} \int_{\Omega_t} u^{q-1} \varphi \, dx }{ \left(\int_{\Omega_t} {\mid u\mid }^q\, dx \right)^{2/q}}$$
Using integration by parts yields,
$$\frac{-2 \int_{\Omega_t} \Delta u \varphi dx + 2 \int_{\partial \Omega_t} \partial_{\nu} u \varphi \, ds - 2 \left ( \int_{\Omega_t} u^q \right)^{-1} \int_{\Omega_t} u^{q-1} \varphi \, dx \int_{\Omega_t} {\mid \nabla u \mid}^2 \, dx +2 \alpha \int_{\partial \Omega_t} u \varphi \, ds }{\left(\int_{\Omega_t} {\mid u\mid }^q \, dx \right)^{2/q}}$$
$$ -\frac{2 \alpha \int_{\partial \Omega_t} u^2 \, ds \left(\int_{\Omega_t} {\mid u \mid}^q \right)^{-1} \int_{\Omega_t} u^{q-1} \varphi \, dx }{\left(\int_{\Omega_t} {\mid u\mid }^q \, dx \right)^{2/q}}$$
and thus, we get the equation
$$ \Delta u + \lambda \mu (u) u^{q-1} = 0 \mathtt \;on\; \Omega_t $$
together with the boundary condition
$$ \partial_{\nu} u + \alpha u = 0 \mathtt \; on \; \partial \Omega_t $$
where
$$\mu(u) = \left ( \int_{\Omega_t} u^{q-1} \, dx \right )^{-1} $$
Can someone check if the resulted equations are the right necessary conditions ? I am looking also for sufficient conditions on $q \in \mathbb R$ which guarantee the existence of a minimizer $u^*$.