Consider the energy functional $e(\cdot)$ \begin{align*} e(f,Q)&=\int_a^b \bigg\{f^4\bigg[1+\\frac{d}{dr}Q\^2+f^2\dot f^2\bigg]\bigg\} \,dr, \end{align*} over the space of \begin{equation*} {\mathcal E}:=\left\{ (Q, f) : \begin{array}{l} Q \in W^{1,4}([a,b],{\bf SO}(n)),\\ Q(a)=Q(b)=I,\\ f \in W^{1,4}[a,b],\\ \dot f>0 \mbox{ ${\cal L}^1$a.e. on $(a,b)$},\\ f(a)=a, f(b)=b. \end{array} \right\} \end{equation*} It is easy to show that, energy functional $e(\cdot)$ is coercive when $Q\in W^{1,2}$ and $f\in W^{1,2}$, in another words, there exists $d=d(n, a, b)>0$ such that \begin{equation*} e(f, Q) \ge d ( \Q\^2_{W^{1,2}} + \f\^2_{W^{1,2}}). \end{equation*} Now my question is: The energy functional $e(\cdot)$ is coercive on the space $\mathcal E$ or not? In another words could we find $\gamma=\gamma(n, a, b)>0$ such that \begin{equation*} e(f, Q) \ge \gamma ( \Q\^4_{W^{1,4}} + \f\^4_{W^{1,4}}). \end{equation*} for all $(f, Q)\in \mathcal E$.
No. The reason is that
$$e(f,Q)=\int_a^b \bigg\{f^4\bigg[1+\\frac{d}{dr}Q\^2+f^2\dot f^2\bigg]\bigg\} \,dr,
$$
with $b\geq f\geq a$ (since $f$ is continuous and monotonous) gives
$$
e(f,Q) \leq \left(1+\frac{b^4}{2}\right)\left(\f\^4_{W^{1,4}} + \Q\^2_{W^{1,2}}\right)
$$
So your inequality would imply that there exists $C_1,C_2>0$ such that for all $f,Q\in\mathcal{E}$
$$
C_1\Q\^2_{W^{1,2}} + C_2\f\^4_{W^{1,4}} \geq \Q\^4_{W^{1,4}}.
$$
Now, fix $f=a+b\frac{ta}{ba}$, and it becomes for all $Q\in\mathcal{E}$
$$
C_3(\Q\^2_{W^{1,2}} + 1) \geq \Q\^4_{W^{1,4}},
$$
if you cookup a sequence $Q_n$ with $\Q_n\_{W^{1,2}}=1$ and $\Q_n\_{W^{1,4}}>n$ you have a contradiction.
Note that $W^{1,4}$ is not natural, since you can also obtain a bound from above of the form $$ e(f,Q) \leq C(a,b)\left(\f\^2_{W^{1,2}} + \Q\^2_{W^{1,2}}\right). $$  edited after the comment to the first part of the answer
If you are looking for a minimizer, it is immediate to note that $$ e(f,Q)\geq e(f,I) $$ for any $f$ since it is a sum of squares, and $(f,I)\in \mathcal{E}$. Therefore you just want to solve $$ \min_{f\in \mathcal{X}} \int_a^b \left( f^4 +f^6 \left(\frac{df}{dt}\right)^2\right) \,dt, $$ with $\mathcal{X}=\{f \in W^{1,2}(a,b)~:~f(a)=a,f(b)=b, f^\prime>0~a.e.\}$, a convex set. A calculus of variation problem in one dimension with a monotone integrand> textbook question. The infimum will be in $\bar{\mathcal{X}}$ in general, and $f$ satisfies either the EulerLagrange equation, or is constant. To find a simple answer, let us assume that $0<a<b$. Then $f>0$ in $\mathcal{X}$, and if you set $g=f^4$, you find that your problem is also $$ \min_{g\in \mathcal{Y}} \int_a^b \left( g + \frac{1}{16} \left(\frac{dg}{dt}\right)^2\right) \,dt, $$ with $\mathcal{Y}=\{f \in W^{1,2}(a,b)~:~g(a)=a^4,g(b)=b^4, g^\prime>0~a.e.\}$, a convex, open set. Now, the EulerLagrange equation is simply $$1+\frac{1}{8}g^{\prime\prime}=0.$$ You can solve it by hand, and playing with it, you find for example that when $$a> \frac{\sqrt{6}}{8}\frac{\left(3554+2(33)^{3/2}\right)^{1/3}}{\sqrt{7}\left(3554+2(33)^{3/2}\right)^{2/3}+(33)^{3/2}+116\left(3554+2(33)^{3/2}\right)^{1/3}+1777)},$$ the solution with $g(a)=a^4$ and $g(b)=b^4$ is strictly increasing and therefore is in $\mathcal{Y}$, whereas otherwise the positivity of the gradient constraint comes into play for suitable $b$s.

$\begingroup$ Indeed what I'm trying to do is to show that the energy functional $e(f, Q)$ admit local minimizers. I would be grateful if you let me know yours ideas. $\endgroup$– MSSHDMar 10 '14 at 14:29


$\begingroup$ What about $W^{1,4}$, Actually I would like to prove we have the minimizer in $W^{1,4}$ not in $W^{1,2}$. $\endgroup$– MSSHDMar 11 '14 at 5:36
that is a good idea !
i guess that your conjecture may has a counterexample !
see : http://www.stanford.edu/class/math220b/handouts/calcvar.pdf
that is because :
$e(f, Q) \ge d ( \Q\^2_{W^{1,2}} + \f\^2_{W^{1,2}})$$\Longrightarrow$$f^4(1+\vert{\nabla{Q}}\vert^2+f^2\vert{\nabla{f}}^2\vert)$$\ge$$d(Q^2+\vert{\nabla{Q}}\vert^2+f^2+\vert{\nabla{f}}^2\vert)$
we select $d=f^4$ to get :
$1+\vert{\nabla{Q}}\vert^2+f^2\vert{\nabla{f}}^2\vert$$\ge$$Q^2+\vert{\nabla{Q}}\vert^2+f^2+\vert{\nabla{f}}^2\vert$$\ge$$4Q\nabla{Q}+4f\nabla{f}$$\Longrightarrow$$f\nabla{f}(4f\nabla{f})+Q\nabla{Q}(4\frac{\nabla{Q}}{Q})\le1$
on the other hand ,
$f^4(1+\vert{\nabla{Q}}\vert^2+f^2\vert{\nabla{f}}^2\vert)$$\gamma(Q^4+\nabla{Q}^4+f^4+\nabla{f}^4)\ge0$$\Longrightarrow$$1+\vert{\nabla{Q}}\vert^2+f^2\vert{\nabla{f}}^2\vert$$\ge$$4Q^2\nabla{Q}^2+4f^2\nabla{f}^2$$\Longrightarrow$$3f^2\nabla{f}^2+Q^2\nabla{Q}^2(4\frac{1}{Q^2})\le1$
by comparing the two inequality above, we have :
$3f^2\nabla{f}^2+Q^2\nabla{Q}^2(4\frac{1}{Q^2})\le$$f\nabla{f}(4f\nabla{f})+Q\nabla{Q}(4\frac{\nabla{Q}}{Q})$
$\Longrightarrow$$4f\nabla{f}(f\nabla{f}1)+4Q\nabla{Q}(Q\nabla{Q}1)\le0$
now, it is clear that your problem is related with the convexity and the concavity of the function you defined on $[a,b]$ !
The EulerLagrange Equations of the $e(f, Q)$ over the admissible space $\mathcal E$ arise as the following system \begin{align*} \left \{ \begin {array}{ll} (i)\ \ \frac{d}{dr} \bigg[ f^4 Q^t \frac{d}{dr} Q \bigg] ={\bf 0},\\ \\ (ii)\ \frac{d}{dr} \bigg[ f^2 \dot f \bigg] = 2f^3 +3f^5 \dot f^2 + 2f^3 dQ^2, \end{array} \right. \end{align*} Now as the functional $e(\cdot, \cdot)$ is coercive on $\mathcal E$ when we choose the $f, Q$ in $W^{1,2}$ instead of $W^{1,4}$ and an application of direct methods shows the system has a solution. Well, could we claim this result in $\mathcal E$ by $f, Q$ in $W^{1,4}$?

$\begingroup$ you just need to show that the minimiser you obtain is in $W^{1,4}$. Note that $\mathcal{E}$ is not closed, so it is an infimum and not a minimum in general, cf. detailed answer above. $\endgroup$– usernameMar 11 '14 at 10:01
It is evident that your functional $e(f, Q)$ has minimizer in the space $W^{1,2}$ since you have coercivity in this space and an application of direct method of calculus of variation givee you the required result in $W^{1,2}$. So the system of EulerLagrange equation you wrote in below, has solution in $W^{1,2}$. Now from the first equation (equation (i) you can compute $\dot{Q}$ and $Q$, in fact we have $$ \dot{Q} = \frac{1}{f^4}Q C$$ in which $C$ is a (skew symmetric) constant matrix. therefore $\dot{Q}^4 < \infty$ and hence $Q \in W^{1,4}$ (in fact by this argument you can show that the solution $Q$ is smooth). Similarly form the equation (ii) you can compute $f''$ in terms of $f$ and $f'$ and $\dot{Q}$ which shows that $f''$ exists (note that $f$ is bounded on [a,b]) so $f'$ is continuous and consequently for the solution $f\in W^{1,2}$ of the above system we have $f\in W^{1,4}$. In fact by the above argument one can show that the solution $f$ of the above system is also smooth (similar to the solution $Q$). Therefore the solution (Q,f) which first was proved to be in the space $W^{1,2}$ is in the space $W^{1,4}$ and moreover by applying the system of equation (i) and (ii) similarly we could prove that this solution $(Q, f)$is a $C^{\infty}$ solution.