Consider a real polynomial $H$ of degree $n+1$ in the plane. A closed, connected component of a level curve $H=t$ is denoted by $\gamma(t)$ and called an oval of $H$. Let $\omega$ be a real 1form with polynomial coefficients of degree at most $n$. We say that an oval $\gamma(t)$ generates a limit cycle of the perturbed system $dH + \epsilon \omega$ if there exists a family of closed curves $l(\epsilon)$ defined for small $\epsilon$ such that $l(\epsilon)$ is a limit cycle of $dH + \epsilon \omega$ when $\epsilon \neq 0$ and $l(0)=\gamma(t)$. Consider the abelian integral \begin{equation*} I(t) = \int_{\gamma(t)} \omega \end{equation*} The Pontryagin criterion says that if $\gamma(t)$ generates a limit cycle in the above sense then $I(t)=0$. Conversely, if $I(t)=0$ and $I'(t) \neq 0$ then the oval $\gamma(t)$ generates a limit cycle of the perturbed system. What can be said about the STABILITY of limit cycles generated in this way? Can we couple this result to Floquet theory or a related concept in a convenient way?

$\begingroup$ Higher Melnikov function described in Perko books gives simillar results for the case of non simple roots of the corresponding abelian integral. I am also interested in these topics please see my question here : In this question the unperturbed system is not Hamiltonian: mathoverflow.net/questions/157966/… $\endgroup$ – Ali Taghavi Jun 24 '16 at 6:01
Well, it looks like this is simply a matter of a Poincare map. If you look at a proof of Pontryagin's criterion, it turns out that the integral $I(t)$ is the first derivative of the Poincare map with respect to the parameter $\varepsilon$, and then evaluated at $\varepsilon=0$. Indeed, assume you have a crosssection $\Sigma$ of the orbits of $\ker dH$ and parametrize this crosssection by $H_{\Sigma} : \Sigma \to \mathbb{R}$. For small enough $\varepsilon$ the transverse arc $\Sigma$ is also a crosssection of the orbits of $\ker( dH + \varepsilon \omega)$. Fix a point $t$ on $\Sigma$. Then let $\gamma_{\varepsilon, t}$ be the orbit of $\ker( dH + \varepsilon \omega)$ starting from $t \in \Sigma$ and returning to $\Sigma$ again for the first time at the point $t_1 \in \Sigma$. This is the Poincare firstreturn map (guaranteed to exist for small enough $\varepsilon$) and we denote it by $P_{\varepsilon}(t):=t_1$, where the map $P_{\varepsilon} : \Sigma' \to \Sigma\,\,\,$ is defined on an open subset $\Sigma' \subseteq \Sigma$. Now, since $\gamma_{\varepsilon, t}$ is a an orbit of $\ker( dH + \varepsilon \omega)$, we can write this fact alternatively as $( dH + \varepsilon \omega)_{{\gamma}_{\varepsilon, t}}(\dot{\gamma}_{\varepsilon, t}) = 0$. Therefore if we integrate the latter equality along $\gamma_{\varepsilon, t}$ we obtain $$0 = \int_{\gamma_{\varepsilon, t}} dH + \varepsilon \omega = \int_{\gamma_{\varepsilon, t}} dH + \varepsilon \int_{\gamma_{\varepsilon, t}} \omega = H_{t_1}  H_{t} + \varepsilon \int_{\gamma_{\varepsilon, t}} \omega = t_1  t + \varepsilon \int_{\gamma_{\varepsilon, t}} \omega. $$ I have slightly abused notation, identifying the endpoints of $\gamma_{\varepsilon, t}$ on the crosssection $\Sigma$ with their coordinates $t$ and $t_1$. Now, by construction $P_{\varepsilon}(t)=t_1$ so the result of the integration above yields $$0 = P_{\varepsilon}(t)  t + \varepsilon \int_{\gamma_{\varepsilon, t}} \omega$$ written also as $$P_{\varepsilon}(t)  t =  \varepsilon \int_{\gamma_{\varepsilon, t}} \omega.$$ Division by $\varepsilon$ takes us to
$$\frac{P_{\varepsilon}(t)  t}{\varepsilon} =  \int_{\gamma_{\varepsilon, t}} \omega.$$ If we let $\varepsilon \to 0$, by continuous dependence on parameters, $\gamma_{\varepsilon, t} \to \gamma(t)$, the latter being the oval orbit of $\ker dH$ through $t$. Therefore the last equality converges to $$\frac{\partial P_{\varepsilon}(t) }{\partial \varepsilon}\Big{}_{\varepsilon = 0} =  \int_{\gamma(t)} \omega =  I(t).$$ This last conclusion allows us to Taylor expand the Poincare map with respect to $\varepsilon$ near $0$: $$P_{\varepsilon}(t) = t  \varepsilon I(t) + \varepsilon^2 R(t, \varepsilon).$$ Now, a fixed point of the Poincare map, $P_{\varepsilon}(t_{\varepsilon}) = t_{\varepsilon},$ gives rise to a periodic orbit of $\ker( dH + \varepsilon \omega)$ passing through $t_{\varepsilon} \in \Sigma$. So, in search of periodic orbits of $\ker( dH + \varepsilon \omega)$, we try to solve the fixed point equation $$f_{\varepsilon}(t):=\frac{1}{ \varepsilon }(P_{\varepsilon}(t)  t) =  I(t) + \varepsilon R(t, \varepsilon) = 0.$$ The conditions for $I$ you mention are simply the conditions for the implicit function theorem applied to the latter equality, i.e. $f_{\varepsilon}(t) = 0$. Thus, if we find $t_0 \in \Sigma$ such that $I(t_0) = 0$ but $I'(t_0) \neq 0$, then we know that there is a one parameter continuous family of initial conditions $t_{\varepsilon}$ for which $f_{\varepsilon}(t_{\varepsilon}) = 0$, which in its own turn implies the existence of one parameter family of cycles $l(\varepsilon)$ of $\ker( dH + \varepsilon \omega),$ passing through the points $t_{\varepsilon}$ so that $t_{\varepsilon}_{\varepsilon = 0} = t_0$ and so $l(\varepsilon)$ continuously homotopes to $l(0) = \gamma(t_0)$. Finally, we write the Poincare map as $$P_{\varepsilon}(t) = t  \varepsilon I(t) + \varepsilon^2 R(t, \varepsilon) = t + \varepsilon \big(  I(t) + \varepsilon R(t, \varepsilon)\big) = t + \varepsilon f_{\varepsilon}(t),$$ and its first derivative with respect to $t$ is $$P'_{\varepsilon}(t) = 1  \varepsilon I'(t) + \varepsilon^2 \frac{\partial R}{\partial t}(t,\varepsilon)=1 + \varepsilon f'_{\varepsilon}(t).$$
Now, the first derivative of $f_{\varepsilon}(t)$ with respect to $t$ evaluated at $t_{\varepsilon}$ is $$ g(\varepsilon) = f'_{\varepsilon}(t_{\varepsilon}) =  I'(t_{\varepsilon}) + \varepsilon \frac{\partial R}{\partial t}(t_{\varepsilon},\varepsilon),$$ so we see that $g(\varepsilon)$ is a continuous function with respect to $\varepsilon.$ Assume for instance that $I'(t_0) > 0$. For $\varepsilon = 0$ we obtain $$g(0) = f'_0(t_0) = I'(t_0) < 0.$$ By continuity, $g(\varepsilon) = f'_{\varepsilon}(t_{\varepsilon}) < 0$ for small enough values of $\varepsilon$. Therefore $$0 < P'_{\varepsilon}(t_{\varepsilon}) =1 + \varepsilon f'_{\varepsilon}(t_{\varepsilon}) < 1,$$ whenever $\varepsilon > 0$ and it is small enough. Analogously, $$ P'_{\varepsilon}(t_{\varepsilon}) =1 + \varepsilon f'_{\varepsilon}(t_{\varepsilon}) > 1,$$ for $\varepsilon < 0$ near $0$. The former case implies that the limit cycle is stable, while the latter one implies that the cycle is unstable. The alternative assumption that $I'(t_0) < 0$ is handled in the same way.
Notice that these arguments are more general and it is irrelevant whether $H$ and $\omega$ have polynomial nature.

$\begingroup$ Thank you for a very nice answer. If I understand correctly, the punchline is: Assume $\epsilon >0$. Then the limit cycle is stable if $I'(t_0)>0$ and unstable if $I'(t_0)<0$? This seems remarkably simple, but I'm left wondering if there is any notion of semistability? Moreover, do you have a good reference to confirm your proof of Pontryagin's criterion? I am a little unsure how much rigour is contained in the one you have provided. (and by $ker(a)=0$ do you just mean $ker(a)$?) $\endgroup$ – hsp99 Jun 24 '16 at 13:44

$\begingroup$ Yes, it is exactly as you say: for small enough $\varepsilon >0$ the limit cycle is stable if $I'(t_0)>0$ and unstable if $I'(t_0)<0$. The function $I(t)$ conveniently encodes a lot of information about the qualitative behavior of the dynamical system in the relevant neighborhood: existence of periodic orbits as well as stability of the latter. Otherwise, to hope for semistability, assuming the dynamical system is at least real analytic, or even polynomial, one probably needs to have $I(t_0)=0, \, I'(t_0) = 0$ but $I''(t_0) \neq 0$. BTW, I've already fixed the $\ker$ typo. Thanks. $\endgroup$ – Futurologist Jun 28 '16 at 0:32

$\begingroup$ In terms of literature, I am not too sure where exactly one can see the proof, although I have seen $I(t)$ used in all articles related to the topic of infinitesimal 16th Hilbert problem. But maybe in the books written by some of the experts in the field? $\endgroup$ – Futurologist Jun 28 '16 at 0:37