Let us restate the problem for ease of reference. The equations are
$\dot{x}(t)=y(t)$
$\dot{y}(t)= - 4 x(t) + y(t)^2$
Physically they describe an anharmonic oscillator with spatial coordinate $x(t)$ and velocity $y(t)$ in the potential
$U = 2 x^2 - \frac{1}{3} x^3$
The problem is then: given two solutions $(x1(t), y1(t))$, $(x2(t), y2(t))$ corresponding to the initial conditions
$s1 = (x1(t=0) = x10, y1(t=0) = y10)$ $s2 = (x2(t=0) = x20, y2(t=0) = y20)$
Assuming that $y10$ and $y20$ do not vanish and obey the condition
$y10 + y20 = 0$
show that the quantity
$d(t) = y1(t) + y2(t)$
becomes "!=0" for times $t>0$.
For small t we have up to first order in t
d(t) = y10 + t \dot y1 (0) + y20 + t \dot y2 (0) = (y10 + y20) + t ( - 4 (x10+ x20) + (x10^2 + x20^2) )
Since we have y10 + y20 = 0, d(t) = 0 requires
0 = - 4 (x10+ x20) + (x10^2 + x20^2)
On the other hand, for d(t) = 0 to hold for arbitrary long times the period of motion for s1 and s2 must be the same. This in turn requires the energy E to be the same, which means
0 = (6 x10^2- x10^3)-(6 x20^2- x20^3 )
The orbits obey the equation
$\frac{d y}{d x}=\frac{y^2-4 x}{y}$
Multiplying by $y$ this can be written as
$\frac{1}{2}\frac{d y^2}{\text{dx}}=y^2-4 x$
and is easily integrated to give
$y^2=c \exp (2 x)+4 x+2$
where $c$ is a constant of integration.
If the initial state is $( x(0) = x0$, $y(0) = y0 )$
then
$c=\left(-4 x0+y0^2-2\right) \exp (- 2 \text{x0})$
and the orbit corresponding to this Initial state is
$y^2=\left(-4 \text{x0}+\text{y0}^2-2\right) \exp (2 (x-\text{x0}))+4 x+2$
Now the sign of the factor in front of the exponential
$s=\left(-4 \text{x0}+\text{y0}^2-2\right)$
defines distinguishes between finite ($s<0$) and unbounded ($s>0$) orbits. We therefore call the orbit $s=0$ the separatrix.
Second integration, time dependence, frequency of oscillation
Because of
$dx/dt = y[x] = Sqrt[2 + 4 x + c Exp[2 x]]$
variables can be separated an we get the time $t$ as a function of $x$:
$t=\int_{\text{x0}}^x \frac{1}{\sqrt{c \exp (2 z)+4 z+2}} \, dz+\text{t0}$