Let us restate the problem for ease of reference. The equations are
$\dot{x}(t)=y(t)$
$\dot{y}(t)= - 4 x(t) + y(t)^2$
Physically they describe an anharmonic oscillator with spatial coordinate $x(t)$ and velocity $y(t)$ in the potential
$U = 2 x^2 - \frac{1}{3} x^3$
and the energy
$E = \frac{1}{2} y^2 + U(x)$
The problem is then: given two solutions
$s1(t) = (x1(t), y1(t))$
$s2(t) = (x2(t), y2(t))$
corresponding to the initial conditions
$s1(0) = (x1(t=0) = x10, y1(t=0) = y10)$ $s2(0) = (x2(t=0) = x20, y2(t=0) = y20)$
Assuming that $y10$ and $y20$ do not vanish show that the quantity
$d(t) = y1(t) + y2(t)$
becomes $\neq 0$ for times $t>0$ even if it is $= 0$ for $t = 0$, i.e.
$y10 + y20 = 0$.
We proceed by expanding the solution $y(t)$ into a power series in t.
$y(t) = y(0) + t \dot{y}(0) + \frac{1}{2} \ddot{y}(0)$ + \frac{1}{3!} \dddot{y}(0)$
For small $t$ we have up to first order in $t$
$d(t) = y10 + t \dot{y1} (0) + y20 + t \dot {y2} (0)$
$ = (y10 + y20) + t ( - 4 (x10+ x20) + (x10^2 + x20^2) ) $
Since we have $y10 + y20 = 0$, $d(t) = 0$ requires
$0 = - 4 (x10+ x20) + (x10^2 + x20^2) $
On the other hand, for $d(t) = 0$ to hold for arbitrary long times the period of motion for $s1$ and $s2$ must be the same. This in turn requires the energy $E$ to be the same, which means
$0 = (6 x10^2- x10^3)-(6 x20^2- x20^3) $
These two equations for $x10$ and $x20$ have the solutions
$sol1 = (x10 = x20 = 0)$
$sol2 = (x10 = x20 = 4)$
The orbits obey the equation
$\frac{d y}{d x}=\frac{y^2-4 x}{y}$
Multiplying by $y$ this can be written as
$\frac{1}{2}\frac{d y^2}{\text{dx}}=y^2-4 x$
and is easily integrated to give
$y^2=c \exp (2 x)+4 x+2$
where $c$ is a constant of integration.
If the initial state is $( x(0) = x0$, $y(0) = y0 )$
then
$c=\left(-4 x0+y0^2-2\right) \exp (- 2 \text{x0})$
and the orbit corresponding to this Initial state is
$y^2=\left(-4 \text{x0}+\text{y0}^2-2\right) \exp (2 (x-\text{x0}))+4 x+2$
Now the sign of the factor in front of the exponential
$s=\left(-4 \text{x0}+\text{y0}^2-2\right)$
defines distinguishes between finite ($s<0$) and unbounded ($s>0$) orbits. We therefore call the orbit $s=0$ the separatrix.
Second integration, time dependence, frequency of oscillation
Because of
$dx/dt = y[x] = Sqrt[2 + 4 x + c Exp[2 x]]$
variables can be separated an we get the time $t$ as a function of $x$:
$t=\int_{\text{x0}}^x \frac{1}{\sqrt{c \exp (2 z)+4 z+2}} \, dz+\text{t0}$