For this curve $y^2=x^3+b^2x^2a^2b^2x$ where $a \neq b$ and $a,b$ are rational. I can prove that if $b^2+4a^2$ is square then torsion group of curve is $\mathbb Z2 \times \mathbb Z2$, and when $b^2+4a^2$ is not square I want prove that torsion group is $\mathbb Z 2$. So for second part I can prove there is not point of order 3,4 but really I don't know how I can prove there is not point of order 5 . How I can prove that there is not point of order 5 when $b^2+4a^2$ is not square?
First of all, your question depends on only one and not on two parameters: Upon setting $b=ta$ for a new variable $t$, and replacing $x$ and $y$ with $a^2x$ and $a^3y$, your elliptic curve assumes the simpler form $y^2=x^3+z(x^2x)$ with $z=t^2$. The $j$invariant of this elliptic curve is $\frac{256(z+3)^3}{z+4}$. If the curve has a point of order $5$, then it has an isogeny of degree $5$. By an old result (for instance in Fricke's 1922 book on elliptic functions), the $j$invariant of an elliptic curve with an isogeny of degree $5$ has the form $\frac{(s^2+10s+5)^3}{s}$. So one first has to solve \begin{equation} \frac{256(z+3)^3}{z+4}=\frac{(s^2+10s+5)^3}{s}, \end{equation} and then to check when $z=t^2$ for a rational $t$. This equation simplifies upon setting $(z+3)w=s^2+10s+5$ for a new variable $w$, and eliminating $z$. The resulting curve in $s$ and $w$ has genus $0$, and Maple's function parametrization (do with(algcurves) before that) gives a rational parametrization, and hence $z=R(u)$ for a rational function $R$ and a variable $u$. The equation $t^2=R(u)$ is easily transformed to an elliptic curve. One can compute (with Sage) that this elliptic curve has torsion order $2$ and rank $0$. So there are at most finitely many candidates for $t$ to check. It seems (but I didn't check very carefully) that no $t$ gives an isogeny of degree $5$. 

