If $N=6$ then we have a nice identity:
$$(x_1+x_2)(x_1+x_3)(x_2+x_3)=-(x_4+x_5)(x_4+x_6)(x_5+x_6).$$
(Loo-Keng Hua used such identities in his *"Additive Theory of Prime Numbers"*.) In particular if $x_5=x_6=0$ then LHS vanishes. From this observation easily follows that $N>4$.