The Hodge index theorem for smooth projective varieties (or compact Kähler manifolds) says that the signature is $$\sum_{p,q} (-1)^p h^{p,q}(X)$$
For toric varieties, $h^{p,q}=0$ unless $p=q$, and equals the Betti number $h^{2p}(X)$, so the signature is $\sum_p (-1)^p h^{2p}(X)$.
The Betti number $h^{2p}$ is equal to $\sum_{i=p}^n (-1)^{i-p} (-1)^{i-p} \binom{i}{p} N_i$ where $N_i$ is the number of $i$-dimensional cells in the polytope.
So the signature is $$ \sum_p(-1)^p \sum_{i=p}^n (-1)^{i-p} (-1)^{i-p} \binom{i}{p} N_i = \sum_i (-1)^i \sum_{o=0}^i \binom{i}{p} N_i = \sum_i (-1)^i2^i N_i = \sum_i (-2)^i N_i.$$
Thus, for a two-dimensional variety, this is the number of vertices minus twice the number of edges plus four times the number of faces. For a polygon, the number of vertices is the number of edges, and the number of faces is one, so this is four minus the number of vertices, which should be your formula.
Similarly, in higher dimensions we can use the Dehn-Somerville equations for simplicial polyhedra to eliminate half of the variables, as smooth toric varieties have simplicial moment polytopes, as pointed out by David Speyer in the comments.