For a curve, that's all of course, since the polynomial is linear.
Now let's say $X$ is a smooth surface with an ample divisor $H$ and canonical
divisor $K$, we have the Hilbert polynomial
$$\chi(\mathcal{O}_X(nH))= \frac{1}{2}nH(nH-K) + \chi(\mathcal{O}_X)$$
by Riemann-Roch. So the linear coefficient gives you the degree of the canonical
divisor. In higher dimensions, the more general form of Riemann-Roch
$$\chi(\mathcal{O}_X(nH)) = \int_X ch(\mathcal{O}(nH))td(X)$$
tells you that you're basically getting certain Chern numbers in $X$ and $H$ as coefficients.

What is perhaps simpler, is to use  the recurrence formula
$$\chi(\mathcal{O}_{H\cap Y}(nH)) =\chi(\mathcal{O}_Y(nH))- \chi(\mathcal{O}_Y((n-1)H))$$
to see that the Hilbert polynomial determines and is determined by the arithmetic genera of $X$ and
 the complete intersections $H$, $H_1\cap H_2$ etc.