For a curve, that's all of course, since the polynomial is linear. Now let's say $X$ is a smooth surface with an ample divisor $H$ and canonical divisor $K$, we have the Hilbert polynomial $$\chi(\mathcal{O}_X(nH))= \frac{1}{2}nH(nH-K) + \chi(\mathcal{O}_X)$$ by Riemann-Roch. So the linear coefficient gives you the degree of the canonical divisor. In higher dimensions, the more general form of Riemann-Roch $$\chi(\mathcal{O}_X(nH)) = \int_X ch(\mathcal{O}(nH))td(X)$$ tells you that you're basically getting certain Chern numbers in $X$ and $H$ as coefficients.
What is perhaps simpler, is to use the recurrence formula $$\chi(\mathcal{O}_{H\cap Y}(nH)) =\chi(\mathcal{O}_Y(nH))- \chi(\mathcal{O}_Y((n-1)H))$$ to see that the Hilbert polynomial determines and is determined by the arithmetic genera of $X$ and the complete intersections $H$, $H_1\cap H_2$ etc.