Let me summarize comments/kill time during office hours.
- abx points out that $Q$ should be integer valued, otherwise it's not an interesting notion.
- (Assuming integrality) the answer is no because the categories of Hodge structures of type $\{(1,0), (0,1)\}$ and complex tori are equivalent. The polarizable ones correspond to abelian varieties, which is a proper subset.