The short answer is no. The best you can say is the following: there is a functor rat from mixed Hodge modules to perverse sheaves, and for smooth $X$, $\mathrm{rat}(\mathscr M)$ is a local system iff $\mathscr M$ is smooth. But a smooth mixed Hodge module contains more information than its underlying local system (think about the case when $X$ is a point!). The precise extra data you need on a local system to define a mixed Hodge module is that of an admissible variation of mixed Hodge structure, see Steenbrink and Zucker: Variation of mixed Hodge structure I.
Edit: On second thought, maybe I want to change my answer in a different direction. The category of smooth mixed Hodge modules on $X$ is neutral tannakian, with the basepoint you wisely fixed providing the fiber functor. Then such mixed Hodge modules are in fact the same as representations of some horribly complicated pro-algebraic group. Maybe this is not the kind of answer you were after, but there it is.