Given an algebraic stack $\mathcal{X}$ there is a canonically associated topological space $|\mathcal{X}|$ of points of $\mathcal{X}$. A point is an equivalence class of morphisms $\text{Spec}(k) \to \mathcal{X}$, where $k$ is a field. Two morphisms $\text{Spec}(k) \to \mathcal{X}$ and $\text{Spec}(k') \to \mathcal{X}$ are equivalent if $\text{Spec}(k) \times_\mathcal{X} \text{Spec}(k')$ is nonempty (as an algebraic stack, i.e., there is a point). Given a morphism $f : \mathcal{X} \to \mathcal{Y}$ of algebraic stacks, the obvious map $|\mathcal{X}| \to |\mathcal{Y}|$ is continuous and if $f$ is smooth then it is open. This together with the requirement that $|\mathcal{X}|$ should be the usual topological space when $\mathcal{X}$ is a scheme, uniquely determines the topology.

Definition: An open substack $\mathcal{U} \subset \mathcal{X}$ is *dense* if $|\mathcal{U}| \subset |\mathcal{X}|$ is dense.

IMHO opinion this is the correct definition. More generally, topological properties should be, as much as possible, defined in terms of $|\mathcal{X}|$. For example a morphism $f : \mathcal{X} \to \mathcal{Y}$ of algebraic stacks is *universally closed* if for all morphisms $\mathcal{Z} \to \mathcal{Y}$ the map $|\mathcal{Z} \times_\mathcal{Y} \mathcal{X}| \to |\mathcal{Z}|$ is closed.