At least for *effective* divisors, the answer is strongly related to *Zariski decomposition*.

If $D$ is an effective divisor on a smooth surface $X$, Zariski proved in **[Z62]** that there exists a unique decomposition $D=P + N$, where

- $P$ is a nef $\mathbb{Q}$-divisor
- $N$ is an effective $\mathbb{Q}$-divisor
- $PC=0$ for every curve $C$ appearing in $\operatorname{Supp}(N)$
- if $N \neq 0$ and $\operatorname{Supp}(N)=C_1 \cup \ldots \cup C_k$, then the intersection form $I(C_1, \ldots, C_k)$ is negative defined.

Furthermore, one also shows that, for all positive integers $m$, the natural map $$H^0(X, \left \lfloor{mP}\right \rfloor) \to H^0(X, \, mD) $$ is an isomorphism. This means that "$P$ carries all sections of $D$" and so $$\operatorname{vol}_X(D)=\operatorname{vol}_X(P)= P^2,$$ where the last equality is a consequence of the asymptotic form of Riemann-Roch theorem, because $P$ is nef.

Thus, $\operatorname{vol}_X(D)=D^2$ is equivalent to $D^2=P^2$; since $PN=0$, this happens if and only if $N^2=0$. But the intersection form must be negative defined on $N$ when $N \neq 0$, hence the only possibility is $N=0$, namely, $D=P$.

Summing up:

given an effective divisor $D$ on a smooth surface $X$, we have
$\operatorname{vol}_X(D)=D^2$ if and only if $D$ is nef.

More generally, the difference $$D^2-\operatorname{vol}_X(D)$$ equals the self-intersection $N^2$ of the negative part of the Zariski decomposition of $D$, so it can be seen as a measure of how much "$D$ fails to be nef".

**References.**

**[Z62]** O. Zariski The theorem of Riemann-Roch for high multiples of an effective divisor on an algebraic surface, *Ann. Math.* (2) **76**, 560-615 (1962). ZBL0124.37001.