First let me note that there is an unfortunate clash in terminology: the arithmetic genus of a smooth complex projective variety $X$ of dimension $n$ can mean either
a) The number $\chi (X, \mathcal O_X)$: the Hirzebruch arithmetic number in which you are interested .
b) The number $p_a(X)=(-1)^n(\chi (X, \mathcal O_X)-1)$, the Severi arithmetic genus, which has historical precedence but of course was defined non-cohomolgically.
For example, for projective space we have $\chi (\mathbb P^n, \mathcal O_{\mathbb P^n})=1$ but $p_a (\mathbb P^n)=0$.
Hirzebruch introduced his definition mainly because it has the powerful multiplicativity property $$\chi (X\times Y,\mathcal O_{X\times Y})= \chi (X,\mathcal O_{X})\cdot \chi ( Y,\mathcal O_{ Y})$$ which certainly is a step toward the geometric interpretation you are seeking.
Another step in the right direction is that for a finite covering $X\to Y$ of degree $d$ we have the pleasant relation $\chi (X,\mathcal O_{X})=d\cdot \chi ( Y,\mathcal O_{ Y})$.
But the most important geometric property is that $\chi (X,\mathcal O_{X})$ is a birational invariant, because each number $dim_\mathbb C H^i(X,\mathcal O_{X})$ is already a birational invariant.
Arithmetic genus is reasonably easy to compute: for a hypersurface $H\subset \mathbb P^n$ of degree $d$ you have $p_a(H)=\binom {d-1}{n}$, which for $n=2$ gives the well-known elementary formula $p_a(C)=\frac {(d-1)(d-2)}{2}$ for the plane curve $C$.
For a surface you have Max Noether's formula $\chi (S, \mathcal O_S)=\frac {c_1^2(S)+c_2(S)}{12}$, where $c_2(S)$ (=second Chern class of $S$) is also the purely topological Euler-Poincaré characteristic of $S$, equal to the alternating sum of the Betti numbers of the underlying toplogical space.$S_{top}$.
Finally, Fulton has given an axiomatic characterization of the arithmetic genus in algebraic geometry over an arbitrary algebraically closed field here.
In a sense it may be considered an explanation of the geometric significance of the arithmetic genus: if you want it to satisfy certain geometric properties, the definition is forced upon you.