First let me note that there is an unfortunate clash in terminology: the arithmetic genus of a smooth complex projective variety $X$ of dimension $n$ can mean either
a) The number $\chi (X, \mathcal O_X)$: the Hirzebruch arithmetic number in which you are interested .
b) The number $p_a(X)=(-1)^n(\chi (X, \mathcal O_X)-1)$, the Severi arithmetic genus, which has historical precedence but of course was defined non-cohomologically.
For example, for projective space we have $\chi (\mathbb P^n, \mathcal O_{\mathbb P^n})=1$ but $p_a (\mathbb P^n)=0$.
Hirzebruch introduced his definition mainly because it has the powerful multiplicativity property $$\chi (X\times Y,\mathcal O_{X\times Y})= \chi (X,\mathcal O_{X})\cdot \chi ( Y,\mathcal O_{ Y})$$ which certainly is a step toward the geometric interpretation you are seeking.
Another step in the right direction is that for a finite covering $X\to Y$ of degree $d$ we have the pleasant relation $\chi (X,\mathcal O_{X})=d\cdot \chi ( Y,\mathcal O_{ Y})$.
But the most important geometric property is that $\chi (X,\mathcal O_{X})$ is a birational invariant, because each number $dim_\mathbb C H^i(X,\mathcal O_{X})$ is already a birational invariant.
Arithmetic genus is reasonably easy to compute: for a hypersurface $H\subset \mathbb P^n$ of degree $d$ you have $p_a(H)=\binom {d-1}{n}$, which for $n=2$ gives the well-known elementary formula $p_a(C)=\frac {(d-1)(d-2)}{2}$ for the plane curve $C$.
[This formula (and others) can be found in Hartshorne, Chapter I, Exercise 7.2, page 54]
For a surface you have Max Noether's formula $\chi (S, \mathcal O_S)=\frac {c_1^2(S)+c_2(S)}{12}$, where $c_2(S)$ (=second Chern class of $S$) is also the purely topological Euler-Poincaré characteristic of $S$, equal to the alternating sum of the Betti numbers of the underlying toplogical space.$S_{top}$.
Finally, Fulton has given an axiomatic characterization of the arithmetic genus in algebraic geometry over an arbitrary algebraically closed field here.
In a sense it may be considered an explanation of the geometric significance of the arithmetic genus: if you want it to satisfy certain geometric properties, the definition is forced upon you.
Edit (added by Charles with Georges's permission): Fulton's axiomatic characterization may be described as follows: There is a unique assignment of a number $\mathcal{A}(X)$ to every [smooth irreducible projective variety over a fixed algebraically closed field] (hereafter simply "variety"), such that the following three axioms are satisfied:
$\mathcal{A}$ respects isomorphism classes.
If $X$ is a point, then $\mathcal{A}(X) = 1$.
Let $X$, $Y$, and $Z$ be (smooth) varieties of the same dimension. Suppose that $X$, $Y$, and $Z$ can be embedded as codimension-one subvarieties of a common (smooth) variety $W$, such that
- $X$ and $Y+Z$ are linearly equivalent as divisors in $W$, and
- $Y$ and $Z$ intersect transversely in a disjoint union of (smooth) varieties $V_1, \dotsc, V_{\ell}$.
Then $$\mathcal{A}(X) = \mathcal{A}(Y) + \mathcal{A}(Z) - \sum_i \mathcal{A}(V_i).$$
This assignment takes $X$ to its "Hirzebruch arithmetic number" $\mathcal{A}(X) = \chi(X, \mathcal{O}_X)$.