First let me note that there is an unfortunate clash in terminology: the arithmetic genus of a smooth complex projective variety  $X$ of dimension $n$ can mean either 
 
a) The number $\chi (X, \mathcal O_X)$: the Hirzebruch arithmetic number in which you are interested .  
b) The number $p_a(X)=(-1)^n(\chi (X, \mathcal O_X)-1)$, the Severi arithmetic genus, which has historical precedence but  of course was defined non-cohomolgically.  
 For example, for projective space we have $\chi (\mathbb P^n, \mathcal O_{\mathbb P^n})=1$ but $p_a (\mathbb P^n)=0$.   

Hirzebruch introduced his definition mainly because it has the powerful multiplicativity property  $$\chi (X\times Y,\mathcal O_{X\times Y})= \chi (X,\mathcal O_{X})\cdot \chi ( Y,\mathcal O_{ Y})$$  which certainly is a step toward the geometric interpretation you are seeking.  

Another step in the right direction is that for a finite covering  $X\to Y$ of degree $d$ we have the pleasant relation $\chi (X,\mathcal O_{X})=d\cdot  \chi ( Y,\mathcal O_{ Y})$.  
But the  most important geometric property is that $\chi (X,\mathcal O_{X})$ is a birational invariant, because each number $dim_\mathbb C H^i(X,\mathcal O_{X})$ is already a birational invariant.  

Arithmetic genus is reasonably easy to compute: for a hypersurface $H\subset \mathbb P^n$ of degree $d$ you have $p_a(H)=\binom {d-1}{n}$, which for $n=2$ gives the well-known elementary formula $p_a(C)=\frac {(d-1)(d-2)}{2}$ for the plane curve $C$.   
For a surface you have Max Noether's formula $\chi (S, \mathcal O_S)=\frac {c_1^2(S)+c_2(S)}{12}$, where $c_2(S)$  (=second Chern class of $S$) is also the purely *topological* Euler-Poincaré characteristic of $S$, equal to the alternating sum of the Betti numbers of the underlying toplogical space.$S_{top}$.  

Finally, Fulton has given an axiomatic characterization of the arithmetic genus in algebraic geometry over an arbitrary algebraically closed field [here](http://www.jstor.org/discover/10.2307/2374145?uid=3738016&uid=2129&uid=2&uid=70&uid=4&sid=55977165963).   
 In a sense it may be considered an explanation of the geometric significance of the arithmetic genus: if you want it to satisfy certain geometric  properties, the definition is forced upon you.