For any reductive group $G$ over a number field $F$, so in particular for the orthogonal group of a quadratic form, there exists a theory of Hecke operators. Let's say you have a congruence arithmetic group $\Gamma\subset G({\mathbb Q})$ and $\alpha\in G({\mathbb Q})$. Let's also say that a modular form is a $\Gamma$-left-invariant function $f$ on $G$, then the function $L_{\alpha}f(x)=f(\alpha^{-1}x)$ is not $\Gamma$-invariant any more, but only invariant under $\Sigma=\alpha\Gamma\alpha^{-1}\cap\Gamma$. Fortunately, this group has finite index in $\Gamma$, so you can form the Hecke-operator $T_\alpha$ defined by
$$
T_\alpha f(x)=\sum_{\gamma\in \Gamma/\Sigma} L_\gamma L_\alpha f(x).
$$
Then $T_\alpha f$ is $\Gamma$-invariant again, so the Hecke-operator maps modular forms for $\Gamma$ to modular forms for $\Gamma$.

Depending on where you want to go, there are other definitions of Hecke operators, for instance, you can go adelic, then modular fomrs become $G({\mathbb Q})$-invariant functions on $G({\mathbb A})$, where $\mathbb A$ is the adele-ring of $F$. Then Hecke-operators become integral operators, where you integrate the $G(F_v)$-action for a local completion $F_v$ of $F$. As to literature, this usually splits between books that mainly treat the $GL(n)$-case and others that go totally general. But anyway, there's a vast literature, you only need to google ''automorphic form'' and ''Hecke operator''.