Definition of Hecke operators on orthogonal modular forms In his paper Automorphic forms with singularities on Grassmannians, Borcherds poses Problem 16.5: 
"Describe how the correspondence in this paper behaves under
the
action of Hecke operators."
Since the "correspondence in this paper" is the construction of special orthogonal modular forms ("Borcherds' products"), this seems to me to imply the existence of a Hecke theory for modular forms on orthogonal groups. 
Unfortunately I wasn't able to find any literature dealing with (or even defining) this kind of orthogonal Hecke operators. 
Does anyone know any reference for this?
 A: For any reductive group $G$ over a number field $F$, so in particular for the orthogonal group of a quadratic form, there exists a theory of Hecke operators. Let's say you have a congruence arithmetic group $\Gamma\subset G({\mathbb Q})$ and $\alpha\in G({\mathbb Q})$. Let's also say that a modular form is a $\Gamma$-left-invariant function $f$ on $G$, then the function $L_{\alpha}f(x)=f(\alpha^{-1}x)$ is not $\Gamma$-invariant any more, but only invariant under $\Sigma=\alpha\Gamma\alpha^{-1}\cap\Gamma$. Fortunately, this group has finite index in $\Gamma$, so you can form the Hecke-operator $T_\alpha$ defined by
$$
T_\alpha f(x)=\sum_{\gamma\in \Gamma/\Sigma} L_\gamma L_\alpha f(x).
$$
Then $T_\alpha f$ is $\Gamma$-invariant again, so the Hecke-operator maps modular forms for $\Gamma$ to modular forms for $\Gamma$.
Depending on where you want to go, there are other definitions of Hecke operators, for instance, you can go adelic, then modular fomrs become $G({\mathbb Q})$-invariant functions on $G({\mathbb A})$, where $\mathbb A$ is the adele-ring of $F$. Then Hecke-operators become integral operators, where you integrate the $G(F_v)$-action for a local completion $F_v$ of $F$. As to literature, this usually splits between books that mainly treat the $GL(n)$-case and others that go totally general. But anyway, there's a vast literature, you only need to google ''automorphic form'' and  ''Hecke operator''.
A: To supplement Anton's general answer, here are some specific references for modular forms on orthogonal groups:


*

*Freitag's notes for O(2,$n$) on his webpage (though he doesn't do loads with Hecke operators--there are also some papers that deal with cases for special $n$, but I haven't read them)

*Pei-Yu Tsai's thesis (though she only works locally)

*Many people have studied Hecke operators for PGSp(4) = SO(5).  Besides the well known stuff for the split form (corresponding to usual degree 2 Siegel modular forms), people have also worked on non-split forms.  E.g., see Lansky and Pollack for some explicit work with the nonsplit form, or more recent work by Dembele and Cunningham.  This fits in Gross's context of algebraic modular forms.
