# Definition of Hecke operators on orthogonal modular forms

In his paper Automorphic forms with singularities on Grassmannians, Borcherds poses Problem 16.5:

"Describe how the correspondence in this paper behaves under the action of Hecke operators."

Since the "correspondence in this paper" is the construction of special orthogonal modular forms ("Borcherds' products"), this seems to me to imply the existence of a Hecke theory for modular forms on orthogonal groups.

Unfortunately I wasn't able to find any literature dealing with (or even defining) this kind of orthogonal Hecke operators.

Does anyone know any reference for this?

For any reductive group $G$ over a number field $F$, so in particular for the orthogonal group of a quadratic form, there exists a theory of Hecke operators. Let's say you have a congruence arithmetic group $\Gamma\subset G({\mathbb Q})$ and $\alpha\in G({\mathbb Q})$. Let's also say that a modular form is a $\Gamma$-left-invariant function $f$ on $G$, then the function $L_{\alpha}f(x)=f(\alpha^{-1}x)$ is not $\Gamma$-invariant any more, but only invariant under $\Sigma=\alpha\Gamma\alpha^{-1}\cap\Gamma$. Fortunately, this group has finite index in $\Gamma$, so you can form the Hecke-operator $T_\alpha$ defined by $$T_\alpha f(x)=\sum_{\gamma\in \Gamma/\Sigma} L_\gamma L_\alpha f(x).$$ Then $T_\alpha f$ is $\Gamma$-invariant again, so the Hecke-operator maps modular forms for $\Gamma$ to modular forms for $\Gamma$.
Depending on where you want to go, there are other definitions of Hecke operators, for instance, you can go adelic, then modular fomrs become $G({\mathbb Q})$-invariant functions on $G({\mathbb A})$, where $\mathbb A$ is the adele-ring of $F$. Then Hecke-operators become integral operators, where you integrate the $G(F_v)$-action for a local completion $F_v$ of $F$. As to literature, this usually splits between books that mainly treat the $GL(n)$-case and others that go totally general. But anyway, there's a vast literature, you only need to google ''automorphic form'' and ''Hecke operator''.
• Freitag's notes for O(2,$n$) on his webpage (though he doesn't do loads with Hecke operators--there are also some papers that deal with cases for special $n$, but I haven't read them)