It has bug me for a while that I don't have a good understanding of the theory of Hecke operators. For elliptic modular forms, it was explained in Koblitz's book that they arose from viewing the modular forms as function on modular points (lattices in $\mathbb{C}$, possibly with additional structures) but I feel this is very particular to elliptic modular forms as there doesn't seem to be a similar interpretation for other kind of modular forms such as Siegel modular forms. For Siegel modular forms of general level $\Gamma = \Gamma^{(2)}(N)$, it was defined by Andrianov in his book Modular Forms and Hecke Operators: He proved that the commeasurator of $\Gamma^{(2)}(N)$ in $G = GSp_4(\mathbb{Q})^+$ is the whole group $G$ (Lemma 3.1) and commented that
Using Lemma 3.1 as a point of departure, one could determine the Hecke ring of the pair $(\Gamma, G)$ and then consider its representations on spaces of modular forms for the group $\Gamma$. However, the structure of the Hecke rings that arise is in general unknown, and one does not yet have a concrete general theory of Hecke operators.
Because our constructions are not meant as an end in themselves, but rather as a means for studying Diophantine problems in number theory, we shall simplify the situation by, in the first place, limiting ourselves to the types of congruence subgroups that arise in arithmetic, and, in the second place, considering certain subrings of the Hecke ring of the pair $(\Gamma, G)$, rather than the entire Hecke ring.
(I changed Andrianov's $K$ to $\Gamma$ and $S$ to $G$.)
The subring Andrianov mentioned is defined on page 124, namely the ring of Hecke operator associated to the Shimura pair $(\Gamma, S(\Gamma))$ for certain subgroup $S(\Gamma)$ of $G$ defined in equation (3.5).
Now, I have no idea how Andrianov came up with that $S(\Gamma)$. Is there a systematic explanation of the ring? I am aware that for level 1, the theory seems to be explainable from the representation theoretic perspective and Satake isomorphism.
I would imagine that a similar theory could be developed for Hermitian modular forms (as in Ikeda's paper https://www.math.kyoto-u.ac.jp/~ikeda/hermitian) but I am not so sure what one should change from the Siegel case. What should be "the" Hecke algebra for Hermitian modular forms of level $N$? (The group $U(2,2)$ I think does not split so I don't know how Satake isomorphism will work here.)