A complex torus of dimension $d$ can be written as a quotient $\mathbb C^d/\mathbb Z^{2d}$. Thus it is determined by a map $\mathbb Z^{2d} \to \mathbb C^{d} \cong \mathbb R^{2d}$. We can specify this with an element of $GL_{2d} (\mathbb R)$.

However, sometimes two elements of $GL_{2d}(\mathbb R)$ give the same complex torus. This can happen in two ways. Either we can reparametrize $\mathbb C^d$, via an element of $GL_d(\mathbb C)$, or we can reparametrize $\mathbb Z^{2d}$, via an element of $GL_{2d} (\mathbb Z)$. Thus, the moduli space of complex tori is $GL_d(\mathbb C) \backslash GL_{2d}(\mathbb R) / GL_{2d}(\mathbb Z)$.

Riemann proved that a torus is an abelian variety if and only if it has a Riemann form. One can construct complex tori that do not have Riemann forms. If one is interested in abelian varieties, the moduli spaces that are best-behaved are usually the moduli space of abelian varieties with a fixed Riemann form. One can classify these by fixing a Hermitian form on $\mathbb C^d$ and associated symplectic form. Similarly, fix a symplextic form on $\mathbb Z^{2d}$. Then the embedding $\mathbb Z^{2d} \to \mathbb C^d$ must be symplectic, corresponding to an element of $SL_{2d}(\mathbb R)$. The reparamaterization on the complex side now comes from an element preserving the Hermitian form, meaning an element of $U_d(\mathbb C)$. On the integer side, it is an element of $SL_{2d}(\mathbb Z)$. This gives the double coset description $U_d(\mathbb C)\backslash SL_{2d}(\mathbb R)/SL_{2d}(\mathbb Z)$.

I think that is the source of the "action of $SL_2(\mathbb R)$" you are looking for.

A complex torus of dimension $d$ can be written as a quotient $\mathbb C^d/\mathbb Z^{2d}$. Thus it is determined by a map $\mathbb Z^{2d} \to \mathbb C^{d} \cong \mathbb R^{2d}$. We can specify this with an element of $GL_{2d} (\mathbb R)$.

However, sometimes two elements of $GL_{2d}(\mathbb R)$ give the same complex torus. This can happen in two ways. Either we can reparametrize $\mathbb C^d$, via an element of $GL_d(\mathbb C)$, or we can reparametrize $\mathbb Z^{2d}$, via an element of $GL_{2d} (\mathbb Z)$. Thus, the moduli space of complex tori is $GL_d(\mathbb C) \backslash GL_{2d}(\mathbb R) / GL_{2d}(\mathbb Z)$.

Riemann proved that a torus is an abelian variety if and only if it has a Riemann form. One can construct complex tori that do not have Riemann forms. If one is interested in abelian varieties, the moduli spaces that are best-behaved are usually the moduli space of abelian varieties with a fixed Riemann form. One can classify these by fixing a Hermitian form on $\mathbb C^d$ and associated symplectic form. Similarly, fix a symplextic form on $\mathbb Z^{2d}$. Then the embedding $\mathbb Z^{2d} \to \mathbb C^d$ must be symplectic, corresponding to an element of $SP_{2d}(\mathbb R)$. The reparamaterization on the complex side now comes from an element preserving the Hermitian form, meaning an element of $U_d(\mathbb C)$. On the integer side, it is an element of $SP_{2d}(\mathbb Z)$. This gives the double coset description $U_d(\mathbb C)\backslash SP_{2d}(\mathbb R)/SP_{2d}(\mathbb Z)$.

I think that is the source of the "action of $SP_{2d}(\mathbb R)$" you are looking for.