So consider the $\mathbb{Q}$-vector space $V$ of functions which satisfy the following conditions

(1) $f:\mathbb{H}\rightarrow\mathbb{C}$ is holomorphic. Here $\mathbb{H}$ stands for the upper half plane.

(2) $f(z+1)=f(z)$

(3) The Fourier series of $f$ at infinity has the form $\sum_{n\geq 1} a_nq^n$ where $q=e^{2\pi iz}$ where the $a_n$'s are rational numbers

(4) $\frac{1}{Nz^2}f(\frac{-1}{Nz})=\pm f(z)$

Examples of non-zero elements of $V$ are given for example by $\mathbb{Q}$-linear combinations of modular forms associated to rational elliptic curves of conductor $N$
that have the same sign in their functional equation.
Let us denote this sub vector space by $W$. Unless I made a mistake in my calculation, an example of an element of $V$ which is not in $W$ could be
```
$$
\sum_{d|N} d a_d E_2(dz)
$$
```

with $\sum d a_d=0$ and $a_d=a_{N/d}$. If $N$ is sufficiently composite then we may find such $a_d$'s. Here $E_2$ is the weight $2$ Eisenstein series suitably normalized.

Q: How big is $V$ and is it possible to describe it in some interesting way?

isin the normalizer of $\Gamma_0(N)$ in ${\rm SL}_2({\bf R})$. However, once $N>4$ Fricke and $z \mapsto z+1$ do not generate the full normalizer. $$ $$ Also I don't think $E_2(z) - NE_2(Nz)$ works: that's a modular form of weight $2$ for all of $\Gamma_0(N)$ [proportional to the logarithmic differential of the modular function $\Delta(z) / \Delta(Nz)$], but it does not vanish at $q=0$. – Noam D. Elkies Sep 6 '11 at 23:47