Very briefly: until work of Hans Maass c. 1949, "modular" or "automorphic" both referred to holomorphic functions invariant-up-to-cocycle (that is, invariant holomorphic sections of a bundle) on a quotient $\Gamma\backslash X$. For $X$ the upper half-plane, these were *ellipic* modular forms, visible since the 19th century. For $X$ a product of upper half-planes, these were Hilbert-Blumenthal modular forms, also studied by Hecke and Siegel. For $X$ a Siegel upper half-space, *Siegel* modular forms, studied by Siegel and Hel Braun in 1939. The "elliptic" case arose partly from moduli problems concerning elliptic curves, indeed, and had its roots in the work of Abel and Jacobi in the early 19th century. The Hilbert-Blumenthal story seems to have been a conscious generalization, once the number-theoretic content of the elliptic modular case was illustrated in the late 19th century, e.g., as in in Fricke-Klein. The "Siegel modular" case has physical setting the moduli space for principally polarized abelian varieties of a fixed size.

The "general" case of discrete subgroups (Fuchsian, etc.) of $PSL(2,\mathbb R)$ acting on the upper half-plane (or, equivalently, $PSU(1,1)$ on the disk) was investigated by Poincare and others in the very late 19th century, and called "automorphic".

After Hecke saw that binary theta series give Dedekind zetas of complex quadratic extensions of $\mathbb Q$, he gave Maass the thesis problem of finding an analogue for real quadratic. This led Maass to discover "waveforms", solutions of $(\Delta-\lambda)f=0$ where
$\Delta=y^2({\partial^2\over \partial x^2}+{\partial^2\over \partial y^2})$
is the $SL(2,\mathbb R)$-invariant Laplacian on the upper half-plane. The Eisenstein series $E_s(z)=\sum'_{c,d} y^s/|cz+d|^{2s}$ are the most explicit examples, and also the "special waveforms" Maass found (that Mellin transform essentially to Dedekind zetas of real fields and Hecke L-functions thereupon) are explicit. These are "automorphic forms/functions".

Selberg, Roelcke, Gelfand-et-al, and a few others continued looking at the "analytic" side of these things in the 1950s. Siegel and Braun continued to work on holomorphic modular/automorphic forms on higher-dimensional spaces, with Braun initiating the investigation of "Hermitian" modular forms, that is, attached to the group $U(n,n)$ rather than $Sp(n,\mathbb R)$ for Siegel modular forms.

Starting in the late 1950s and 1960s, Shimura considered the algebraic geometry and arithmetic (that is, Hasse-Weil zeta functions, generation of classfields, and such) on many higher-dimensional "modular" varieties, subsumed mostly under the "PEL-type" label: polarization, endomorphism, level. By this point, it seems that "automorphic" was used to refer to these "general" situations, even though they had connections with moduli problems.

Also in the late 1950s and 1960s, Gelfand and his collaborators (Pieatetski-Shapiro, especially) emphasized the representation-theoretic possibilities in studying not only holomorphic modular/automorphic forms/function, but also Maass waveforms and other "generalizations". In particular, by about 1960 it was clear that from a repn theoretic viewpoint "holomorphic modular forms" and "real-analytic waveforms" had the commonality that both generated irreducible repns of the Lie group acting. Further, for congruence subgroups, being an eigenfunction for Hecke operators essentially meant generating irreducibles for the p-adic groups acting, as well. (In fact, even the "bad prime" behaviors are included nicely, if less formulaically, under this umbrella.)

Langlands' work on the spectral theory of automorphic forms and Eisenstein series in general, in the 1960s, and conjectures relating Artin L-functions to general cuspforms on $GL(n)$, etc., gave a big impetus to the "general" theory starting in the late 1960s. In part, this was made feasible by progress in the repn theory of semi-simple real Lie groups, especially by Harish-Chandra. The repn theory of p-adic groups, initiated mostly by MacDonald and Gelfand-et-al, started off a little more slowly, but also proved to be sufficiently robust as to be a "help" rather than "hindrance" in this aspect of the theory of afms.

The general study of moduli problems similarly needed additional inputs to continue to make progress, and Grothendieck-et-al's newer algebraic geometry, in the hands of Deligne and others, turned out to be a good language/viewpoint for this.

There is a lot more to be said, naturally. By this year, "modular" suggests "holomorphic", as well as "related to moduli problem". "Automorphic" suggests "something more general", but also can be used as an umbrella term. Holomorphic-except-for-singularities, that is, meromorphic forms, have also arisen in higher-dimensional settings, as in Borcherds products. On another hand, the "weak" automorphic/modular forms of Zwegers-et-al allow a controlled extension of the moderate-growth condition on waveforms.

Edit... : and it may be worth noting that various more-formal "definitions" are highly non-trivial to compare to each other, often depending upon appreciation of big theorems from repn theory, algebraic geometry, and number theory, that are usually _not_named_ during a formal discussion of "definitions". Further, there are more-elementary incompatibilities that are often harmless in a given context, but are not overtly acknowledged. For example, the "moderate growth" condition is not met by $L^2$ automorphic forms, for the same reason that functions in $L^2(\mathbb R)$ are typically not of moderate growth. Another is that requiring $\mathfrak z$-finiteness precludes taking an $L^2$ closure. But these awkwardnesses of the formal language are not genuine obstacles.