I was similarly curious when writing the introduction of my PhD thesis,
since the context was moduli spaces of Abelian differentials.
I felt the need to dig a bit and include a discussion there.

Here is an English translation of what I wrote then. The original is
here
(see page 9 of the introduction).

**1. Moduli spaces**

The term "moduli space" is often preferred to "parameter space" in contexts
where one is interested in describing geometric objects up to a certain
equivalence.

For instance, we consider the modulus of a cylinder (ratio of its height to
its circumference) when we are interested in its shape without caring about
its size. The moduli space of cylinders is the set of positive reals.

A more instructive example is the moduli space of tori.

**1.1. Tori.**

Here again, we seek to describe the shape of a torus (of real dimension 2,
endowed with a complex structure) without taking its size into account.

A torus can be defined as a quotient $\mathbb{C}/\Lambda$ where $\Lambda$
is a lattice in $\mathbf{C}$. Consider tori equivalent when they correspond
to lattices obtained from each other by rotation and dilatation.
Given a torus, we can then consider it as a quotient $\mathbb{C}/\Lambda$
where $\Lambda$ is a lattice in $\mathbb{C}$ with basis $(1, \tau)$. Two
parameters $\tau$ and $\tau'$ define equivalent tori if their difference
is an integer, or if they are opposite or inverse of each other.
We can therefore assume that $\lvert \operatorname{Re} \tau \rvert < 1/2$,
$\operatorname{Im} \tau > 0$, and $\lvert \tau \rvert > 1$. This draws a
domain in the upper half-plane, located between the vertical lines at
x-coordinate $-1/2$ and $1/2$, and outside the circle of radius $1$
centered at the origin. Some points in this domain still need to be
identified: vertical half-lines at x-coordinate $-1/2$ and $1/2$ by
horizontal translation, and the two halves of the arc of circle which
bounds the domain below by the map $z \mapsto -1/z$. The identifications
of these parts of the boundary of the domain can be described globally:
they are obtained by reflection about its (vertical) symmetry axis.

We call moduli space of tori the space obtained after performing these
identifications. Its topology is that of a sphere minus a point.
Its geometry is richer, and is inherited from the hyperbolic metric
on the upper half-plane. The points corresponding to $i$ and to
$i 2 \pi / 3$ represent the square torus and the hexagonal torus,
which respectively admit automorphisms of order 4 and 6. They are
cone points of respective angles $\pi$ and $2\pi/3$ in the moduli
space of tori.

This very classical moduli space is called the modular surface or
the modular curve, depending on whether one chooses to adopt the real
or complex viewpoint. Because of these cone points, it is not quite
a manifold; we call it an orbifold.

I stopped the discussion there but wish I had expanded a bit on
how Riemann found that it takes "more moduli" to describe the
geometry of closed compact Riemann surfaces of higher genus, and
concluded it took "$3g-3$ complex moduli" for genus $g > 1$.

Or if one prefers, the "modulus" of a genus $g$ closed compact
Riemann surface, for $g > 2$, is no longer a real number or a
complex number or a point in a quotient of the upper half-plane,
but a point in a $(3g-3)$-dimensional complex orbifold...