Are there 'analytic' $p$-adic modular forms. The most elementary way to define $p$-adic modular forms is via limits of classical modular forms.
More precisely $f \in \mathbb{Z}_p[[q]]$ is called a $p$-adic modular form
if there are modular forms $f_n$ with integral coefficients such that
$f \equiv f_n \mod p^n$ (as $q$-expansions). Note it does not really make sense to attribute 'a weight' to $f$ since 
the $f_n$ are allowed to have different (increasing weights). This is the older definition by Serre.
I know (but I do not understand) a newer definition by Katz, which has a more geometric flavor.
See here math.arizona.edu/~swc/notes/files/01BuzzardL2.pdf.
So we have an approach using the $q$-expansion and we have an approach to $p$-adic modular forms
using geometric ideas.
My question now is, whether there is also a developed theory on analytic p-adic modular forms?
Some ideas what this might mean. For example we could consider the Eisenstein series
$$ E_4(\tau)=\sum_{n,m \in \mathbb{Z}} \frac{1}{(n\tau+m)^4} $$
as function of $\tau$ not being an element of the upper half-plane but of some subset of $\mathbb{C}_p$.
Does this sum even converge somewhere in $\mathbb{C}_p$. And is it (up to a constant) a classical or Katz $p$-adic modular
form? Does it even equal (mod p) the Eisenstein series $E_4$?
A similar question could be posed for the infinite product usually defining the $\Delta$-function
$$ q\prod_{n=1}^\infty (1-q^n)^{24}. $$
Does this converge somewhere when $q$ is in some subset of $\mathbb{C}_p$. Is it a $p$-adic modular form?
If there is no such theory? Why not? Is it not interesting?
 A: I personally would say that the root of the problem is the absence of a globally defined exponential map on $\mathbb{C}_p$. In the complex world $z \mapsto q(z) = \exp(2 \pi i z)$ is an isomorphism between $\mathbb{C} / \mathbb{Z}$ and $\mathbb{C}^\times$; but $\mathbb{C}_p / \mathbb{Z}$ is a horrible mess, and has nothing to do with $\mathbb{C}_p^\times$.
This makes it hard to bridge between the world of $q$-expansions and the world of analytic functions satisfying nice functional equations. The functional equation of a modular form looks nice if you write it in terms of $z$, but if you try and write it in terms of $q$ alone, it will make no sense. 
That's why one has to take a different approach, building $p$-adic modular forms directly as $p$-adic analytic functions (more precisely: analytic sections of line bundles) on modular curves (more precisely: the rigid spaces attached to modular curves), rather than trying to build them as functions on some other space satisfying functional equations which have the post facto consequence that they descend to modular curves, as in the theory over $\mathbb{C}$.
A: Since I'm not an expert, this will be just a minor note in addition to the excellent answers already given.
The example modular forms $E_4$ and $\Delta$ you've written down are defined over the integers (after suitable normalization).  That is, they eat elliptic curves with nowhere-vanishing differentials over any base, and produce numbers (i.e., functions on the base) in a suitably canonical way.  In particular, they are already "classical" $p$-adic modular forms by straightforward base change, and they are analytic forms, because they can be canonically extended to rigid-analytic input data. 
The problem with the definition of $E_4$ that you wrote is that it uses the additive uniformization of elliptic curves ($\mathbb{C}/\Lambda \overset{\simeq}{\longrightarrow} E$) in an essential way, and elliptic curves only have an additive uniformization over the complex numbers.  However, the $q$-expansion $E_4(q) = 1 + 240 \sum_{n \geq 1} \sigma_3(n) q^n$ makes sense $p$-adic analytically, since the multiplicative uniformization $\mathbb{G}_m/q^{\mathbb{Z}} \overset{\simeq}{\longrightarrow} E$ works in both analytic worlds.
Random points:


*

*Katz's definition of modular form (see his paper in Springer Lecture Notes 350, scanned on his web page) can be straightforwardly modified to take analytic inputs (real, complex, or $p$-adic).

*When people say "$p$-adic modular form", they might mean only non-classical forms, i.e., those that aren't defined in the "too supersingular" locus.  The overconvergent forms, namely those that extend from the ordinary locus to the "not too supersingular" locus, are necessarily analytic in nature.

*These objects are interesting in part because they can be used to construct interesting Galois representations through some magic I don't understand.  I should say something clever using phrases like "$p$-adic Langlands" and "eigenvariety" here.

A: There is such a theory, but the analytic object that the forms live on is an analytified modular curve, not simply $\mathbb{C}_p$ (though there is a "$p$-adic upper-half plane" that can be used to uniformize some similar moduli spaces, but as far as I know not the usual modular curves).
Basically, if $f$ is a classical modular form of some weight $k$, $f$ can be realized as a section of a sheaf $\omega^{\otimes k}$ on a complex-analytic modular curve such as $X_1(N)$ obtained via quotient from the complex upper-half plane.  These curves and sheaves have algebraic models defined over $\mathbb{Q}$, and (under mild hypotheses) the form $f$ actually arises from a section of the associated sheaf $\omega^{\otimes k}$ on the modular curve defined over $\mathbb{Q}$ (or some finite extension).
Now you can go in another direction and consider the rigid-analytic space over $\mathbb{Q}_p$ associated to the smooth algebraic curve $X_1(N)$ and your form $f$ gives rise to a $p$-adic analytic object on this curve.  Now one can play games like considering subspaces obtained by removing disks around supersingular points to obtain general $p$-adic modular forms (such as limits of classical forms of varying weight) and overconvergent modular forms.
For what it's worth, these curves are $p$-adic analytic moduli spaces, and this point of view on $p$-adic modular forms essentially differs from Katz's by thinking about rigid spaces as opposed to formal schemes (the Raynaud point of view on $p$-adic analytic geometry).
Look at the papers of Coleman (such as his $p$-adic Banach spaces paper or the eigencurve paper with Mazur) if you would like to read more on this point of view.
