This is extremely late, but hopefully it's still of some use/interest to you.
p-adic L-functions are usually described as 'p-adic interpolations of special values of L-functions'. For Kubota--Leopoldt's p-adic interpolation of the Riemann zeta function, the relevant special values are all the values $\zeta(1-k)$, where $k$ is a positive integer.
Riemann zeta is a complex meromorphic function $\zeta : \mathbb{C} \to \mathbb{C}$. Probably the most naive hope is that you can construct a p-adic L-function as an $p$-adic meromorphic function $\zeta_p : \mathbb{Z}_p \to \mathbb{C}_p$ with the appropriate interpolation property, namely that $\zeta_p(1-k) = (*) \zeta(1-k)$ for all $k > 0$ and some correction factor $(*)$. Unfortunately this doesn't quite work: there is no single such function that captures all the special values. In particular, $\mathbb{Z}_p$ is the 'wrong' domain for the p-adic L-function.
From complex analytic functions to complex-valued measures: To fix this, as Chris mentioned in his comment, we use Tate's measure-theoretic interpretation of L-functions. You can identify $\mathbb{C}$ with the space of characters $\mathbb{R}_{>0} \to \mathbb{C}$, where $s$ corresponds to the character $x \mapsto x^s$. Then we view an $L$-function as a function on $\mathrm{Hom}_{\mathrm{cts}}(\mathbb{R}_{>0},\mathbb{C})$, which looks like a $\mathbb{C}$-valued measure on $\mathbb{R}_{>0}$. You can expand this by adding in the finite places to obtain a measure-theoretic description on the ideles $\mathbb{A}^\times$, capturing not only the original $L$-function but all of its twists by Dirichlet characters.
p-adic measures: In the p-adic world, we might instead consider $\mathbb{C}_p$-valued measures on the ideles. For topological reasons this essentially reduces (up to finite-order twist) to considering measures on $\mathbb{Z}_p^\times$, and - hey presto! - you get the measure-theoretic language in which you see statements about p-adic L-functions. Now, there is a single (pseudo-)measure $\mu$ on $\mathbb{Z}_p^\times$ that does interpolate all the special values $\zeta(1-k)$. (Here pseudo-measure = 'measure with simple poles'). In particular, we have
$$\int_{\mathbb{Z}_p^\times} x^k \cdot \mu = (1-p^{k-1})\zeta(1-k)$$
for all $k > 0$, a very clean interpolation.
p-adic measures to p-adic analytic functions: From this, one can indeed recover a description as an analytic function. There is a correspondence (Fourier or Amice transform) between $\mathbb{C}_p$-valued measures on $\mathbb{Z}_p^\times$ and bounded rigid analytic functions on $\mathrm{Hom}_{\mathrm{cts}}(\mathbb{Z}_p^\times,\mathbb{C}_p)$. The latter is the $\mathbb{C}_p$-points of the weight space $\mathcal{W}$, that shows up in the theory of p-adic families of modular forms/Hida theory/the eigencurve. The p-adic L-function, then, is naturally a p-adic meromorphic function on $\mathcal{W}(\mathbb{C}_p)$, with at worst simple poles.
Now, what about that original naive guess? There is a theory in this direction; it just isn't as conceptually nice. Note $\mathcal{W}(\mathbb{C}_p)$ breaks up into $p-1$ connected components (if $p$ is odd), each indexed by a character $\psi$ of $(\mathbb{Z}/p)^\times$. The possible characters are $\omega^i$, where $\omega$ is the Teichmuller character and $0 \leq i \leq k-2$. On the connected component corresponding to a fixed $\omega^i$, we are then reduced to considering characters of $1+p\mathbb{Z}_p$. This naturally contains $\mathbb{Z}_p$, where $s \in \mathbb{Z}_p$ corresponds to the character $x \mapsto x^s := \mathrm{exp}(s \ \mathrm{log}(x))$. (Note we're 'reversing' the process we adopted over $\mathbb{C}$!) Define, then,
$$\zeta_{p,i} : \mathbb{Z}_p \to \mathbb{C}_p, \ \ \ s \mapsto \int_{\mathbb{Z}_p^\times} \omega^i(x) \langle x \rangle^{1-s} \cdot \mu,$$
where $\langle - \rangle : \mathbb{Z}_p^\times \to 1 + p\mathbb{Z}_p$ is the projection. We get $p-1$ p-adic meromorphic functions $\zeta_{p,0},...,\zeta_{p,k-2}$ on $\mathbb{Z}_p$. Apart from a simple pole for $i=0$, each $\zeta_{p,i}$ is a power series, as you might hope. The function $\zeta_{p,i}$ has the following interpolation:
$$ \zeta_{p,i}(1-k) = (1-p^{k-1})\zeta(1-k) \ \ \ \ \forall k \equiv i \mod p-1.$$
In particular, each $\zeta_{p,i}$ only sees some of the special L-values, and you need to consider the collection of all $p-1$ of them -- or, in other words, the measure/analytic function on $\mathcal{W}(\mathbb{C}_p)$ -- to see all of them.
Twists by Dirichlet characters: Here's another benefit, that to me is an absolute clincher for the measure-theoretic language. Let $\chi$ be any Dirichlet character of $p$-power conductor. Then the measure-theoretic p-adic L-function also satisfies the following magic interpolation:
$$\int_{\mathbb{Z}_p^\times} \chi(x)x^k \cdot \mu = (1- \chi(p)p^{k-1}) L(\chi,1-k),$$
for any $k > 0$. In other words, the measure-theoretic object sees so many more L-values than the corresponding analytic function on $\mathbb{Z}_p$. (You can write down twisted analytic functions on $\mathbb{Z}_p$ that each sees some of these $L$-values; but somehow the single measure-theoretic object is parcelling together an infinite number of interesting analytic functions on $\mathbb{Z}_p$ into one package!) This is very much in the spirit of Tate's original observations, in which an L-function and its twists get parceled into the same measure-theoretic object in complex analytic terms.
Everything I've written here is explained in more detail in the notes I wrote with Joaquin Rodrigues on Kubota--Leopoldt: https://da380198-735d-4021-b7f2-6247f0586806.filesusr.com/ugd/946d8a_60256058271940f7b9d2c14dc721bf91.pdf
Generalisations: If you weren't already convinced, here's one final observation. If you want to construct p-adic L-functions for motives/automorphic forms over more general base fields, then the shape of the theory changes. Here one can observe that $\mathbb{Z}_p^\times$ is either the ray class group $\mathrm{Cl}_{\mathbb{Q}}^+(p^\infty)$, or the Galois group of the maximal abelian extension of $\mathbb{Q}$ unramified outside $p\infty$. For a number field $F$, measures on $\mathbb{Z}_p^\times$ generalise to measures/distributions on $\mathrm{Cl}_F^+(p^\infty)$, or on the Galois group of the maximal abelian extension of $F$ unramified outside $p\infty$. This is a very clean generalisation; for example, if $F$ is imaginary quadratic, then you get two-variable measures, with cyclotomic and anticyclotomic variation. This admits a clean description in terms of analytic functions on the corresponding weight space. The analogous generalisation of analytic functions on $\mathbb{Z}_p$ is a mess that doesn't seem natural to me at all.
