Another way, somewhat related with the above answers, is the $p$-adic Volkenborn integral. You can find this, for example, in Schikhof's or in Alain Robert's books on $p$-adic calculus, or Henri Cohen vol. 2 of his books on number theory. This approach is useful because of the relation of Bernoulli numbers and L-functions: one can easily define good and elementary $p$-adic zeta functions using the Volkenborn integral (actually, this was Kubota and Leopoldt's original approach).
Let $\mathbb{Z}_p$ be the ring of $p$-adic integers and let $\mathbb{C}_p$ be the topological completion of an algebraic closure of the field of fractions $\mathbb{Q}_p$ of $\mathbb{Z}_p$ (a nice and large field for doing $p$-adic analysis). Let $f:\mathbb{Z}_p\to\mathbb{C}_p$ be an analytic function, that is, $f$ is of the form
$$f(x)=\sum_{n\ge0}a_n\frac{x^n}{n!},\qquad a_n\in\mathbb{C}_p,\quad \frac{a_n}{n!}\to0.$$
(We suppose $f$ analytic for simplicity and because of what you are asking). Then the Volkenborn integral of $f$ is defined by the following $p$-adic limit:
$$\int_{\mathbb{Z}_p}f(t)dt=\lim_{m\to\infty}p^{-m}\sum_{k=0}^{p^m-1}f(k).$$
Then, one has the following relation with Bernoulli numbers and polynomials:
$$\int_{\mathbb{Z}_p}t^ndt=B_n$$ and $$\int_{\mathbb{Z}_p}(x+t)^ndt=B_n(x).$$
This Volkenborn integral is a continuous linear operator on a Banach space of functions (see the books mentioned above). Hence, with $f$ as above, one obtains:
$$\int_{\mathbb{Z}_p}f(t)dt=\sum_{n\ge0}a_n\frac{B_n}{n!}$$ and $$\int_{\mathbb{Z}_p}f(x+t)dt=\sum_{n\ge0}a_n\frac{B_n(x)}{n!}.$$
Hope this helps.
Note: This integral is a special case of "$p$-adic distributions", which are one of the main tools that are now used to define $p$-adic zeta functions attached to arthmetic objects. See, for example, Washington or Lang books on cyclotomic fields for a nice introduction.
PS: For a nice "general zeta functions" interpretation of your question, see Lemma 2.4 in this article by Friedman and Pereira https://arxiv.org/abs/1105.2603 It was published in the IJNT, but the arxiv version is the same as the published version.