Theo, the answer is basically "yes". It's a qualified "yes", but only very lightly qualified.
Precisely: if a natural transformation between functors $\mathcal{C} \to \mathcal{D}$ is pointwise epi then it's epi. The converse doesn't always hold, but it does if $\mathcal{D}$ has pushouts. Dually, pointwise mono implies mono, and conversely if $\mathcal{D}$ has pullbacks.
The context for this --- and an answer to your more general question --- is the slogan
(Co)limits are computed pointwise.
You have, let's say, two functors $F, G: \mathcal{C} \to \mathcal{D}$, and you want to compute their product in the functor category $\mathcal{D}^\mathcal{C}$. Assuming that $\mathcal{D}$ has products, the product of $F$ and $G$ is computed in the simplest possible way, the 'pointwise' way: the value of the product $F \times G$ at an object $A \in \mathcal{C}$ is simply the product $F(A) \times G(A)$ in $\mathcal{D}$. The same goes for any other shape of limit or colimit.
For a statement of this, see for instance 5.1.5--5.1.8 of these notesthese notes. (It's probably in Categories for the Working Mathematician too.) See also sheet 9, question 1 at the page linked to. For the connection between monos and pullbacks (or epis and pushouts), see 4.1.31.
You do have to impose this condition that $\mathcal{D}$ has all (co)limits of the appropriate shape (pushouts in the case of your original question). Kelly came up with some example of an epi in $\mathcal{D}^\mathcal{C}$ that isn't pointwise epi; necessarily, his $\mathcal{D}$ doesn't have all pushouts.