Connected sum of topological manifolds A definition of the connected sum of two $n$-manifolds $M$ and $M'$  begins by considering two $n$-balls $B$ in $M$, $B'$ in $M'$, and glueing the varieties $M\setminus \mathring B$ and $M'\setminus \mathring B'$ along their boundary (an $(n-1)$-sphere) by an orientation-reversing homeomorphism.  The construction depends a priori on these various choices, but it is asserted at many places of the litterature (Lee's book on topological manifolds for example, as well as Wikipedia) 
 that the result does not depend on these choices.
In the differentiable case, a reference is given to a theorem of Palais (Natural operations on differential forms, Thm. 5.5) which asserts — roughly — that two embedding of $n$-balls differ by a global diffeomorphism which is isotopic to identity.
Are the details of this independence written somewhere in the litterature,
both in the continuous and the smooth case?
 A: In the topological category the proof that connected sum is well-defined depends on the Annulus Theorem, first proved by Kirby; the necessity of the Annulus Theorem is seen from Bruno Martelli's answer. So you are not likely to find a proof before Kirby's paper. Perhaps someone jotted a proof down, maybe someone who thought about the Annulus Theorem when it was still a conjecture, and realized that well-definedness of connected sum was a good application. But, I do not know. 
Anyway, the proof is straightforward once you have the Annulus Theorem. Here's a sketch.
There's a couple of missing hypotheses. One must assume $M,M'$ are connected. One must also assume $M,M'$ are oriented. And one must assume the balls $B,B'$ are "nicely embedded"; at the minimum, assume that the boundary spheres $S,S'$ are locally bicollared, which implies globally bicollared by Brown's theorem. This rules out nastiness like an Alexander horned ball.
Now one shows that the connected sum is independent of the choice of gluing map $S \to S'$ (assumed to reverse orientation). This follows from the fact that any two homeomorphisms $S \to S'$ which agree on orientations are isotopic: once that is known, one absorbs the isotopy into the collar neighborhoods. Proving this fact may already require the Annulus Theorem.
For the rest, it suffices to prove that for any two nicely embedded balls $B_1,B_2 \subset M$ there exists an orientation preserving homemeorphism of $M$ taking $B_1$ to $B_2$, in fact an ambient isotopy. Using the boundary bicollaring, we may assume $B_1,B_2$ are contained respectively in open balls $U_1,U_2$, which are centered on points $p_1,p_2$ in some coordinate chart. We can also assume that $p_1=p_2$, because there is an ambient isotopy of $M$ taking $p_1$ to $p_2$: connect $p_1$ to $p_2$ by a path, cover the path by finitely many charts, and concatenate a sequence of ambient isotopies supported in these finitely many charts, moving $p_1$ along the path step by step to $p_2$. We can also replace $B_1$ by an arbitrarily small subball in $U_1$ centered at $p_1$, and similarly for $B_2$; this is straightforward to check using an ambient isotopy supported in the coordinate charts for $U_1$ and $U_2$. In particular, we can assume $B_1$ is contained in the interior of $B_2$.
Now apply the annulus theorem: the difference $B_2 \setminus B_1$ is homeomorphic to a sphere crossed with an interval. Using this, one can then ambiently isotope $B_2$ to $B_1$.
A: In the smooth setting, I would suggest you to read the paper of Kervaire and Milnor on homotopy spheres. 
The paper explains clearly that connected sum must be defined with some care in higher dimensions: if you choose any orientation-reversing diffeomorphism of the two $(n-1)$-spheres, then the resulting smooth manifold is not determined (low-dimensional topologists don't care about that, because problems arise only when $n\geq 7$).
In fact, every exotic sphere is obtained by gluing two $n$-discs via some diffeomorphism of their boundaries, and this would imply that every exotic sphere is the connected sum of two $n$-spheres! Indeed, to define unambiguously the connected sum you need to choose a particular isotopy class of diffeomorphisms of the two boundary $(n-1)$-spheres, and luckily this can be done by taking one that extends to the two removed discs. This is clearly explained in the paper of Milnor and Kervaire. 
With that requirement, connected sum is well-defined and produces a unique result (up to diffeomorphism). The independence of the result is obtained using a theorem of Palais and Cerf that states that two smooth embeddings of the $n$-disc in a connected $n$-manifold are always related by a diffeomorphism of the manifold.
A: The previous version of this post was incorrect.
EDIT:
I just taught a related fact in my undergraduate topology class (in the topological category), and in fact I usually give a version of this as a homework exercise (with hints). 
The idea is to define connected sum along the coordinate balls, i.e. the balls that are mapped via a fixed coordinate chart to standard balls in $\mathbb R^n$. For such balls the argument is easy, and no annulus theorem is needed because any two metric balls in $\mathbb R^n$ are ambiently isotopic by an isotopy that equals the identity outside a compact subset  (this can be done with bare hands). Since the homeomorphism group of manifold acts transitively on the manifold (nice exercise), any two points can be brought into the same chart. This provides some clue (not a complete proof) why the connected sum operation is independent of the choices involved. 
