I believe that the usual remedy is a collar. That is, for any smooth manifold there is a suitable diffeomorphism from a neighborhood of $\partial M$ to $[0,1)\times \partial M$, or in other words a smooth embedding $[0,1)\times \partial M\to M$ that is "the identity" on the boundary. This allow you to glue along the boundary and get a smooth manifold. To see that the result is independent of the choice of an embedding you use the fact that any two such embeddings are smoothly isotopic.