What kind of object do you want to consider $M \# N$ to be, an oriented manifold, or unoriented? Presumably you're taking the connect sum up to some kind of equivalence. You'll also need for $M$ and $N$ to be connected if you want connect-sum to be well-defined in any sense.
In the oriented sense, $M \# N$ is well-defined up to orientation-preserving diffeomorphism, and contains both the punctured $M$ and the punctured $N$ as oriented submanifolds, so it depends on the orientations of $M$ and $N$ respectively.
As an unoriented object taken up to diffeomorphism, a connect-sum is well defined provided either input manifold has an orientation-reversing diffeomorphism.
Explicit examples where you can see there is or is not diffeomorphisms between such are connect sums of complex projective spaces (and/or their orientation reverses). There's also examples with $3$-dimensional lens spaces but working out which ones of those admit orientation-reversing diffeomorphisms is more work. All $1$ and $2$-manifolds admit orientation-reversing diffeomorphisms so there's no good examples there.
edit: in detail for $\mathbb CP^2$, the intersection form on $H_2(\mathbb CP^2 \# \mathbb CP^2)$ is definite, regardless of what orientation you give the connect-sum. But if you take the connect sum with one factor orientation-reversed $H_2(\mathbb CP^2 \# \overline{\mathbb CP^2})$, the intersection form is indefinite. For $3$-dimensional lens spaces, the torsion linking form is a good analogous invariant.
Have you looked at a book like Kosinski's Differential Topology? It covers these kinds of operations on manifolds.
If you want a lower-dimensional example that's easier to see, you could take the analogous connect-sum of knots in $S^3$. This is well-defined for oriented knots, but for unoriented knots you only get a well-defined operation when the knots are invertible, which means there's an orientation-preserving diffeo of $S^3$ that preserves the knot and reverses the orientation of the knot.