Let $M$ be a connected compact topological $n$-dimensional manifold without a boundary and with a CW-structure $M= \bigcup M^i$. We have that $$ (\#^g M)\smallsetminus D^n \simeq \bigvee_{i=1}^gM^{n-1}$$ The symmetric group $S_g$ acts on $\bigvee_{i=1}^gM^{n-1}$ by permuting the summands. In particular every element $\sigma\in S_g$ induces an automorphism $\sigma_*\colon \bigvee_{i=1}^gM^{n-1}\to\bigvee_{i=1}^gM^{n-1}$, which extends (non-uniquely) to a homotopy self-equivalence of $(\#^g M)\smallsetminus D^n$.
My question: Could we extend $\sigma_*$ to be a homotopy self-equivalence of $(\#^g M)\smallsetminus D^n$ that preserves the boundary pointwise?