Let $V$ be a smooth manifold obtained by attaching the ``open collar'' $[0,1)\times \partial N$ to a compact smooth manifold $N$ along the boundary. Let $\mathrm{Emb}(N, V)$ be the space of embeddings of $N$ into $V$ isotopic to the inclusion. The identity component $\mathrm{Diff\,} N$ of the diffeomorphism group of $N$ embeds into $\mathrm{Emb}(N, V)$ by precomposing a diffeomorphism with the inclusion.

Question: Under what conditions the embedding $\mathrm{Diff\,} N\to\mathrm{Emb}(N, V)$ is not a homotopy equivalence?

It is relevant to consider the $\mathrm{Diff\,} V$-action on $\mathrm{Emb}(N, V)$ by postcomposing. The orbit map $\mathrm{Diff\,} V\to\mathrm{Emb}(N, V)$ is a homotopy equivalence: its homotopy fiber consists of diffeomorphisms that are identity on $N$, and the group of such diffeomorphisms is contractible by the Alexander trick towards infinity.

The case when $V=\mathbb R^n$ and $N=D^n$ is fairly well understood: $\mathrm{Diff\,} \mathbb R^n$ deformation retracts onto $O(n)$ while $\mathrm{Diff\,}\mathbb D^n$ is homotopically more complicated when $n$ large. (Computing $\mathrm{Diff\,}\mathbb D^n$ involves pseudoisotopy and Waldhausen K-theory).

Naively I hope the case of a disk be used to produce elements of $\pi_i(\mathrm{Diff\,} N)$ that die in $\pi_i(\mathrm{Emb}(N,V))\cong\pi_i(\mathrm{Diff\,} V)$, e.g. by considering boundary connected sum of $N$ with $D^n$, and extending diffeomorphisms of $D^n$ that are identity on $\partial D^n$ to $N$ by the identity. Has this ever been done?

I am even more interested in ways to produce elements of $\pi_i(\mathrm{Emb}(N,V))$ that do not come from $\pi_i(\mathrm{Diff\,} N)$.