Let $X$ be an oriented, closed $4$-manifold and $Y$ be an oriented, closed $3$-manifold smoothly embedded in $X$. Now if we have a diffeomorphism $f:X \to X$ which is fixing $Y$ as a set, i.e., $f(Y)=Y$. And moreover, if we assume that $f$ is topologically isotopic to identity. Does it imply that upto some further smooth isotopy $f$ is identity on $Y$, i.e., fixing $Y$ pointwise? [Also for simplification, one can assume that $X$ is simply connected.]

One thing in dim-4 that one needs to be care about about is, existence of topological isotopy may not imply smooth isotopy [Danny Ruberman found existence of such examples first]. 

One thing to be noted that if $\mathrm{Diff}^+(Y)$ is connected, then above question has a positive answer. (For example $S^3$.)