Let $X$ be a oriented, closed 4-manifold and $Y$ be an oriented, closed 3-manifold smoothly embedded in $X$. Now if we have a diffeomorphism $f:X \to X$ which is fixing $Y$ as a set, i.e $f(Y)=Y$. And moreover, if we assume that $f$ is topologically isotopic to Identity. Does it imply that upto some further smooth isotopy $f$ is identity on $Y$, i.e fixing $Y$ point-wise? [Also for simplification, one can assume that $X$ is simply-connected.]

One thing in dim-4 that one needs to be care about about is, existence of topological isotopy may not imply smooth isotopy [Danny Ruberman found existence of such examples first]. 

One thing to be noted that if $Diff^+(Y)$ is connected, then above question has a positive answer. (For example $S^3$.)