Let $X$ be an oriented, closed $4$-manifold and $Y$ be an oriented, closed $3$-manifold smoothly embedded in $X$. Suppose we have a diffeomorphism $f:X \to X$ which is fixing $Y$ as a set, i.e., $f(Y)=Y$. Moreover, we assume that $f$ is topologically isotopic to the identity map. Does it imply up to some further smooth isotopy that $f$ is identity on $Y$, i.e., fixing $Y$ pointwise? (Also for simplification, one can assume that $X$ is simply connected.)

One thing in dim-4 that one needs to be careful about about is, an existence of topological isotopy may not imply smooth isotopy (Danny Ruberman found the existence of such examples).

One thing to be noted that if $\mathrm{Diff}^+(Y)$ is connected, then the above question has a positive answer. (For example $S^3$.)