In Chapter 8 of the book Gradient Flows In Metric Spaces and in the Space of Probability Measures by Ambrosio et al., the tangent space to the space of distributions on $X$ (let's say $X=\mathbb{R}^d$) with finite second moment $\mathscr{P}_2(X)$ is formally defined as $$\text{Tan}_\mu\mathscr{P}_2(X):= \overline{\{\nabla\varphi: \varphi\in C_c^\infty(X)\}}^{L^2(\mu,X)}$$ that is the $L^2$-closure of smooth gradient fields on $X$. I understand the equivalence between absolutely continuous curves on $\mathscr{P}_2(X)$ and minimal norm vector fields, which tells us that the above definition generalizes the construction of the tangent space in finite-dimensional manifolds as classes of differentiable curves.
However, I am curious about how well $\text{Tan}_\mu\mathscr{P}_2(X)$ can serve as a local model for $\mathscr{P}_2(X)$ near $\mu$. In finite-dimensional manifolds, the exponential map gives a local diffeomorphism between the tangent space and the underlying space. Is there a similar construction for $\mathscr{P}_2(X)$? I have read that $\mathscr{P}_2(X)$ is not even a Banach manifold and the above association is merely formal, so the answer is probably no, but I am not sure how severe the failure is.
This paper by J. Lott seems to suggest that restricting to the space $\mathscr{P}^\infty(X)$ of smooth positive densities turns the space into a true manifold and rigorously justifies many geometric concepts. Is there an exponential map in this case, for example?
For motivation, I am trying to prove a version of the center-stable manifold theorem for Wasserstein gradient flows (either weak or strong solutions, with any required degree of regularity). There is already a result for Banach spaces, but to extend the result I need to be able to lift local dynamics on $\mathscr{P}_2(X)$ to the tangent space in some sense.