The aim is to sample distributions using Fisher information (as mass matrix in Hamiltonian MCMC sampling). Details can be found in http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.190.580&rep=rep1&type=pdf (Riemann Manifold Langevin and Hamiltonian Monte Carlo).

Assuming differentiability, Fisher information is the expectation of the covariance of gradient(with respect to the statistical parameters) of logarithm of the distribution. In the Bayesian setup, where we want to sample from the parameter space, we use this Fisher information as the mass matrix.

**This assumes a parametric model for the distribution** to be sampled which is common in the Bayesian setup. However, if I just want to sample some distribution $p(\theta)$ with no connection to a statistical model, how do we proceed?
One way could be to associate an artificial statistical model with parameters $\theta$ with each $\theta$ giving the distribution $N(0,1)$. Here, the covariance need not be integrated. Is this a valid/useful way?
Further, we can use this to even handle sampling from parameter space in Bayesian case by taking the observation vector as constant and associating $N(0,1)$ distributions with all the parameters. Is this an acceptable algorithm?

Edit - We can also attach something more interesting than $N(0,1)$ to help us sample better.