Answer is: Yes, one can recover $M$ if it is a compact manifold. See [J. V. Whittaker: On Isomorphic groups and homeomorphic spaces, Annals of Math 1963.][1] **EDIT** Actually, one knows a lot more, see, for example Tomasz Rybicki Journal: Proc. Amer. Math. Soc. 123 (1995), 303-310. MSC: Primary 58D05; Secondary 17B66, 22E65, 57R50 MathSciNet review: 1233982 And references therein... [1]: http://dl.dropbox.com/u/5188175/whit.pdf **ANOTHER EDIT** A quite different proof of a stronger theorem (actually a large set of theorems) than Whittaker's (actually, Whittaker's paper seems to be rather badly written) is given by Matatyahu Rubin in Rubin, Matatyahu(3-SFR) On the reconstruction of topological spaces from their groups of homeomorphisms. Trans. Amer. Math. Soc. 312 (1989), no. 2, 487–538.