This space is certainly not compact in general, unless I misunderstood your question.
For instance, assume the probability space in question is $[0,1]$ with Lebesgue measure.
Then endow the isometry group with the pointwise convergence topology; it would be a compact group if the space were compact. The automorphism group of $([0,1],\lambda)$ (i.e measure-preserving bijections) , sits inside that group as a closed subgroup (it is exactly the space of isometries that fix $\emptyset$, but you do not need to know that to see that a limit of measure-preerving bijections is still a measure-preserving bijection) and is certainly very much noncompact: a very indirect way to see this is to note that there exist dense conjugacy classes (any aperiodic element has a dense conjugacy class by Rokhlin's lemma), whereas conjugacy classes would be closed if the group were compact.
Actually, this space is, in a sense, as far as possible from being compact while staying separable: endow it with a group structure by using symmetric difference; the the corresponding Polish group is extremely amenable, i.e every continuous action of this group on a compact space has a (global) fixed point. This is thus impossible for the space to be compact, obviously, but also to be locally compact.
(Note: I'm used to working with groups so this is what I used in my answer, you can probably remove them very easily...)