I can't explain the group theoretic structure of $\Gamma$, but I can explain the topological structure. (Warning, this post is mostly a continuous stream of thoughts. I hope it is well organized and truthful, but you should check the details.)

The space of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow \mathbb{R}$

There are a number of metrics and norms on spaces of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow \mathbb{R}$. Of course there are

• The $L^p$ norms (on the subspace of $p$-integrable functions).

Also, there are lesser-known metrics which give the topology of convergence in probability (a.k.a. convergence in measure). These two metrics are equivalent:

• The Ky-Fan metric $$\rho_\textit{Ky-Fan} (f,g) = \inf\left\{\varepsilon > 0 : \lambda \left\{x : |f(x) - g(x)| \geq \varepsilon\right\} \leq \varepsilon\right\}.$$ This definition makes more sense when you consider the definition of convergence in probability.

• The metric $$\rho (f,g) = \int \min \left\{|f-g|,1\right\} \, d\lambda$$ (If you know the name for this metric, please answer this MO question!) Notice the similarity between this metric and the $L^1$ metric. Also notice, that for indicator functions, this metric becomes the familiar metric $\rho(\mathbf{1}_A,\mathbf{1}_B) = \lambda(A \triangle B)$.

The space of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow I$

For the subspace of functions $f\colon(I,\mathcal{B},\lambda) \rightarrow I$, it is easy to see that this last metric is exactly the same as the $L^1$ metric. Moreover, one can show on this space that all the $L^p$ metrics are equivalent (easy exercise).

The space of measure preserving automorphisms $\textrm{Aut}(I,\mathcal{B},\lambda)$

The measure preserving automorphisms form a subspace of the previous space. It is closed. This is because the push-forward map $f \mapsto \lambda_f$ is continuous in any of the above metrics, where the topology on the codomain is given by the Levy-Prokohorov metric, that is the metric of convergence in distribution.

This space is therefore a complete separable metric space (Polish space) under any of the above metrics. However, the usual candidates for a countable dense set (e.g. polynomials with rational coefficients) don't work. Instead, the following functions form a nice dense set: For each $n$ choose, consider a permutation $\pi$ on $\{0,\ldots,2^n - 1\}$. Then let $f^n_\pi \colon [0,1] \rightarrow [0,1]$ be as follows. Break up $[0,1]$ into $2^n$ equally spaces dyadic intervals and let $f^n_\pi$ rearrange the intervals according to $\pi$.

(Actually, consider the $L^1$ metric on this subset of basic functions. Take two such "basic functions" $f^n_\pi$ and $f^n_\sigma$. (WLOG, they break up $[0,1]$ into the same number of intervals.) Then the distance $\| f -g\|_1$ is $2^{-n}\sum_{i=0}^{2^n} (\pi(i) - \sigma(i))$.

In this way, one can think of this space as a continuum sized extension of the countable group $G = \bigcup_n S_{2^n}$ where we embed $S_{2^n}$ into $S_{2^{n+1}}$. (Although, our metric necessarily breaks the symmetry of $S_{2^n}$.)

This space is not compact. (One can find a sequence of such basic functions which does not have a convergence subsequence.)

The space of ergodic measure preserving automorphisms $\textrm{Aut}_\textrm{Ergodic}(I,\mathcal{B},\lambda)$

This is dense in the previous space. To see this, consider an irrational shift $g_\alpha(x) = x + \alpha \mod 1$. Then compose it with a basic function. It only changes the $L^1$ norm of the basic function slightly, but this composed function is now ergodic. (This takes a little thought.)

I can't explain the group theoretic structure of $\Gamma$, but I can explain the topological structure. (Warning, this post is mostly a continuous stream of thoughts. I hope it is well organized and truthful, but you should check the details.)

The space of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow \mathbb{R}$

There are a number of metrics and norms on spaces of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow \mathbb{R}$. Of course there are

• The $L^p$ norms (on the subspace of $p$-integrable functions).

Also, there are lesser-known metrics which give the topology of convergence in probability (a.k.a. convergence in measure). These two metrics are equivalent:

• The Ky-Fan metric $$\rho_\textit{Ky-Fan} (f,g) = \inf\left\{\varepsilon > 0 : \lambda \left\{x : |f(x) - g(x)| \geq \varepsilon\right\} \leq \varepsilon\right\}.$$ This definition makes more sense when you consider the definition of convergence in probability.

• The metric $$\rho (f,g) = \int \min \left\{|f-g|,1\right\} \, d\lambda$$ (If you know the name for this metric, please answer this MO question!) Notice the similarity between this metric and the $L^1$ metric. Also notice, that for indicator functions, this metric becomes the familiar metric $\rho(\mathbf{1}_A,\mathbf{1}_B) = \lambda(A \triangle B)$.

The space of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow I$

For the subspace of functions $f\colon(I,\mathcal{B},\lambda) \rightarrow I$, it is easy to see that this last metric is exactly the same as the $L^1$ metric. Moreover, one can show on this space that all the $L^p$ metrics are equivalent (easy exercise).

The space of measure preserving automorphisms $\textrm{Aut}(I,\mathcal{B},\lambda)$

The measure preserving automorphisms form a subspace of the previous space. It is closed. This is because the push-forward map $f \mapsto \lambda_f$ is continuous in any of the above metrics, where the topology on the codomain is given by the Levy-Prokohorov metric, that is the metric of convergence in distribution.

This space is therefore a complete separable metric space (Polish space) under any of the above metrics. However, the usual candidates for a countable dense set (e.g. polynomials with rational coefficients) don't work. Instead, the following functions form a nice dense set: For each $n$ choose, consider a permutation $\pi$ on $\{0,\ldots,2^n - 1\}$. Then let $f^n_\pi \colon [0,1] \rightarrow [0,1]$ be as follows. Break up $[0,1]$ into $2^n$ equally spaces dyadic intervals and let $f^n_\pi$ rearrange the intervals according to $\pi$.

(Actually, consider the $L^1$ metric on this subset of basic functions. Take two such "basic functions" $f^n_\pi$ and $f^n_\sigma$. (WLOG, they break up $[0,1]$ into the same number of intervals.) Then the distance $\| f -g\|_1$ is $2^{-n}\sum_{i=0}^{2^n} (\pi(i) - \sigma(i))$.

In this way, one can think of this space as a continuum sized extension of the countable group $G = \bigcup_n S_{2^n}$ where we embed $S_{2^n}$ into $S_{2^{n+1}}$. (Although, our metric necessarily breaks the symmetry of $S_{2^n}$.)

This space is not compact. (One can find a sequence of such basic functions which does not have a convergence subsequence.)

The space of ergodic measure preserving automorphisms $\textrm{Aut}_\textrm{Ergodic}(I,\mathcal{B},\lambda)$

This is dense in the previous space. To see this, consider an irrational shift $g_\alpha(x) = x + \alpha \mod 1$. Then compose it with a basic function. It only changes the $L^1$ norm of the basic function slightly, but this composed function is now ergodic. (This takes a little thought.)