I can't explain the group theoretic structure of $\Gamma$, but I can explain the topological structure.  (Warning, this post is mostly a continuous stream of thoughts.  I hope it is well organized and truthful, but you should check the details.)

**The space of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow \mathbb{R}$**

There are a number of metrics and norms on spaces of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow \mathbb{R}$. Of course there are 

 - The $L^p$ norms (on the subspace of $p$-integrable functions).

Also, there are lesser-known metrics which give the topology of convergence in probability (a.k.a. convergence in measure). These two metrics are equivalent:

 - The *Ky-Fan metric* $$\rho_\textit{Ky-Fan} (f,g) = \inf\left\{\varepsilon > 0 : \lambda \left\{x : |f(x) - g(x)| \geq \varepsilon\right\} \leq \varepsilon\right\}.$$ This definition makes more sense when you consider the definition of convergence in probability.

 - The metric 
$$\rho (f,g) = \int \min \left\{|f-g|,1\right\} \, d\lambda$$
(If you know the name for this metric, please answer [this MO question][1]!)  Notice the similarity between this metric and the $L^1$ metric.  Also notice, that for indicator functions, this metric becomes the familiar metric $\rho(\mathbf{1}_A,\mathbf{1}_B) = \lambda(A \triangle B)$. 

**The space of measurable functions $f\colon(I,\mathcal{B},\lambda) \rightarrow I$**

For the subspace of functions $f\colon(I,\mathcal{B},\lambda) \rightarrow I$, it is easy to see that this last metric is exactly the same as the $L^1$ metric.  Moreover, one can show on this space that all the $L^p$ metrics are equivalent (easy exercise).

**The space of measure preserving automorphisms $\textrm{Aut}(I,\mathcal{B},\lambda)$**

The measure preserving automorphisms form a subspace of the previous space.  It is closed.  This is because the push-forward map $f \mapsto \lambda_f$ is continuous in any of the above metrics, where the topology on the codomain is given by the Levy-Prokohorov metric, that is the metric of convergence in distribution.

This space is therefore a complete separable metric space (Polish space) under any of the above metrics.  However, the usual candidates for a countable dense set (e.g. polynomials with rational coefficients) don't work.  Instead, the following functions form a nice dense set:  For each $n$ choose, consider a permutation $\pi$ on $\{0,\ldots,2^n - 1\}$.  Then let $f^n_\pi \colon [0,1] \rightarrow [0,1]$ be as follows.  Break up $[0,1]$ into $2^n$ equally spaces dyadic intervals and let $f^n_\pi$ rearrange the intervals according to $\pi$.

(Actually, consider the $L^1$ metric on this subset of basic functions.  Take two such "basic functions" $f^n_\pi$ and $f^n_\sigma$. (WLOG, they break up $[0,1]$ into the same number of intervals.)  Then the distance $\| f -g\|_1$ is $2^{-n}\sum_{i=0}^{2^n} (\pi(i) - \sigma(i))$.

In this way, one can think of this space as a continuum sized extension of the countable group $G = \bigcup_n S_{2^n}$ where we embed $S_{2^n}$ into $S_{2^{n+1}}$.  (Although, our metric necessarily breaks the symmetry of $S_{2^n}$.)

This space is not compact.  (One can find a sequence of such basic functions which does not have a convergence subsequence.)

**The space of ergodic measure preserving automorphisms $\textrm{Aut}_\textrm{Ergodic}(I,\mathcal{B},\lambda)$**

This is dense in the previous space.  To see this, consider an irrational shift $g_\alpha(x) = x + \alpha \mod 1$.  Then compose it with a basic function.  It only changes the $L^1$ norm of the basic function slightly, but this composed function is now ergodic.  (This takes a little thought.)


  [1]: https://mathoverflow.net/questions/139113/does-this-metric-have-an-official-name-levy-metric-ky-fan-metric