Logically these are two different subjects.

1. Ergodic theory is about transformations of a space equipped with a measure (and the transformation preserves this measure). The measure is given in advance. The last 3 notions you mention are related to this situation, and express the various degrees of "mixing".

2. Topological dynamics is about continuous transformations of a topological space. The first 3 notions you mention are related to "mixing" in the topological sense. In general, there is no measure given in advance.

In many interesting situations BOTH structures are present: topology and measure. Actually both theories have their origin in Classical mechanics, where we have a smooth transformation of a manifold, which also preserves a measure on this manifold.

In general, if you have a continuous transformation of a topological space, no measure is given. And vice versa, when you have a transformation of a measure space, this space may not be equipped with any topology. (Like in probability).

In topological dynamics, when the measure is not given a priori, but the transformation is chaotic in the topological sense, it is possible to INTRODUCE an invariant measure, related to your transformation, and to bring the methods of ergodic theory to the study of smooth dynamics. Invariant measures exist for continuous transformations of a compact space, but there are very many of them. (This is called the Bogoliubov-Krylov theorem).

One famous way to introduce a "nice" invariant measure, intrinsically related to a smooth transformation is called the SRB-measure (Sinai-Ruelle-Bowen)

On all this, I recommend Sinai's books. They explain the ergodic theory itself, but also its applications to smooth dynamics.

Logically these are two different subjects.

1. Ergodic theory is about transformations of a space equipped with a measure (and the transformation preserves this measure). The measure is given in advance. The last 3 notions you mention are related to this situation, and express various degrees of "mixing".

2. Topological dynamics is about continuous transformations of a topological space. The first 3 notions you mention are related to "mixing" in the topological sense. In general, there is no measure given in advance.

In many interesting situations BOTH structures are present: topology and measure. Actually both theories have their origin in Classical mechanics, where we have a smooth transformation of a manifold, which also preserves a measure on this manifold.

In general, if you have a continuous transformation of a topological space, no measure is given. And vice versa, when you have a transformation of a measure space, this space may not be equipped with any topology. (Like in probability).

In topological dynamics, when the measure is not given a priori, but the transformation is chaotic in the topological sense, it is possible to INTRODUCE an invariant measure, related to your transformation, and to bring the methods of ergodic theory to the study of smooth dynamics. Invariant measures exist for continuous transformations of a compact space, but there are very many of them. (This is called the Bogoliubov-Krylov theorem).

One famous way to introduce a "nice" invariant measure, intrinsically related to a smooth transformation is called the SRB-measure (Sinai-Ruelle-Bowen)

On all this, I recommend Sinai's books. They explain the ergodic theory itself, but also its applications to smooth dynamics.