There are two principled information-theoretic approaches to identifying a good regulator for an organism's environment. Within the context of reinforcement learning, both approaches are unsupervised in the sense that organisms don't need an explicit reward function. Instead of maximising a reward function, the organism's objective is to construct a suitable model of their environment for decision-making and planning.

## Empowerment:

The empowerment approach, developed by Daniel Polani and his collaborators consists of approximating the channel capacity in the perception-action loop:

\begin{equation}
C = \max_{p(x)} I(X;Y)
\end{equation}

where $X$ represents the space of possible actions and $Y$ represents the sensory space(i.e. percepts). One practical challenge facing this approach is that it generally requires identifying an environment's state-transition function which is intractable for large and complex state-spaces. That said, scientists at Deep Mind have recently demonstrated that variational approximations of the channel capacity $C$ address certain scalability issues [2].

## Free Energy Principle:

The Free Energy Principle, developed by Karl Friston rests upon the ergodic assumption that self-organising biological agents minimise the expected surprise or the entropy of their sensory states:

\begin{equation}
H(y) = - \int p(y|m) \ln p(y|m) dy = \lim_{T \to \infty} \frac{1}{T} \int_{0}^T - \ln p(y|m) dt
\end{equation}

where $m$ represents the model used by the agent and $y$ represents its sensory input.

Some scientists have pointed out that Friston's theory is a theory of cognitive dissonance minimisation which leaves it vulnerable to the dark room problem. However, scientists at Google Brain and Google DeepMind recently argued that the Free Energy Principle is a theory of niche construction which forces organisms' to find the ecological niche they are most suited for [5].

## Challenges facing a definitive proof:

An important challenge facing any mathematical formalism for good regulators is that the practical consequences of such a formalism in complex environments requires experimentation which escapes the analytical process.

## References:

Christoph Salge, Cornelius Glackin and Daniel Polani. Empowerment — An Introduction. 2013.

Shakir Mohamed and Danilo J. Rezende. Variational Information Maximisation for
Intrinsically Motivated Reinforcement Learning. 2015.

Karl Friston. The free-energy principle: a rough
guide to the brain? Cell press. 2009.

Zekun Sun & Chaz Firestone. The Dark Room Problem. Cell Press. 2020.

Danijar Hafner et al. Action and Perception as Divergence Minimization. 2020.

1more comment