I think it’s not uncommon for arguments in elementary number theory to be “philosophically” information-theoretic in nature. But this is not a very deep fact - at the end of the day it all comes down to the well-known heuristic principle that the events of a “random integer” being divisible by different primes are approximately independent events. It’s making that heuristic notion precise that always requires considerably more subtle arguments involving complex analysis etc, in which information theoretic thinking plays no apparent role. So I‘m reasonably sure the answer to your title question about the prime number theorem is “no”.

Nonetheless, one can do some cute things with this idea. Here is an example I noticed many years ago, related to the standard estimate
$$
\textrm{(*)} \qquad
\sum_{p \le n} \frac{\log p}{p} = \log(n) + O(1)
\qquad (n\to \infty)
$$
(summation over primes) — a series that is very much related to the sum of the reciprocals of the primes and to estimates of $p_n$ which you and others were discussing here. I will give (*) the following information-theoretic interpretation (which will give a rigorous proof of a bound in one direction):

Take a random variable $X$ that is chosen uniformly at random from the numbers $1,\ldots,n$. For each prime $p\le n$, let $V_p$ be the $p$-valuation of X (the exponent of $p$ dividing $X$). A key observation is that knowing all the $V_p$’s, one can recover $X$. That means that there is an inequality of Shannon entropies:
$$
\log_2(n) = H(X)
\le H(\{V_p, p\le n\})
\le \sum_{p\le n} H(V_p),
$$
by well-known properties of the entropy function $H(\cdot)$ (subadditivity and monotonicity with respect to the $\sigma$-field, precisely analogous to the properties of Kolmogorov complexity that Fortnow uses in his notes).

Now, what is the entropy $H(V_p)$ of the random variable $V_p$? It’s the expression
$$
- \sum_{k\ge 0} \operatorname{Pr}(V_p=k)
\log \operatorname{Pr}(V_p=k),
$$
which can be bounded from above by the $k=1$ term
$$
-\operatorname{Pr}(V_p=1)
\log \operatorname{Pr}(V_p=1) =
-\left( \frac{1}{n} \lfloor n/p \rfloor \right)
\log \left( \frac{1}{n} \lfloor n/p \rfloor \right),
$$
plus all the other parts (which I’ll leave as an exercise to show is $O(1)$).
And this last quantity is approximately equal to $\frac{\log p}{p}$ as long as $p$ is smaller in order of magnitude than $n$. Thus after some more simple estimates one gets essentially the “$\ge$” side of (*), together with the added insight that the error term in approximations such as (*) says something about the extent to which divisibility properties of a typical number by different primes are correlated with each other.

As I was saying at the beginning, this is kind of interesting at a philosophical level, but I’m not aware that anyone has found a way to make these sorts of arguments precise enough to prove something as strong as the prime number theorem, or indeed anything stronger than the most elementary sorts of estimates that already have very short and direct proofs.