Entropy of a measure Let $\mu$ be a probability measure on a set of $n$ elements and let $p_i$ be the measure of the $i$-th element. Its Shannon entropy is defined by
$$
E(\mu)=-\sum_{i=1}^np_i\log(p_i)
$$
with the usual convention that $0\cdot(-\infty)=0$.
The following are two fundamental properties:

Property 1: $E(\mu)$ takes its minimum on the Dirac measures.
Property 2: $E(\mu)$ takes its maximum on the uniform probability measure.

Now, for some application, I am really interested in a possible generalization when $\mu$ is a finitely additive probability measure on the natural numbers.
Question: Is it possible to define a notion of entropy of a finitely additive probability measure on the natural numbers in such a way that it verifies the following properties:


*

*it takes its minimum on the Dirac measures

*it takes its maximum on the finitely additive translation invariant probability measures


Any reference? Idea?
Thanks in advance,
Valerio
 A: I don't know if someone has already defined such entropy and I am not an expert on these things, but (depending what is wanted) I would start with something like
$$E(\mu)=\sup\left\{-\sum_{i=1}^n\mu(A_i)\log(\mu(A_i)) : \mathbb{N} = \bigcup_{i=1}^n A_i, A_i \text{ pairwise disjoint}\right\}.$$
Some properties this entropy would have are


*

*It equals the Shannon entropy for measures that are concentrated on finitely many numbers.

*It gives value $+\infty$ for finitely additive translation invariant probability measures.


A related definition for arbitrary measure spaces $(X,m)$ is the relative Shannon entropy (a.k.a. the Kullback–Leibler divergence)
$$E(\mu) = \int_X \frac{d\mu}{dm} \log\left(\frac{d\mu}{dm}\right)dm,$$
where $\frac{d\mu}{dm}$ is the Radon-Nikodym derivative of $\mu$ w.r.t. $m$. One could fix a finitely additive measure $m$ and try to work with that.
A: In the vein of what Tapio suggested, one place to look for some ideas is the Lott-Villani paper on optimal transport where they discuss various entropies (section 3.2 on p 923 on the version I linked)
In particular they discuss the Shannon entropy as discussed by Tapio. A side note - it is also called the Boltzmann H-functional. (See also definition 3.28 for another possible direction to try)

To answer your comment to Tapio's post:
If you use $m$ as the counting measure, then I believe that Tapio's definition agrees with the "limiting version" of what you wrote in some sense. If $m$ is the counting measure, then any probability measure $\mu$ is absolutely continuous wrt $m$. Then, its clear that $d\mu/dm (x) = \mu(x)$, so plugging this into the Shannon entropy formula from Tapio's post, we get 
$$
\tag{E} Ent(\mu|m) = \sum_{x\in \mathbb{N}} \mu(x) \log\mu(x)
$$
I believe that the signt discrepancy is because Tapio's formula is usually called the Boltzman H-functional, whereas Shannon entropy is usually referred to as the negative of what Tapio wrote there but I am not exactly sure on these semantics. 
Suppose that $\mu(n) = \frac {1}{2^n}$. Then 
$$
Ent(\mu|m) = \sum_{n=1}^\infty \frac{n \log 2}{2^n} \in (0,\infty) 
$$
Perhaps I have misunderstood your question/comment however? The entropy defined above does satisfy the first condition you give, i.e. that it takes its minimum on dirac measures (by jensen's inequality $Ent \geq 0$ and clearly $Ent(\delta_x|m) = 0$. I am a bit confused as to what a finitely additive translation probability measure on the natural numbers is? Does "finitely additive" mean a weaker condition than just "probability measure"? Can you give an example of one?
A: I just wanted to say that topologically, your hand is forced:
Tapio's supremum definition is the way to go.
Give the space $X$ of all maps from $\mathcal{P}(\mathbb{N})$ to $[0,1]$ 
the topology of pointwise convergence.
The space $FAM$ of finitely additive probability measures on 
$\mathbb{N}$ is then a closed subspace of $X$.
Let $FSM$ be the set of finitely supported measures in $FAM$.
Let $FSA$ be the set of finite subalgebras of of $\mathcal{P}(\mathbb{N})$. 
Given $G\in FSA$ and $\mu\in FAM$, it is easy to (definably) choose $\mu_G\in FSM$ that agrees with $\mu$ on $G$: $\mu_G(\{\min(A)\})=\mu(A)$ for every atom $A\in G$. 
In $FAM$, each point $\mu$ has a neighborhood base consisting of
sets of the form $U(\mu,G,\varepsilon)$, which I use to denote
the set of $\nu\in FAM$ that agree with $\mu$ on $G$ up to error $\varepsilon$. 
Therefore, $FSM$ is dense in $FAM$.
The Shannon entropy Ent is continuous on $FSM$, and 
Ent extends uniquely to a continuous map from $FAM$ to $[0,\infty]$
given by Tapio's $Ent(\mu)=\sup\{Ent(\mu_G): G\in FSA\}$ (using my notation). 
To see this, the important step is to check that $Ent(\mu_G)\leq Ent(\mu_H)$ if $G\subseteq H$.
