Given a probability distribution $(X,p)$, its entropy is defined as $H=-\sum_{x\in X} p(x)\log p(x)$.

Given a sample of observations $x_n,n=1..N$, one can estimate $p(x)=\frac{\#\{i:x_i=x\}}{N}$ and thus $H$.

Suppose, however, that $X=P(B)$ the power set of set $B$ and the observations are subsets of $B$: $x_n\subset B$ and the number of observations $\#B=k\ll N\ll 2^k=\#P(B)$. E.g., $k=50$ and $N=10^4$.

I.e., there are not enough observations to estimate the probability of every separate subset.

**How would I estimate the entropy of the distribution?**

E.g., one can define a probability distribution on $B$ as

\begin{align} m(b)&=\sum_{x\ni b}p(x) \\ M &= \sum_{b\in B} m(b) \\ p_B(b) &= \frac{m(b)}{M}. \end{align}

**What can be said about entropy of $(X=P(B),p)$ based on the entropy of $(B,p_B)$?**
(the latter is easy to estimate based on the sample).

1more comment