1
$\begingroup$

Let $X_1, \dots, X_n, Y$ be random variables, each taking values in $\{0,1\}$. Assume that we are interested in estimating, for each $v=(v_1,\dots,v_n)\in \{0,1\}^n$, the probability $$ p(v) = P[Y=1|X_1=v_1,\dots,X_n=v_n]. $$ If we have no information at all about the distribution of random vector $(X_1, \dots, X_n, Y)$, the best we can do is to assume that all $2^{n+1}$ possible values of it are equally likely, which implies that $p(v)=1/2$ for all $v$. The formal justification of this is maximal entropy principle: the uniform distribution in $\{0,1\}^{n+1}$ has the highest Shannon entropy among all distributions with the same support.

Now assume that we know the probability $P[Y=1]$, the conditional probabilities $P[Y=1|X_i]$ for all $i$, and, more generally, for some (but not all) subsets $S=\{i_1,\dots,i_k\} \subset \{1,\dots,n\}$ and $u=(u_1,\dots,u_k)\in \{0,1\}^k$, we know that $$ P[Y=1 | X_{i_1}=u_1, \dots, X_{i_k}=u_k] = p(S,u) $$ where $p(S,u)$ are some given numbers. To estimate $p(v)$ with this information, it would be logical to find the distribution of random vector $(X_1, \dots, X_n, Y)$ consistent with this information but as uncertain as possible otherwise. This can be formalized as finding the maximal entropy distribution under the given set of constraints.

Now, the questions are: is this a well-known approach explained in the literature? If yes, what are the references and the best key words to find the method? Are there explicit formulas and/or efficient algorithms for finding maximal entropy distributions with this type of constraints? I am ok if we impose some extra (natural) assumptions.

Finally, if this maximal entropy approach is difficult to implement, what are the other models for estimating $p(v)$ using the information in the form as above?

$\endgroup$
2
  • $\begingroup$ The jargon for this is the "principle of minimum discrimination information", where instead of maximizing entropy one minimizes the KL divergence from a prior; if the prior is uniform then this recovers the principle of maximum entropy. (In engineering, the jargon is "principle of minimum cross entropy", noting that cross entropy differs from KL-divergence by a constant for these purposes.) There are explicit formulas for this stuff, often obtained via Lagrange multipliers, and there are also some good Python libraries out there for doing numerical calculations. $\endgroup$ Commented Dec 7, 2021 at 14:53
  • $\begingroup$ Searching for “maximum entropy copula” gets a bunch of relevant results, often with hydrological applications: duckduckgo.com/?q=maximum+entropy+copula $\endgroup$
    – user44143
    Commented Dec 7, 2021 at 15:41

0

You must log in to answer this question.