Say I have a set $X=x_1,x_2,\ldots,x_n$ of random variables, and would like to find a size $k\leq|X|$ subset that contains as much information as possible. This is complicated because the variables may contain redundant information.
It seems, intuitively, that "information" behaves like the size or volume of a set. In particular, I would like to know if some result similar to http://en.wikipedia.org/wiki/Maximum_coverage_problem holds for selecting size $k$ subsets of $X$ that approximately maximize information. If all I can measure is the information content of an ensemble of variables, how good an approximation can I get by, starting from the empty set, always picking the variable that would maximize the information content of my subset ?
I have been told that I need to understand measure theory even to phrase this question correctly, and I apologize if my wording of the question is not rigorous.