# Closed form Information measure on a dataset

Given a set of points ( 1-Dimensional), Can we get a measurement of the entropy of the data without calculating the underlying distribution. I mean , Is there a closed form for measuring entropy given a set of data-points. The closed form need not give me exact entropy..but any measure of information content would help.

-
it would probably be helpful if you could give more insight as to what "measures of information content" were acceptable –  Suresh Venkat Feb 23 '10 at 8:48

It's not really well defined. What you need is an estimate of the cdf from which you get an estimate of the density. The "easiest" way to do this is to bin the points into intervals, maintain the counts within each interval and use that to get an estimate. All of this is horribly nonrigorous, but in the limit as the bin sizes go to zero, you will get some kind of estimate.

The problem though is that such estimates are known to not be consistent, even if the underlying distribution is discrete (i.e defined over a discrete support). The above estimate is related to the MLE, and the typical problem is how to assign probabilities to empty bins. There's a wealth of literature on this topic, and at least for the discrete case, you should take a look at Liam Paninski's article:

Paninski, L. (2003). Estimation of entropy and mutual information. Neural Computation 15: 1191-1253.

-

The only other way that I am aware of for calculating entropy requires knowing the number of configurations of the system, i.e. the multiplicity. This is often equivalent to knowing the distribution, but can, in some cases, be found without knowing the distribution, per se. In this case the entropy is simply the log (to your favorite base) of the multiplicity.

-