Given a set of points ( 1-Dimensional), Can we get a measurement of the entropy of the data without calculating the underlying distribution. I mean , Is there a closed form for measuring entropy given a set of data-points. The closed form need not give me exact entropy..but any measure of information content would help.
1
-
$\begingroup$ it would probably be helpful if you could give more insight as to what "measures of information content" were acceptable $\endgroup$– Suresh VenkatCommented Feb 23, 2010 at 8:48
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
It's not really well defined. What you need is an estimate of the cdf from which you get an estimate of the density. The "easiest" way to do this is to bin the points into intervals, maintain the counts within each interval and use that to get an estimate. All of this is horribly nonrigorous, but in the limit as the bin sizes go to zero, you will get some kind of estimate.
The problem though is that such estimates are known to not be consistent, even if the underlying distribution is discrete (i.e defined over a discrete support). The above estimate is related to the MLE, and the typical problem is how to assign probabilities to empty bins. There's a wealth of literature on this topic, and at least for the discrete case, you should take a look at Liam Paninski's article:
Paninski, L. (2003). Estimation of entropy and mutual information. Neural Computation 15: 1191-1253.
answered Feb 23, 2010 at 8:34
$\begingroup$
$\endgroup$
The only other way that I am aware of for calculating entropy requires knowing the number of configurations of the system, i.e. the multiplicity. This is often equivalent to knowing the distribution, but can, in some cases, be found without knowing the distribution, per se. In this case the entropy is simply the log (to your favorite base) of the multiplicity.
$\begingroup$
$\endgroup$
I'm not sure I entirely understand the statement of the question, but it sounds to me as though it's similar to the question, "If you have access to a time series of observations/data coming from some dynamical system, but know nothing about the underlying system itself, what properties of that system can you determine based solely on your observations?"
Some of the key words relating to this question are "correlation dimension" and "correlation entropy", which originate in work of Grassberger and Procaccia ("Measuring the Strangeness of Strange Attractors", Physica D: Nonlinear Phenomena 9, 1983, 189‒208). These let you determine certain properties of the system (analogous to Hausdorff dimension of the attractor or Kolmogorov-Sinai entropy) based only on the time series, without knowing anything about the underlying dynamics, which seems not unrelated to your original question.
answered May 18, 2010 at 18:10
$\begingroup$
$\endgroup$
there are quite a few methods for estimating the entropy without estimating the underlying distribution. this is particularly important for data that are not independent, since then you run out of data pretty quickly. some discussion and a lot of references are given here: http://pages.cs.aueb.gr/users/yiannisk/PAPERS/neuro.pdf yiannis
