Timeline for What is entropy, really?
Current License: CC BY-SA 3.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 1 at 8:59 | history | made wiki | Post Made Community Wiki by Stefan Kohl♦ | ||
| Apr 8, 2020 at 13:58 | comment | added | gfdsal | Omg yes! It makes so much sense now. Im glad I found your answer. | |
| Apr 8, 2020 at 1:40 | comment | added | Timothy Chow | @gfdsal : Flip your coin 1000 times, and each time you get tails, draw a black pixel, and each time you get heads, draw a white pixel. Does this help you see how to convert between the night sky and your coin? | |
| Apr 7, 2020 at 23:09 | comment | added | gfdsal | ok so in general rare events require more bits to be encoded then more probable events. Right? Now weird thing is that I understood that in the night-sky example, celestial bodies are more useful and are sparse (low frequency) then empty black pixels. But I am unable to translate that intuition to the coin example of how heads is more informative with 0.2 probability :| | |
| Apr 7, 2020 at 22:45 | comment | added | Timothy Chow | @gfdsal : As for your other question, no, I would not say that more information means less entropy. It's true that in everyday speech, we tend to think of entropy as "disorder" or "randomness," which is the opposite of meaningful content or "information." But that is not how the words are used in information theory. The uniform distribution has the highest entropy; strings of uniformly random bytes tend to have high information, because to tell you exactly what bytes I got, I have to communicate a lot of information to you. I usually can't say, "They're all 0" or something simple like that. | |
| Apr 7, 2020 at 22:33 | comment | added | Timothy Chow | @gfdsal : If you're looking for an intuitive explanation, imagine that someone is sending you a black-and-white photo of the night sky, one pixel at a time, uncompressed. Most of the pixels are black, and a few are white (indicating stars or other celestial objects). Intuitively, the white pixels are giving you more useful information, right? It's the less probable values that are carrying the "interesting information." If you were to compress the image, you could compress long stretches of black pixels into just a few bits because they're not carrying much information. | |
| Apr 7, 2020 at 22:20 | comment | added | gfdsal | +1 Kuddos for best intuitive explanation. I am slightly confused somehow, in a non-uniform distribution, why is low-probability events are regarded as carrying high information? So if we have unbiased coin and chances for head is 0.2 and for tail is 0.8 then why do we say outcome head has more information? Also isnt entropy inverse of information meaning if we have more information then there is less entropy? $$-log_2(0.2)>-log_2(0.8)$$, $$2.32>0.32$$ | |
| May 22, 2019 at 9:28 | comment | added | user36212 | Due to Shannon's theorem, one can make this measure of information content precise: the entropy of a distribution is the expected amount of information you need to transmit in order to tell someone (who knows the distribution and your encoding scheme) a sample from the distribution. This is often a very useful perspective; in particular if you believe your distribution should be close to a product of independent uniform random variables, finding an efficient coding scheme and looking at entropy can often prove this (in a crude form that most of the distributions are close to uniform indept). | |
| Nov 1, 2013 at 0:19 | history | edited | Timothy Chow | CC BY-SA 3.0 |
added 1093 characters in body
|
| Nov 1, 2013 at 0:00 | history | answered | Timothy Chow | CC BY-SA 3.0 |