[HN Gopher] Shannon Entropy Imposes Fundamental Limits on Commun...
       ___________________________________________________________________
        
       Shannon Entropy Imposes Fundamental Limits on Communication
        
       Author : harscoat
       Score  : 19 points
       Date   : 2022-09-07 20:07 UTC (2 hours ago)
        
 (HTM) web link (www.quantamagazine.org)
 (TXT) w3m dump (www.quantamagazine.org)
        
       | _wldu wrote:
       | One example I like to use (when talking about entropy):
       | 
       | A four digit numeric PIN (that we know) has 0 bits of entropy.
       | There is no uncertainty about what the PIN actually is. A
       | randomly selected one (that we do not know) has just over 13
       | bits.
       | 
       | print(math.log(10)/math.log(2)*4)
       | 
       | 13.28771237954945
       | 
       | The more entropy, the more uncertain we are.
       | 
       | However, humans are not random. We use the year we were born,
       | some keyboard sequence or some other predictable number as our
       | PIN. We don't know exactly how much entropy these PINS have
       | (there is some degree of uncertainty), but we do know they are
       | significantly less than 13 bits.
        
       | acjohnson55 wrote:
       | The information theory class I took in grad school was one of the
       | most mindbending classes I've ever taken. It made doing gnarly
       | math problems fun.
       | 
       | Claude Shannon was a titan. He's also known for publishing what
       | some call the most important masters thesis of all time,
       | https://en.wikipedia.org/wiki/A_Symbolic_Analysis_of_Relay_a....
        
       | bob1029 wrote:
       | Information theory is so cool. I feel like having a solid grasp
       | of the fundamentals is critical for achieving mastery in software
       | development. Closely related would be digital signal processing
       | and frequency domain techniques, both also extremely useful in
       | many branches of software.
       | 
       | The actual information content of a thing has always surprised
       | me, especially when factoring in slightly imperfect (but
       | passable) representations such as JPEG or MP3. To lose some
       | information seems entirely acceptable in many cases.
        
       | mpalmer wrote:
       | Information theory is fascinating to anyone who likes the mathy
       | underpinnings of seemingly non-mathy things, and it's
       | _everywhere_. Cool stuff.
       | 
       | Speaking of information and decompression, my brain initially
       | interpreted the title as a breaking news headline about a female
       | supervillain who's also a theory nerd.
       | 
       | It was only the added context of the domain name that fixed the
       | error.
        
       | rkp8000 wrote:
       | For anyone interested in these ideas, I'd highly recommend
       | reading Shannon's original 1948 paper (linked in the article).
       | While technical and precise it is actually quite digestible for
       | someone with a bit of a quantitative background.
        
         | ivan_ah wrote:
         | link to paper abstract:
         | https://ieeexplore.ieee.org/abstract/document/6773024
         | 
         | direct link to PDF:
         | https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=677...
        
         | prophesi wrote:
         | I'd also highly recommend watching The Bit Player documentary.
         | Not as info-dense as an article or paper, but it's a fun ride.
        
       ___________________________________________________________________
       (page generated 2022-09-07 23:01 UTC)