[HN Gopher] What Is Entropy?
___________________________________________________________________
What Is Entropy?
Author : jfantl
Score : 99 points
Date : 2025-04-14 18:32 UTC (4 hours ago)
(HTM) web link (jasonfantl.com)
(TXT) w3m dump (jasonfantl.com)
| IIAOPSW wrote:
| Its the name for the information bits you don't have.
|
| More elaborately, its the number bits needed to fully specify
| something which is known to be in some broad category of state
| but the exact details to calculate it are unknown.
| alganet wrote:
| Nowadays, it seems to be a buzzword to confuse people.
|
| We IT folk should find another word for disorder that increases
| over time, specially when that disorder has human factors (number
| of contributors, number of users, etc). It clearly cannot be
| treated in the same way as in chemistry.
| soulofmischief wrote:
| Maybe you're confused by entropy? It's pretty well established
| in different domains. There are multiple ways to look at the
| same phenomenon, because it's ubiquitous and generalized across
| systems. It comes down to information and uncertainty. The
| article in question does attempt to explain all of this if you
| read it.
| alganet wrote:
| Maybe I am.
|
| The part of thr article on information theory is more about
| mathematics than software. I don't deny there could be some
| generalization there.
|
| The problem I see is that this could slip to measure human
| actions, which are also source of uncertainty, but although
| the words fit, in this particular case I think associating it
| with classical entropy does more harm than good.
|
| https://en.m.wikipedia.org/wiki/Software_rot
|
| Entropy as described in this article (software entropy), to
| me, does not fall under the same generalization. It is a
| looser use of the word. I used it myself several times, but
| now people are buzzwording entropy all around, and I think
| that looser use should be retracted to avoid thinking of
| humans as numbers or particles.
| petsfed wrote:
| When I use it in an IT (or honestly, any non-physics or non-
| physics) context, I typically mean "how many different ways can
| we do it with the same effective outcome?".
|
| To whit, "contract entropy": how many different ways can a
| contractor _technically_ fulfill the terms of the contract, and
| thus get paid? If your contract has high entropy, then there 's
| a high probability that you'll pay your contractor to not
| _actually_ achieve what you wanted.
| bargava wrote:
| Here is a good overview on Entropy [1]
|
| [1] https://arxiv.org/abs/2409.09232
| perihelions wrote:
| Here's the HN thread about that overview on Entropy,
|
| https://news.ycombinator.com/item?id=41037981 ( _" What Is
| Entropy? (johncarlosbaez.wordpress.com)"_ -- 209 comments)
| brummm wrote:
| I love that the author clearly describes why saying entropy
| measures disorder is misleading.
| glial wrote:
| One thing that helped me was the realization that, at least as
| used in the context of information theory, entropy is a property
| of an individual (typically the person receiving a message) and
| NOT purely of the system or message itself.
|
| > entropy quantifies uncertainty
|
| This sums it up. Uncertainty is the property of a person and not
| a system/message. That uncertainty is a function of both a
| person's model of a system/message and their prior observations.
|
| You and I may have different entropies about the content of the
| same message. If we're calculating the entropy of dice rolls
| (where the outcome is the 'message'), and I know the dice are
| loaded but you don't, my entropy will be lower than yours.
| ninetyninenine wrote:
| Not true. The uncertainty of the dice rolls is not controlled
| by you. It is the property of the loaded dice itself.
|
| Here's a better way to put it. If I roll the dice infinite
| times. The uncertainty of the outcome of the dice will become
| evident in the distribution of the outcomes of the dice.
| Whether you or another person is certain or uncertain of this
| does not indicate anything.
|
| Now when you realize this you'll start to think about this
| thing in probability called frequentists vs. bayesian and
| you'll realize that all entropy is, is a consequence of
| probability and that the philosophical debate in probability
| applies to entropy as well because they are one and the same.
|
| I think the word "entropy" confuses people into thinking it's
| some other thing when really it's just probability at work.
| glial wrote:
| I concede that my framing was explicitly Bayesian, but with
| that caveat, it absolutely is true: your uncertainty is a
| function of your knowledge, which is a model of the world,
| but is not equivalent to the world itself.
|
| Suppose I had a coin that only landed on heads. You don't
| know this and you flip the coin. According to your argument,
| for the first flip, your entropy about the outcome of the
| flip is zero. However, you wouldn't be able to tell me which
| way the coin would land, making your entropy nonzero. This is
| a contradiction.
| nyrikki wrote:
| To add to this.
|
| Both the Bayesian vs frequentist interpretations make
| understanding the problem challenging, as both are powerful
| interpretations to find the needle in the haystack, when
| the problem is finding the hay in the haystack.
|
| A better lens is that a recursive binary sequence (coin
| flips) is an _algorithmically_ random sequence if and only
| if it is a Chaitin 's number.[1]
|
| Chaitin's number is normal, which is probably easier
| understood with decimal digits meaning that with any window
| size, over time the distribution, the distribution of 0-9
| will be the same.
|
| This is why HALT [?] open frame [?] system identification
| [?] symbol grounding problems.
|
| Probabilities are very powerful for problems like The
| dining philosophers problem or the Byzantine generals
| problem, they are still grabbing needles every time they
| reach into the hay stack.
|
| Pretty much any _almost all_ statement is a hay in the
| haystack problem. For example _almost all_ real numbers are
| normal, but we have only found a few.
|
| We can construct them, say with .101010101 in base 2
| .123123123123 in base 3 etc...but we can't access them.
|
| Given access to the true reals, you have 0 percent chance
| of picking a computable number, rational, etc... but a 100%
| chance of getting a normal number or 100% chance of getting
| an uncomputable number.
|
| Bayesian vs frequentist interpretations allow us to make
| useful predictions, but they are the map, not the
| territory.
|
| Bayesian iid data and Frequentist iid random variables play
| the exact similar roles Enthalpy, Gibbs free energy,
| statistical entropy, information theory entropy, Shannon
| Entropy etc...
|
| The difference between them is the independent variables
| that they depend on and the needs of the model they are
| serving.
|
| You can also approach the property that people often want
| to communicate when using the term _entropy_ as effective
| measure 0 sets, null cover, martingales, kolmogorov
| complexity, compressibility, set shattering, etc...
|
| As a lens, null cover is most useful in my mind, as a
| random real number should not have any "uncommon"
| properties, or look more like the _normal reals_.
|
| This is very different from statistical methods, or any
| effective usable algorithm/program, which absolutely depend
| on "uncommon" properties.
|
| Which is exactly the hay in the problem of finding the hay
| haystack problem, hay is boring.
|
| [1]https://www.cs.auckland.ac.nz/~cristian/samplepapers/ome
| gast...
| bloppe wrote:
| Probability is subjective though, because macrostates are
| subjective.
|
| The notion of probability relies on the notion of
| repeatability: if you repeat a coin flip infinite times, what
| proportion of outcomes will be heads, etc. But if you
| actually repeated the toss _exactly the same way_ every time,
| say with a finely-tuned coin-flipping machine in a perfectly
| still environment, you would always get the same result.
|
| We say that a regular human flipping a coin is a single
| macrostate that represents infinite microstates (the
| distribution of trajectories and spins you could potentially
| impart on the coin). But who decides that? Some subjective
| observer. Another finely tuned machine could conceivably
| detect the exact trajectory and spin of the coin as it leaves
| your thumb and predict with perfect accuracy what the outcome
| will be. According to that machine, you're not repeating
| anything. You're doing a new thing every time.
| empath75 wrote:
| > If we're calculating the entropy of dice rolls (where the
| outcome is the 'message'), and I know the dice are loaded but
| you don't, my entropy will be lower than yours.
|
| That's got nothing to do with entropy being subjective. If 2
| people are calculating any property and one of them is making a
| false assumption, they'll end up with a different (false)
| conclusion.
| glial wrote:
| Entropy is based on your model of the world and every model,
| being a simplification and an estimate, is false.
| mitthrowaway2 wrote:
| What if I told you the dice were loaded, but I didn't tell
| you which face they were loaded in favor of?
|
| Then you (presumably) assign a uniform probability over one
| true assumption and five false assumptions. Which is the sort
| of situation where subjective entropy seems quite
| appropriate.
| ponty_rick wrote:
| As a software engineer, I learned what entropy was in computer
| science when I changed the way that a function was called which
| caused the system to run out of entropy in production and caused
| an outage. Heh.
| DadBase wrote:
| My old prof taught entropy with marbles in a jar and cream in
| coffee. "Entropy," he said, "is surprise." Then he microwaved the
| coffee until it burst. We understood: the universe favors
| forgetfulness.
| NitroPython wrote:
| Love the article, my mind is bending but in a good way lol
| gozzoo wrote:
| The visualisation is great, the topic is interesting and very
| well explained. Can sombody recomend some other blogs with
| similar type of presentation?
| floxy wrote:
| If you haven't seen it, you'll probably like:
|
| https://ciechanow.ski/archives/
| nihakue wrote:
| I'm not in any way qualified to have a take here, but I have one
| anyway:
|
| My understanding is that entropy is a way of quantifying how many
| different ways a thing could 'actually be' and yet still 'appear
| to be' how it is. So it is largely a result of an observer's
| limited ability to perceive / interrogate the 'true' nature of
| the system in question.
|
| So for example you could observe that a single coin flip is
| heads, and entropy will help you quantify how many different ways
| that could have come to pass. e.g. is it a fair coin, a weighted
| coin, a coin with two head faces, etc. All these possibilities
| increase the entropy of the system. An arrangement _not_ counted
| towards the system's entropy is the arrangement where the coin
| has no heads face, only ever comes up tails, etc.
|
| Related, my intuition about the observation that entropy tends to
| increase is that it's purely a result of more likely things
| happening more often on average.
|
| Would be delighted if anyone wanted to correct either of these
| intuitions.
| fsckboy wrote:
| > _purely a result of_ more likely things _happening more often
| on average_
|
| according to your wording, no. if you have a perfect six sided
| die (or perfect two sided coin), none/neither of the outcomes
| are more likely at any point in time... yet something
| approximating entropy occurs after many repeated trials. what's
| expected to happen is the average thing even though it's never
| the most likely thing to happen.
|
| you want to look at how repeated re-convolution of a function
| with itself always converges on the same gaussian function, no
| matter the shape of the starting function is (as long as it's
| not some pathological case, such as an impulse function... but
| even then, consider the convolution of the impulse function
| with the gaussian)
| russdill wrote:
| This is based on entropy being closely tied to your knowledge
| of the system. It's one of many useful definitions of entropy.
| 867-5309 wrote:
| > 'actually be' and yet still 'appear to be'
|
| _esse quam videri_
| karpathy wrote:
| What I never fully understood is that there is some implicit
| assumption about the dynamics of the system. So what that there
| are more microstates of some macrostate as far as counting is
| concerned? We also have to make assumptions about the dynamics,
| and in particular about some property that encourages mixing.
| tomnicholas1 wrote:
| Yes, that assumption is called the Ergodic Hypothesis, and
| generally justified in undergraduate statistical mechanics
| courses by proving and appealing to Liouville's theorem.
|
| [1] https://en.wikipedia.org/wiki/Ergodic_hypothesis
| vitus wrote:
| It's worth noting that there's more than just ergodicity at
| play, although that's a fundamental requirement. For
| instance, applying the Pauli Exclusion Principle gives rise
| to Fermi-Dirac statistics.
| tomnicholas1 wrote:
| Isn't that more about enumerating the microstates? The
| Pauli exclusion principle just ends up forbidding some of
| the microstates (forbidding a significant fraction of them
| if you're in the low-temperature regime).
| vitus wrote:
| It is about enumerating the microstates, but in a way
| that takes into account how the particles interact with
| each other (aka making assumptions about the dynamics).
|
| If we didn't take into account any interactions, we'd be
| unable to do anything with statistical mechanics beyond
| rederiving the ideal gas law.
| perihelions wrote:
| This doesn't answer anything, but there's a neat system which
| has dynamics that look very much like irreversible mixing,
| except they're not. It's an illusion: the system is in a
| constantly low-entropy state, and the dynamics are reversible.
|
| https://www.youtube.com/watch?v=_dbnH-BBSNo
| TexanFeller wrote:
| I don't see Sean Carroll's musings mentioned yet, so repeating my
| previous comment:
|
| Entropy got a lot more exciting to me after hearing Sean Carroll
| talk about it. He has a foundational/philosophical bent and likes
| to point out that there are competing definitions of entropy set
| on different philosophical foundations, one of them seemingly
| observer dependent: -
| https://youtu.be/x9COqqqsFtc?si=cQkfV5IpLC039Cl5 -
| https://youtu.be/XJ14ZO-e9NY?si=xi8idD5JmQbT5zxN
|
| Leonard Susskind has lots of great talks and books about quantum
| information and calculating the entropy of black holes which led
| to a lot of wild new hypotheses.
|
| Stephen Wolfram gave a long talk about the history of the concept
| of entropy which was pretty good:
| https://www.youtube.com/live/ocOHxPs1LQ0?si=zvQNsj_FEGbTX2R3
| jwilber wrote:
| There's an interactive visual of Entropy here in the Where To
| Partition section (midway thru the article): https://mlu-
| explain.github.io/decision-tree/
| vitus wrote:
| The problem with this explanation (and with many others) is that
| it misses why we should care about "disorder" or "uncertainty",
| whether in information theory or statistical mechanics. Yes, we
| have the arrow of time argument (second law of thermodynamics,
| etc), and entropy breaks time-symmetry. So what?
|
| The article hints very briefly at this with the discussion of an
| unequally-weighted die, and how by encoding the most common
| outcome with a single bit, you can achieve some amount of
| compression. That's a start, and we've now rediscovered the idea
| behind Huffman coding. What information theory tells us is that
| if you consider a sequence of two dice rolls, you can then use
| even fewer bits on average to describe that outcome, and so on;
| as you take your block length to infinity, your average number of
| bits for each roll in the sequence approaches the entropy of the
| source. (This is Shannon's source coding theorem, and while
| entropy plays a far greater role in information theory, this is
| at least a starting point.)
|
| There's something magical about statistical mechanics where
| various quantities (e.g. energy, temperature, pressure) emerge as
| a result of taking partial derivatives of this "partition
| function", and that they turn out to be the same quantities that
| we've known all along (up to a scaling factor -- in my stat mech
| class, I recall using k_B * T for temperature, such that we
| brought everything back to units of energy).
|
| https://en.wikipedia.org/wiki/Partition_function_(statistica...
|
| https://en.wikipedia.org/wiki/Fundamental_thermodynamic_rela...
|
| If you're dealing with a sea of electrons, you might apply the
| Pauli exclusion principle to derive Fermi-Dirac statistics that
| underpins all of semiconductor physics; if instead you're dealing
| with photons which can occupy the same energy state, the same
| statistical principles lead to Bose-Einstein statistics.
|
| Statistical mechanics is ultimately about taking certain
| assumptions about how particles interact with each other, scaling
| up the quantities beyond our ability to model all of the
| individual particles, and applying statistical approximations to
| consider the average behavior of the ensemble. The various forms
| of entropy are building blocks to that end.
| anon84873628 wrote:
| Nitpick in the article conclusion:
|
| >Heat flows from hot to cold because the number of ways in which
| the system can be non-uniform in temperature is much lower than
| the number of ways it can be uniform in temperature ...
|
| Should probably say "thermal energy" instead of "temperature" if
| we want to be really precise with our thermodynamics terms.
| Temperature is not a direct measure of energy, rather it is an
| extensive property describing the relationship between change in
| energy to change in entropy.
___________________________________________________________________
(page generated 2025-04-14 23:00 UTC)