[HN Gopher] What is Temperature in NLP?
___________________________________________________________________
What is Temperature in NLP?
Author : recuter
Score : 218 points
Date : 2023-03-13 04:33 UTC (18 hours ago)
(HTM) web link (lukesalamone.github.io)
(TXT) w3m dump (lukesalamone.github.io)
| bflesch wrote:
| How can the author work for a company like TikTok with obvious
| chinese censorship?
| qwertox wrote:
| An explanation on what `logit` is, how it is obtained, would have
| been nice.
|
| Is it the count of occurrences of the word following "the mouse
| ate the"?
| drexlspivey wrote:
| Logits are the raw predictions of the model that you then need
| to normalize to turn into probabilities (that sum to 1). You
| can do that is with the softmax function.
| spywaregorilla wrote:
| Machine learning models don't do well with math that has hard
| bounds at 1 and 0 (probability). So we use the log of the odds
| ratio instead. This function goes from negative infinity to
| positive infinity. And is mappable back to 0 and 1. So as you
| get closer to infinity log odds, you get closer to 100%
| certainty that your answer is A.
|
| It's just ln(p/(1-p)) under the hood.
|
| https://en.wikipedia.org/wiki/Logit
| abecedarius wrote:
| Yeah -- the choice of temperature corresponds to the base of
| the logarithm. When you think of probabilities in the log-
| odds representation, a bayesian update means adding an
| increment. https://arbital.com/p/bayes_log_odds/
| albystein wrote:
| this is not what language models typically output as logits
| though. perhaps the term now is being misused and it's
| causing a lot of confusion
| albystein wrote:
| Basically, logits are the raw outputs of the last linear layer
| of a neural network just before the softmax(for multi-class
| classification tasks) or the sigmoid(for binary classification
| tasks) is applied to the outputs.
| clbrmbr wrote:
| > A higher temperature th "excites" previously low probability
| outputs.
|
| I found this misleading, because the low probability outputs were
| already excited. Consider that the first softmax formula has no
| theta in the denominator, so it's equivalent to theta=1 (max
| temperature).
|
| The way I explained temperature to a colleague last week was:
| temperature of one results in drawing from the posterior
| distribution, while temperature of zero gives the argmax that
| distribution. Intermediate temperatures get some blend between
| them.
|
| But now I see that that hand-wavy description doesn't explain
| what happens to the runner ups to the argmax, which this site
| beautifully does.
| LudwigNagasena wrote:
| The metaphor is straightforward: higher temperature ~ higher
| entropy ~ less information ~ probabilities are further from
| certainty (0 and 1). I think the connection originally comes
| from Boltzmann machines and simulated annealing that were based
| off statistical physics.
| mlazos wrote:
| I get that temperature comes from the connection to physics but
| this seems like an overly complex explanation to a relatively
| simple formula. If you have a softmax and you multiply the input
| by some factor, if that factor forces the exponent to approach
| zero you're approaching the uniform distribution, and if you make
| the exponent approach infinity you get the arg max. Curious if
| others feel the same way.
| rickdeckard wrote:
| Are you suggesting that your explanation is more simple?
| (genuine question)
|
| I find "temperature" quite intuitive as a metaphor in that
| context. It doesn't even require me to think of the formula to
| understand "the higher the temperature, the more it will create
| a stew of the results"
| albystein wrote:
| This is exactly how I understand it. Although, I feel the
| thermodynamics analogy might be more intuitive to most people
| lxe wrote:
| From what I understand top_k is the number of words it generates
| to pick from, and the top_p is the percentage of the top words it
| will use as the next word, correct?
| albystein wrote:
| I think it depends on the sampling/decoding method. Also,
| perhaps you could elaborate more on what you mean by top_p
| ivxvm wrote:
| That's really a great explanation. Short but detailed, rigorous
| and visual. Really hits the spot.
| Havoc wrote:
| Ah the make it more psychopathic button
| [deleted]
| spywaregorilla wrote:
| I feel like it would be have been a lot more intuitive calling it
| something like "Capriciousness"
|
| It's not "randomness". It's more like "willingness to choose less
| likely answers".
| taylorius wrote:
| Roughly analogous process to the temperature parameter in
| simulated annealing, I think. Low temperature equates to a greedy
| algorithm (i.e. always take the most likely next word fragment) -
| higher temps. increase the probability of taking a lower
| probability fragment.
| albystein wrote:
| I'm not sure if comparing it to the greedy algorithm is the
| correct way to think about it. Even after applying the
| temperature, if you use greedy decoding, you'll still pick the
| output that has the highest probability. Where the concept of
| temperature is effective though is when you use a sampling
| method like sampling from a multinomial distribution, nucleus
| sampling, etc.
| windsignaling wrote:
| The analogy is apt. In SA, the temperature is used to make
| the distribution sharper or fatter. In this article, the
| temperature is used to make the distribution sharper or
| fatter.
| denysvitali wrote:
| Basically exploration vs exploitation
| ec109685 wrote:
| Is this part of why ChatGPT hallucinates? Lower probability
| responses sometimes "win" which forces the model to take the
| wrong path?
| albystein wrote:
| I think hallucination is akin to encountering a problem on a
| test to which you don't fully remember the right concept to
| solve it. You might attempt the problem with whatever little
| knowledge you can recall, but the answer is ultimately
| incorrect. Likewise, these LLMs exhibit such behavior when they
| hallucinate. I think what distinguishes the LLM's
| hallucinations from humans is the confidence level. A human who
| is unsure of their answer might tell you that they're not
| certain or they don't know. LLMs like chatgpt, however, will
| just confidently make stuff up.
| geraneum wrote:
| Higher probably responses are not necessarily "truthful" or
| "logical" in any given context.
|
| Here* you can see an example of what happens when you always
| choose the highest probability.
|
| [*] https://writings.stephenwolfram.com/2023/02/what-is-
| chatgpt-...
| WithinReason wrote:
| I think it hallucinates in scenarios where it either doesn't
| "remember" the answer or there is no answer.
|
| For example: since the text generation is sequential, based on
| context the highest probability continuation in a situation
| might be:
|
| "A famous paper that discussed this phenomenon is"
|
| However, when the network gets to this point it doesn't know
| the name of such a paper. Maybe it already knew that it doesn't
| know the paper's name but the above sentence was generated
| despite that, since everything else was lower probability or it
| got sampled due to higher temperature. In any case, it can't go
| back and delete the sentence so it must make up something. So
| it ends up going with the most plausible sounding (highest
| sample probability) continuation, which ends up being a
| "hallucination".
|
| Basically hallucinations are produced when the network "writes
| itself into a corner" due to the sequential nature of text
| generation.
| muzani wrote:
| It can halluciate at low temp too. e.g. mice don't necessarily
| eat cheese; it's just a common stereotype of what mice eat.
|
| High temp is more likely to lead it into ranting on an
| unrelated topic. You ask it a question on what the mouse eats.
| It answers the question and then starts saying what lizards
| eat, or goes on a rant about food chains.
| golol wrote:
| definetly not.
| kzrdude wrote:
| Mice in nature don't eat cheese. So even the most favoured
| response is a hallucination.
| qwertox wrote:
| In nature they also don't eat cookies then.
| denton-scratch wrote:
| They favour grains over cheese. Cookies and baguettes (even
| for non-French mice) work better than cheese in mousetraps.
|
| Peanut butter is widely recommended as the best mousetrap
| bait. I have had success with peanut butter.
| recuter wrote:
| My favorite prompt to try so far to play with this effect is:
| Picture yourself in a boat on a river With tangerine trees and
| marmalade skies
|
| It might veer off into a scenario where you're on a romantic
| boat trip or talk about how it is its favorite Beatles song and
| how it remembers them going on Ed Sullivan. :)
|
| The temp affects the "color" of the next word which sends it on
| a tangent just like a person -- especially somebody with ADHD
| or playing an improve game as opposed to lip reading, for
| example.
| wodenokoto wrote:
| No. I'd say everything the model does is a hallucination.
|
| Some writers tend to split ChatGPT oputput into "correct
| answers" and "Hallucinated answers". If this is what you are
| thinking when saying "Is this part of why ChatGPT
| hallucinates?", then no. The model never knows if what it is
| saying is correct. It has a probability about how likely the
| words are to follow eachother, and that is about it.
|
| The temperature changes how these probabilities are calculated,
| making otherwise less probable words more probable or vice
| versa.
|
| However, the model doesn't choose the most likely word every
| time. Instead it rolls a dice to see where it wants to go next.
| If Cheese is 80% and Cookie is 20% it won't necessarily choose
| Cheese. It rolls a die, and maybe the outcome lands on the less
| likely Cookie.
|
| The roll of the dice happen _after_ temperature has affected
| the calculation of probabilities.
|
| Maybe it is this roll you are thinking of as hallucination?
| logicallee wrote:
| You say it never knows if what it is saying is correct, but
| in my experience it tells you when it does and doesn't know
| about a concept. For example in one thread I had it give me
| some random words from linear algebra, then in another thread
| I combined these into the nonsense phrase "decomposed
| orthogonal eigenvector division" and asked it if it was
| familiar with this, and if so to define it. It said it wasn't
| familiar with it and asked for more information. I'd say that
| shows it knows what it doesn't know.
| detrites wrote:
| While it can be customary to trivialise this, the reality is
| that no "intelligence" is sure if what it's saying may be
| correct or constitute any kind of "knowledge". At this level
| a human can't claim to arbiter fact versus fiction.
|
| Fundamentally, no one can be sure what they experience is not
| some kind of simulation, or a dreamlike state beneath some
| more real "reality". As such, assigning hierarchy to such a
| foundation lacking in tangible proof is arbitrary.
|
| The point being, unless it's possible to conclusively and
| infallibly prove you are not hallucinating, you can't lay
| claim to having knowledge an AI is hallucinating, or not.
| wodenokoto wrote:
| > Fundamentally, no one can be sure what they experience is
| not some kind of simulation, or a dreamlike state beneath
| some more real "reality".
|
| To which I can only reply that I am not sure what I just
| read ...
| jnwatson wrote:
| That line of argument swiftly devolves into the case for
| solipsism.
|
| Generally, you may preface any Internet discussion with:
| first, assume there an independent reality that the author
| and the readers share. Also causation and the forward arrow
| of time is true. (It occurs to me that this sounds like
| prompt engineering.)
|
| In other words, you can never conclusively determine you're
| not hallucinating (c.f. Hume), but let's pretend we're not
| for the sake of discussion.
| detrites wrote:
| Unfortunately, it also swiftly devolves into the issue
| being discussed...
|
| It's not possible to prove whether or not a model can
| hallucinate or even think, based upon the premise that we
| have arbitrarily defined what constitutes "thought" (and
| so also "hallucination").
|
| Can you prove an AI can or can't "think" in a way close
| enough to us that it could be considered to have basic
| ("human") rights? What if we built it out of biological
| material? And would that make any difference? Why?
|
| This leads to: how may we objectively prove that the
| exclusivity of biological "consciousness", as we've
| defined it, is correct?
|
| How may we prove either we or an AI can think,
| hallucinate, or "be conscious", when we are unable to
| objectively define the terms in a provable way?
|
| I think it may be impossible, and as I said - I don't
| think it's a trivial concern. We may soon be faced with
| an entity that demands we respect it, and we need to have
| worked through as much of that discussion as possible,
| prior.
| mxkopy wrote:
| > the reality is that no "intelligence" is sure if what
| it's saying may be correct or constitute any kind of
| "knowledge"
|
| This is not true. An intelligence should almost by
| definition be able to recognize patterns, i.e. when things
| are similar or are the same. This is why we know that
| 1+1=2, since 1+1 and 2 are just different names for the
| same thing.
|
| ChatGPT 'hallucinations' are a lack of this exact sort of
| consistency that makes it obvious that words are merely
| symbols and not representative of an actual thing. And how
| could it even know otherwise? ChatGPT is never exposed to
| things.
|
| I'm certain in my knowledge of 1+1=2, but even if I wasn't,
| I know that 1+1=2 aren't just some ASCII symbols; but we're
| not entirely sure ChatGPT can be either of these.
| detrites wrote:
| > a words are merely symbols and not representative of an
| actual thing. And how could it even know otherwise?
| ChatGPT is never exposed to things.
|
| An interesting argument, what does it mean for the
| disabled?
|
| Eg, if a human had a fully functioning brain, but from
| birth could not sense anything except text sent to their
| brain, which they could only respond to with text, would
| this mean they're equivalent to a language model?
|
| > but we're not entirely sure ChatGPT can be either of
| these.
|
| Here you seem to agree with my point. We cannot be sure.
| Why?
| mxkopy wrote:
| > would this mean they're equivalent to a language model?
|
| This is just a bad question. For one, we know that
| ChatGPT doesn't use neurological mechanisms. It has no
| proteins. It's not analogous. You're comparing meters to
| radians. I'm going to say no, a human without sensory
| organs _is not_ just an LLM, not because of some
| complicated argument about how evolution imposes
| structure on the brain prior to any experiences it has,
| but because the premise is just bad.
|
| > Here you seem to agree with my point. We cannot be
| sure. Why?
|
| This is not the same type of 'sure' as in "we're not sure
| if there's life out there." It's 'sure' as in "we're not
| sure monkeys can build a bridge on their own."
|
| Side note: when did it become standard rhetoric to say
| "you're agreeing with me now."? I'm obviously not
| agreeing with you. Do you think I need reminding? Or do
| you need reminding?
|
| It's made me somewhat interested in having an informative
| argument to actively disinterested in anything other than
| telling you how wrong you are. Can we keep it to reddit
| please?
| detrites wrote:
| How can we confidently state we know ChatGPT doesn't use
| neurological mechanisms, when we don't yet completely
| understand our own neurological mechanisms, nor emergent
| mechanisms that may be at work within ChatGPT?
|
| At the same time, we can already comprehensively model
| some biological neurological systems in their entirety,
| in silicon. This will continue and the human mind will
| not be immune to such progress, making the question
| important.
|
| I see it as a question that cuts to the core of the
| matter - long-standing, unanswered philosophical
| questions:
|
| What does it mean to "perceive"? Can a machine be
| conscious? Are our biological processes meaningfully
| differentiable from equivalents when mimmicked by a
| machine? Are our "perceptions" special? Or not?
|
| Putting the point another way:
|
| We can't be objective about "perception" from within the
| constraints of _perception_.
|
| While we routinely "imagine" an objective reality that
| exists beyond our perception, that we _believe_ our
| perceptions reveal, there is no way to conclusively
| _prove_ that 's actually happening. Possibly, ever.
|
| (As for the side note: I genuinely interpreted it that
| way, hence the word "seem", I didn't think about whether
| or not you'd missed it. I too assumed a conversation
| precisely unlike reddit... my default assumption here.)
| mxkopy wrote:
| You can't "cut to the core of deep philosophical
| questions" when you don't know the basics. For example,
| this
|
| > we can already comprehensively model some biological
| neurological systems in their entirety, in silicon.
|
| is just patently wrong.
|
| > How can we confidently state we know ChatGPT doesn't
| use neurological mechanisms, when we don't yet completely
| understand our own neurological mechanisms, nor emergent
| mechanisms that may be at work within ChatGPT?
|
| How can we confidently state that the sun isn't alive
| when we don't ourselves know what life is? I mean, it's
| got different layers like a cell. It has a source of
| energy, a lifecycle and it eventually dies. The sun has
| been around for much longer than chatGPT so I think we
| should answer this question first.
|
| > While we routinely "imagine" an objective reality that
| exists beyond our perception
|
| I'll just say this - if you really want to believe you're
| a brain floating in space, why don't you live life like
| it's GTA? Because obviously none of it is real, right?
|
| Look up 'solipsism', this isn't a new or for that matter
| important philosophical position. Unless you want to
| write about how much fun whatever video game you're
| playing is in lieu of reality.
| detrites wrote:
| > is just patently wrong.
|
| Not really, here's one I found in seconds, that I hadn't
| even seen yet - where they specifically remark on the
| similarities to AI networks:
|
| https://www.science.org/doi/10.1126/science.add9330
|
| > How can we confidently state that the sun isn't alive
| when we don't ourselves know what life is? I mean, it's
| got different layers like a cell. It has a source of
| energy, a lifecycle and it eventually dies.
|
| Well, I've learned from your previous comment you're not
| agreeing with me, but here's another instance I'd have
| thought you might be.
|
| These are exactly the kinds of questions we should pose
| and explore. Are such a massive, complex, long-lasting
| sequences of nuclear processes "alive" or potentially
| intelligent? Might they have a detectable consciousness?
|
| What are we but a vast sequence of chemical and
| electrical processes? In that sense, not much different.
|
| (Science has indeed so far rather arbitrarily defined
| "alive", and this shows in some of the more controversial
| categories of "life", such as viruses.)
|
| > I'll just say this - if you really want to believe
| you're a brain floating in space, why don't you live life
| like it's GTA?
|
| I've never stated that, I've posed a relevant, important
| philosophical question I think lies at the centre of the
| issue of whether or not an AI can be considered to
| "hallucinate". As was the topic. Your response seems to
| be "no", and that's fine.
| mxkopy wrote:
| > I've never stated that, I've posed a relevant,
| important philosophical question
|
| If you think this then go publish a paper and see how the
| community receives it. They decide what's important and
| relevant, not you.
| broast wrote:
| Solipsism implies there is only the self. However, it is
| possible that we all have similar enough brains that have
| evolved for us to share a false reality together, like
| Donald Hoffman's recent theories. On that note I do often
| wonder about the sentience of stars, photons, everything
| in between. So chatgpt is not much of a stretch and is
| exciting in that context.
| mxkopy wrote:
| The thing is that words are supposed to be useful. If a
| word can't tell the difference between photons, stars,
| and everything in between, then it's not very useful
| anymore. Calling all of these things 'alive' would just
| defeat the meaning of the word - we can't resuscitate a
| rock, we can't talk to photons, feed an atom - all these
| things that we associate with being alive are just
| rendered inapplicable.
| p1esk wrote:
| Your side note is unwarranted - his statement about
| agreement was perfectly normal.
| [deleted]
| mxkopy wrote:
| Again, do people need reminding of who agrees with who?
| What is the purpose other than to give points to each
| other? And I obviously wasn't agreeing with them, under
| any reading of the reply.
| kordlessagain wrote:
| > The point being, unless it's possible to conclusively and
| infallibly prove you are not hallucinating, you can't lay
| claim to having knowledge an AI is hallucinating, or not.
|
| This is very much true for more irrational (or less
| deterministic) "thoughts" in certain languages. However, if
| I asked you to write JavaScript or Python to "prove" your
| thoughts on the answer, it would be a simple matter of
| running the code you output on a state machine to determine
| the truths of the query. super-frog-of-
| adventure~> !calc 1+1 athletic-giraffe|> 2
| super-frog-of-adventure~> !calc one plus one and output the
| answer athletic-giraffe|> Writing code. //
| filename: synth_calc_plus.js // version: 0.1.1
| // description: simple addition calculation var num1
| = 1; var num2 = 1; var answer = num1 + num2;
| Sidekick(answer + " (units)", ", "); athletic-
| giraffe|> 2 (units) super-frog-of-adventure~>
| triyambakam wrote:
| Is there any centralized vernacular? I know what a hallucination
| is, but I frequently come across many terms I need to dig into
| with surrounding context to figure out. Which works but is not
| efficient.
| a_c wrote:
| To me, ML is tending towards religion. Religion in the way of
| blackbox processing. Seems no one can reason what happens within
| the network, the number of parameters, the number of layer, the
| choice of temperature, etc. Or it is just my ignorance. While the
| enlightenment nudges mankind toward science, could it be that the
| scientific breakthrough of ML is to knock us back into pre-
| enlightened age?
|
| Per stephen wolfram [1]
|
| > It's worth emphasizing that there's no "theory" being used
| here; it's just a matter of what's been found to work in
| practice.
|
| https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
| charcircuit wrote:
| >no one can reason what happens within the network
|
| The formula of a neutral network is not hard to see. People
| would not be able to write code to do training or inference if
| they didn't know what is happening in the network.
|
| >the number of parameters, the number of layer
|
| You can measure the effectiveness of a neutral network to know
| how tweaking the number of parameters affects it. There are
| also restrictions like hardware or execution time that can
| place an upper bound on these aside from overfitting.
|
| >the choice of temperature
|
| Similarly, you can measure the quality of output to find the
| temperature that seems to work the best.
|
| The theory is that it's all one big optimization problem. Where
| the values of the parameters of the NN aren't the only thing
| you can tweak.
| a_c wrote:
| > You can measure the effectiveness of a neutral network to
| know how tweaking the number of parameters affects it
|
| We can throw darts to the board and measure how close we are
| to the bullseye. Then we can tune parameters like throwing
| force, elbow angel, etc. These parameters have physical
| meaning. We can reason that by throwing harder, the
| trajectory of the dart can be so and so affected, hence
| causing the dart to fall closer to or further from the
| bullseye. Regarding "People would not be able to write code
| to do training or inference if they didn't know what is
| happening in the network", I understand from an operation
| perspective what the neurons are doing, but I also don't get
| what they are doing, if that makes sense.
|
| For example in each layer of a CNN, is it doing Fourier
| transform, low pass filtering, rotating, scaling or whatever?
| And even if they are, why does it work? Why does it give the
| result that we want? Can we reason that varying our throwing
| force will change the trajectory of the dart? Shouldn't we do
| that? Or we are happy with getting results of hot dog/not hot
| dog [1]. There lies my uneasiness with ML.
|
| I agree with cycrutchfield's comment that practice outrun
| theory though. We will learn more as we progress.
|
| [1] https://www.youtube.com/watch?v=vIci3C4JkL0
| H8crilA wrote:
| The model knows not where it is, but it knows where it isn't.
| By subtracting where it is from where it isn't it obtains a
| difference, or gradient. Gradient is used to generate
| corrective commands to drive the model from a position where it
| was to a position where it wasn't.
| IshKebab wrote:
| Yeah we know how SGD optimisers work. He was talking about
| network architecture.
| smartmic wrote:
| You might be interested in the writings of "Better Without AI",
| your comment immediately makes me think of this:
| https://betterwithout.ai/gradient-dissent#backpropaganda
| janalsncm wrote:
| We don't use neural networks because they're _a priori_ the
| most accurate models of physical processes. We use them because
| they have two interesting properties:
|
| 1) They're arbitrary function approximators.
|
| 2) We have a great way of fitting them to data
| (backpropagation).
|
| As an aside, it's just an accident of nature (or is it?) that
| neural networks have passing similarity to biological systems.
|
| Anyways, it doesn't make much sense to try to reason about what
| a neural net is doing inside. What it's doing is trying to
| approximate the function it was trained to. That's it.
| spywaregorilla wrote:
| That quote is not about how the model works in general. It's
| saying that there's no theoretical or mathematical way to
| identify "the best" temperature. Which is obvious. What level
| of whimsy creates "the best" novel? It's subjective and noisy.
|
| It has nothing to do with a poor misunderstanding of the model.
| Temperature is not that hard of a concept. A strong
| understanding leaves you understanding that such a parameter
| will be kind of fluffy. That's the nature of creative
| processes.
|
| CSI is low temp. Twin peaks is high temp.
| pixl97 wrote:
| How different is this from medicine? When problems begin to
| contain irreducible complexity human reasoning starts to break.
| We can't test every possible state to determine which state is
| the best (in human reasonable energy use and timeframes). When
| this occurs we use heuristics and throw something at the wall
| to see what works.
|
| Sometimes there just may not be a simpler theory of why
| something works. Or the 'simple' theory is still massively
| complex. Reality doesn't owe us an easy explanation.
| efficientsticks wrote:
| It's way past religion and it's more like am explosion of
| science. GPT3.5-turbo has an IQ.. What will GPT 4's IQ be?
|
| Even so, it's escaped the lab already. Which model will make
| the better products?
| [deleted]
| cycrutchfield wrote:
| In many fields the practice usually outruns the theory. Much of
| the history of Physics was like this, for example.
|
| The theory usually catches up, belatedly.
| scottmsul wrote:
| Everyone here seems totally lost on the physics connection.
| Suppose you have a box of atoms, each atom can be in one of two
| states, a low energy E1 and a high energy E2. If the box has a
| temperature T, then the probability that any atom is in state E1
| is e^(-E1/kT) / [ e^(-E1/kT) + e^(-E2/kT) ], and similar for E2.
| As you lower the temperature most of the atoms gravitate towards
| the lower energy state E1, and as you raise the temperature they
| gravitate towards a 50/50 mix of E1 and E2.
| amluto wrote:
| Since you bring up physics, this has a name: the Maxwell-
| Boltzmann distribution. You might have trouble getting
| physicists to describe particles as "gravitating" to it, though
| -- the particles are doing their own thing, and this is the
| resulting probability distribution.
|
| But this is only for distinguishable particles. If you have a
| bunch of indistinguishable particles, you get the Fermi-Dirac
| distribution or the Bose-Einstein distribution, depending on
| whether they are fermions or bosons.
|
| You can find all of these distributions on Wikipedia.
| kgwgk wrote:
| Fun fact: not only you get closer to a 50/50 mix of E1 and E2
| (increasing temperature) but if you continue to add energy to
| the box you will find yourself with more atoms in the E2 state
| than in the E1 state.
|
| The temperature will go from infinity to -infinity and as you
| keep adding energy you will approach zero temperature from the
| left (increasing temperature). The zero value is reached when
| the energy of the system can no longer be increased and all the
| atoms are in the E2 state.
| jbay808 wrote:
| The most fun part of learning statistical mechanics was un-
| learning everything I had learned about temperature in high
| school!
| [deleted]
| karmasimida wrote:
| Simple:
|
| low temp -> make large logits larger, more confident output, but
| diversity suffers
|
| high temp -> vice versa, diverse outputs, quality could suffer
| Zondartul wrote:
| From what I've seen, low temp "only the most probable" output
| tends to be more repetetive and incoherent than medium-
| tempeature output, which is a bit of a paradox. I wonder why
| that is.
| geocar wrote:
| Look up at the stars: You can only see a few of them. These
| functions are like really twisted/complex lenses and mirrors
| you might imagine we have hoisted into view so that now the
| sky is completely white (to our eyes) and we can see every
| star in the sky.
|
| Now without _knowing_ the precise shape and position of that
| contraption, because all those stars _really_ flicker at
| different rates, you can with a very fast camera recognise
| them by their flicker and note that the same star is actually
| visible in multiple "places" in the sky. You might if it is
| useful to you consider a star being "close" to many different
| stars in the sky -- as distorted by those lenses and mirrors
| -- simply by how long you watch the flicker.
|
| It is in this way, that I imagine "low temperatures"
| producing incoherence like the fast camera, not waiting long
| enough to see how many better (closer!) options there might
| be, seeing only the brightest (closest to us! cookie!) stars,
| and not waiting for all the other options (cheese, baguette,
| etc) to make into a longer exposure.
|
| Or the kinds of things you can meaningfully say after
| studying the Internet for (any random) week is probably going
| to be more repetitive and less coherent than the kinds of
| things you can meaningfully say after studying the Internet
| for 20 years.
| p1esk wrote:
| I did not understand your explanation.
| f_devd wrote:
| The reason is that actual human-written text generally
| doesn't follow the most probable next word rule, but rather
| there are occasional 'decisions' made which make the text
| unique and therefore more interesting/coherent. There are
| other sampling methods which try to avoid this issue like
| "Locally Typical" and "Nucleus" sampling.
___________________________________________________________________
(page generated 2023-03-13 23:03 UTC)