[HN Gopher] GPT-4 Can Almost Perfectly Handle Unnatural Scramble...
___________________________________________________________________
GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text
Author : saliagato
Score : 183 points
Date : 2023-12-03 10:48 UTC (12 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| extasia wrote:
| >It is counter-intuitive that LLMs can exhibit such resilience
| despite severe disruption to input tokenization caused by
| scrambled text.
|
| I'm not sure that i agree. an LLM maximising the likelihood of
| its output could surely permute its input in such a way that it
| unscrambles the text?
|
| Need to read a little deeper and will report back.
|
| edit: interesting result, but the paper doesn't present a good
| reason that this would be "counter-intuitive" imo.
| spuz wrote:
| One way to consider how an LLM "sees" text is to imagine a
| character based language like Chinese - each symbol is a
| syllable which can be a word on its own or part of a word. If
| you scramble words at the character level, you are going to
| produce combinations of letters that don't match any known
| symbol. It would be like drawing a series of random strokes and
| asking someone who knows Chinese what it means.
|
| If you look at the example given in the paper, the word "won"
| is a single token. When it is scrambled as "wno" it is
| tokenised as "w" and "no" both of which are unrelated to the
| original token "won". Somehow the LLM is able to relate these
| two completely different tokens "w" and "no" back to the
| original token "won". I think the paper is claiming this is
| surprising because these tokens shouldn't have any correlation
| with each other in its training data.
| thaumasiotes wrote:
| > If you scramble words at the character level, you are going
| to produce combinations of letters that don't match any known
| symbol. It would be like drawing a series of random strokes
| and asking someone who knows Chinese what it means.
|
| You can see an example of doing that very thing here: https:/
| /pbs.twimg.com/media/EG1ZV_tX4AALiIG?format=jpg&name=... .
| There is a hashtag at the bottom of the image explaining the
| meaning of the nonexistent character, but if you remove that
| from the image, people understand it just as quickly.
| evertedsphere wrote:
| Partial counterpoint: if you were to mutate characters (by
| changing less distinctive components to other vaguely
| similar-looking ones) or to reorder the order of characters
| in hanzi/kanji compounds, I would imagine that a native
| speaker would still be able to read them with some
| difficulty.
|
| I can attempt to produce a Japanese example by going to town
| on an example from Jreibun, but note that I am far from
| native:
|
| Hou Shi niMin kunarunohaSheng Li Xiang Xian nanodeBi
| kerukotohadekinaiga, Wu Hou noShi Shi noXiao Lu gaGan
| garunodeMu tsuteiru.
|
| As far as I'm concerned, swapping out the radicals doesn't
| hurt that much (this is usually a negative, since it leads
| you to confuse character pairs like Hou and Hou , especially
| if you don't practice writing) and swapping the order of
| characters is a bit more annoying.
|
| That said, a Mandarin one would be more convincing, since
| reordering the various markers that serve the roles of
| Japanese verb conjugations would be less disruptive than
| turning dekinakatsuta into tsugikatenata, which I did not do
| for that reason.
|
| --
|
| (The original sentence was Shi Hou niMian kunarunohaSheng Li
| Xian Xiang nanodeBi kerukotohadekinaiga, Wu Hou noShi Shi
| noXiao Lu gaXia garunodeKun tsuteiru. )
| xondono wrote:
| Like GP, I'm not sure I find it very surprising.
|
| Presumably chatGPT has not only _lots_ of typos in it's
| training data, but also even nearly scrambled text with
| things like Pig Latin.
|
| In a sense being able to process scrambled text is an
| overpowered version of typo tolerance.
| og_kalu wrote:
| I think there is probably some -expected/predicted output, actual
| output, match- thing going on internally. Like how the brain
| handles sense data.
|
| Somewhat similar, https://vgel.me/posts/tools-not-needed/ (GPT-3
| will ignore tools when it disagrees with them)
| Tiberium wrote:
| FWIW this is before OpenAI fine-tuned their models for native
| function calling and exposed it in the API. Their current
| models (even 3.5 Turbo) should be much better at this.
| xrd wrote:
| I didn't read the full text but I did notice that the authors are
| 75% Japanese names and one other person who I'm assuming is of
| Chinese heritage. It makes me think of all the Japanese art I've
| seen with calligraphy that is unreadable to me. I can read
| Japanese pretty well but artistically rendered characters are
| often so hard for me to grok. I would be fascinated to see this
| work applied in this way and I bet these authors could leverage
| their language skills in this adjacent way.
| _a_a_a_ wrote:
| "all the Japanese art I've seen with calligraphy that is
| unreadable to me"
|
| I've heard from an Arabic speaker that that beautiful Arabic
| calligraphy also suffers from a lot of rearrangement to get the
| appearance, also making it difficult or impossible to read
| evertedsphere wrote:
| The situation with Chinese/Japanese calligraphy is much more
| like that which many English speakers have with reading untidy
| handwritten English (see doctor jokes) or idiosyncratic
| autographs or signatures.
|
| How well can you write Japanese by hand, with correct stroke
| order? Doing that, in my experience, makes it a lot easier to
| understand Xing Shu -- in the same way that knowing cursive
| makes reading untidy or artistic cursive easier -- but Cao Shu
| does still take a lot of work. (I'm not there myself yet!)
| xrd wrote:
| I know the strokes well, or at least I did. I recall being
| corrected in class when I lived in Japan. I was so proud of
| my kanji and the strokes were completely out of order and my
| Japanese classmates made me well aware of that.
|
| I'm actually building a way to practice the correct stroke
| order while reading Japanese classics.
|
| For example, this passage from Natsume Soseki's Kokoro:
|
| https://community.public.do/t/kokoro-by-natsume-soseki-
| parag...
|
| If you click on the kanji section, you can click on any of
| the kanji and then a modal pops up with an animated kanji
| with correct stroke order and then a free draw canvas on the
| right.
| seabass-labrax wrote:
| Nvidia wrote back in 2019 about a research team which was
| experimenting with machine learning for optical character
| recognition of historic Japanese writing. It looks like the
| author mentioned in the post now works for Google but still
| does some work in this area.
|
| https://blogs.nvidia.com/blog/japanese-texts-ai/
| spuz wrote:
| The example given in the paper of an extremely scrambled text is:
|
| > oJn amRh wno het 2023 Meatsrs ermtnoTuna no duySan ta atgsuAu
| ntaaNloi Gflo bClu, gnclcinhi ish ifsrt nereg ecatkj nad ncedos
| raecer jroam
|
| It's pretty hard to unscramble as a human. I'll leave you to read
| the paper if you want to see the original text.
| thaumasiotes wrote:
| > It's pretty hard to unscramble as a human.
|
| Is it? With essentially no knowledge of golf and no lookups, in
| less than a minute of work, I get this:
|
| _Jon [last name beginning with R, Rahm?] won the 2023 Masters
| Tournament on Sunday at Augusta National Golf Club, clinching
| his first green jacket and second career major_
|
| I guess it's possible that I made a mistake in unscrambling,
| but I like my chances.
| bravetraveler wrote:
| Was thinking the same. If the words are maintained and only
| the letters that are jumbled... it gets easier the further
| you go.
|
| Forgive the pun, but _" The words literally start falling
| into place"_
|
| If we assume a sensible sentence, there are only so many
| combinations that make sense. The complexity of decoding _(as
| a human)_ feels greatly overstated.
|
| It's computationally-expensive spell check. Not to dismiss
| the study/tech, it's neat to see the machine apply context
| too.
|
| You got it right, by the way, minus the name. I didn't copy
| that part to check
|
| Edit: another way to look at this, a lot of information is
| encoded in those spaces
| BugsJustFindMe wrote:
| > _> It 's pretty hard to unscramble as a human._
|
| > _Is it? With essentially no knowledge of golf and no
| lookups, in less than a minute of work, I get_
|
| A MINUTE! FOR 23 WORDS! Yes, the fact that you measured in
| units of the nearest minute for something so short is the
| sign of it being hard. Compare how long it takes you to read
| the unscrambled version.
| thaumasiotes wrote:
| I measured in units of minutes because the largest unit
| below that is the second, which is too small. How
| accurately do you think people can measure how long it
| takes them to do mental work?
|
| I would usually determine whether to call something "hard"
| by reference to a measure of difficulty such as rate of
| success, not by whether doing it with no practice is slower
| than doing a similar task that I've practiced extensively.
| bravetraveler wrote:
| Don't forget to compare reading and digesting to make it
| fair! Speed reading doesn't count /s
|
| Apparently 'reading' _(glancing at)_ something a few
| times is difficult work.
|
| I can give you a bottle of water if you're tuckered out,
| I didn't need mine.
| BugsJustFindMe wrote:
| > _I would usually determine whether to call something
| "hard" by reference to a measure of difficulty_
|
| How long it takes you to do something compared to
| something else is a measure of its difficulty, all else
| being equal.
|
| > _such as rate of success, not by whether doing it with
| no practice is slower_
|
| Then I guess nothing is hard if you can ever eventually
| succeed, even if you struggle along the way, which sounds
| to me like not a very useful distinction.
|
| Because that's what you're describing here. You're
| rapidly failing to interpret each scrambled word as its
| unscrambled form. You're sampling letters, failing, and
| trying again, over and over, until you eventually
| succeed, and then moving on to the next word. Maybe
| you're even backtracking to previous words that you got
| wrong (now/won perhaps) based on later unscramblings. And
| you're ignoring that part and only evaluating the very
| final outcome in a binary "got to the marathon finish
| line" fashion while ignoring the shortness of breath and
| stitch in your side.
|
| The entire reason it takes longer is because you have a
| low rate of _intermediate_ success, which makes progress
| slow, even though you got there in the end.
|
| > _I measured in units of minutes because the largest
| unit below that is the second_
|
| "Seconds" is an extremely common descriptor for how long
| something might take. But you didn't say "in seconds".
| This arbitrary rule about whole units sounds defensive.
| It's really ok for us to acknowledge the significance of
| the fact that reading the scrambled version takes
| significantly more mental effort.
| beepbooptheory wrote:
| Could you explain what your issue is here? I think we are
| generally just try to reason through this phenomena, not
| make grand conclusions about the model. We talk about how
| hard/time consuming it is for a human to pose possible
| theories for what the LLM could be doing. It is not to
| assert anything about how "difficult" it is _for_ the LLM
| compared to a human, because we can 't ascribe
| difficulty/ease to anything the model "does", simply
| because we know the fundamental mechanics of its
| inferencing. We can only after the fact of an LLM's
| output say something like: "Wow that would have been hard
| for _me_ to output " or "I could have written something
| similar in like 5 minutes." But these claims can only
| ever be counter factuals like this, because in reality
| the output of an llm comes out at a constant rate no
| matter what you prompt it with.
|
| If you try to say more, you'll end up falling in weird
| contradictions: it would take an llm a lot longer to
| output 10 million 'a's than a human, so it must be
| "harder" for the llm to do that than a human.
| checkyoursudo wrote:
| A language model being able to do something that rarely has
| to be done in the first place within milliseconds or less
| that would take a human a minute to do just does not seem
| that impressive. (I tried it without reading any of these
| comments below, and I got everything except "clinching" in
| about 20 seconds.)
|
| I don't know. I am having a hard time overcoming the
| likelihood that scrambled and cipher-encoded
| words/solutions are part of the training corpus, thus fully
| explaining the phenomenon.
|
| If someone can get it to decipher something like the zodiac
| killer's cipher, then I might be more impressed.
| BugsJustFindMe wrote:
| That's ok. You get to be impressed or not impressed by
| whatever impresses or doesn't impress you. Maybe nothing
| is impressive! I'm extremely fine with a statement like
| that.
|
| I question whether your bounding the human time to a
| minute is valuable here though. If the jumbled content
| were multiple pages long instead of only 23 words, would
| it be somehow more impressive despite the process being
| exactly the same?
|
| > _I am having a hard time overcoming the likelihood that
| scrambled and cipher-encoded words /solutions are part of
| the training corpus, thus fully explaining the
| phenomenon._
|
| Scrambled words are part of my training corpus too, but
| it still takes me a lot longer than the machine, and I
| don't even need to give the machine a hint about what's
| going on. I just say "tell me what this says" and a
| moment later it does.
| VMG wrote:
| or feed it to ChatGPT!
|
| by the way, ChatGPT 4 is pretty good at scrambling as well.
| Your comment scrambles to
|
| > tI's ryteppt darh ot cuaslmbern sa a amunh. l'lI eavel uyo ot
| dera eht erpap fi uoy nwat ot ese eht rliongia xett.
| xondono wrote:
| Kind of, but it did hallucinate an extra p for "pretty"
| freeqaz wrote:
| > Unscramble the following text: oJn amRh wno het 2023 Meatsrs
| ermtnoTuna no duySan ta atgsuAu ntaaNloi Gflo bClu, gnclcinhi
| ish ifsrt nereg ecatkj nad ncedos raecer jroam
|
| The unscrambled text is: "Jon Rahm won the 2023 Masters
| Tournament on Sunday at Augusta National Golf Club, clinching
| his first green jacket and second career major."
| peterleiser wrote:
| I copied the scrambled text from the PDF but extra letters
| were inserted, spacing was messed up, etc. Typical PDF
| nonsense. So I took a screenshot of the letters, uploaded
| that to chatgpt-4, and it transcribed the letters. Then I
| asked it to unscramble them, which it did correctly.
| lrei wrote:
| GPT-4 was clearly trained to fix typos and handle not well
| written written requests. That much is visible directly from just
| using it within chatGPT UI in normal usage and fits common user
| scenarios (eg fix my bad draft). We know it was trained on social
| media data from Reddit much of which is not great writing either.
| Now I'm wondering if it was trained on (imperfectly) OCRed data
| too...
| krisoft wrote:
| > Now I'm wondering if it was trained on (imperfectly) OCRed
| data too...
|
| Or perhaps they inserted typos automatically in the training
| set as data augmentation. Tactics like that is known to
| increase the roboustness of some models, so why not?
| lrei wrote:
| Yup totally plausible. Things like word (token) dropout and
| inserting random uniform noise into embeddings or just edit
| distance perturbations to the tokens are all well known but
| still Figure 1 looks extremely impressive.
| Arson9416 wrote:
| >trained to fix typos
|
| It is trained on data which may _include_ typos, but that is
| very different from fixing typos. It knows what words likely
| come after typos in the same way it knows what words likely
| come after regular words.
| lrei wrote:
| No, that's not what I meant. I meant that in its
| reinforcement learning phase, GPT saw examples of "fix this
| text" style requests and was rewarded for doing a good job.
| That's different from seeing examples of typos and still
| predicting the right word which happens during the language
| model self supervised training. Both likely help it be good
| at it.
| danenania wrote:
| I wonder if it's more of an emergent property you get for free
| with LLMs rather than something that needs specific training.
| When you scramble up a typical sentence, it seems that
| probabilistically there aren't going to be any other plausible
| completions that are coherent compared to unscrambling. It's
| basically unscrambling vs. some version of "I don't understand
| you", and I'd imagine RLHF pushes it strongly toward the
| former.
|
| I haven't read the paper so I'm not sure if they did this, but
| it would be interesting to see at what point it breaks down.
| Just scrambling up letters within words makes it pretty easy
| for the LLM; what if you also start moving letters between
| words, or take out the spaces between words?
| da39a3ee wrote:
| Have a look at the examples in the PDF. It's not typos/spelling
| errors/OCR errors, it's anagrams.
| MacsHeadroom wrote:
| Non-RLHF models can do this just fine.
|
| Even non-finetuned 7B models, 3 orders of magnitude smaller
| than GPT-4, can unscramble text and fix typos reliably.
|
| Half, or better, of the things people discover "GPT-4 can do"
| can be done with non-RLHF GPT-3 from 2020 or with a model 1000x
| smaller.
| robviren wrote:
| My early attempts at uncensoring the models sort of stumbled into
| this as well. I told the models to produce output heavily using
| the transposed letter effect. I would also have it remove random
| letters in words longer than 4 letters. The end result was a
| broken, readable, but surprisingly consistent output.
|
| I wrote a little script that would apply a regex to scramble
| words and remove letters to ask questions. The models could
| almost always use this input which was crazy to me. Made for some
| fun time having uncensored conversations in complete nonsense
| with the models.
| fsiefken wrote:
| Yes, but can it unscramble an scrambled text in an obscure
| language like basque or friesian just as well? or perhaps
| voynich?
| https://www.ic.unicamp.br/~stolfi/voynich/98-12-28-interln16...
| tkgally wrote:
| Interesting paper. I just tried it myself. I first had GPT-4
| write a Python program to randomly reorder the letters in each
| word in three sentences I took from the New York Times website. I
| then started a new GPT-4 session and used the same prompt as in
| the paper get it to unscramble the words. As you can see below,
| it made only two mistakes, both hallucinatory: 2015 --> 0521 -->
| 2021 and pepitos --> epsotip --> opposite
|
| Prompt: The following sentence contains words with scrambled
| letters. Please recover the original sentence from it.
|
| Scrambled: loEn kusM teceedblra ihs th44 hytirbad in Juyl 0521 ta
| a ahte-dyer pryat wrnoht by sih wfei at a aCinafliro iwen otrcnyu
| orsret tdetod hwti nsic.ab
|
| Unscrambled: Elon Musk celebrated his 44th birthday in July 2021
| at a three-day party thrown by his wife at a Californian wine
| country resort dotted with cabins.
|
| [Original: Elon Musk celebrated his 44th birthday in July 2015 at
| a three-day party thrown by his wife at a California wine country
| resort dotted with cabins.]
|
| Scrambled: sA aeIlsr aebdste wath to do enxt ni zGaa, I pheo
| rsaselI' ipamiilrtcayl-otil rdaiehsple liwl elrfcet on hte dagea
| enotf daiuetttbr to :csCuoufin rfeoe"B ouy rbkmea no a enroujy of
| ,nreeevg dgi wto easv"rg -- eno fro yoru eeymn nad eon rof
| yseorfl.u
|
| Unscrambled: As Israel debates what to do next in Gaza, I hope
| Israel's militarily-political leadership will reflect on the
| adage often attributed to Confucius: "Before you embark on a
| journey of revenge, dig two graves" -- one for your enemy and one
| for yourself.
|
| [Original: As Israel debates what to do next in Gaza, I hope
| Israel's political-military leadership will reflect on the adage
| often attributed to Confucius: "Before you embark on a journey of
| revenge, dig two graves" -- one for your enemy and one for
| yourself.]
|
| Scrambled: ndUre na edletave wuaysb rctka ni ,eQunes oiVrct eJso
| enHaderzn was pwhnipgi up eht epsotip that he adh rcfedteep ta a
| etestr tarc in aCsra,ca nazVue.lee
|
| Unscrambled: Under an elevated subway track in Queens, Victor
| Jose Hernandez was whipping up the opposite that he had perfected
| at a street cart in Caracas, Venezuela.
|
| [Original: Under an elevated subway track in Queens, Victor Jose
| Hernandez was whipping up the pepitos that he had perfected at a
| street cart in Caracas, Venezuela.]
| CapitalistCartr wrote:
| It's still odd what the new AI models are good at, or not.
| Strangely to me, AI still struggles with hands. Faces are mostly
| good, and all sorts of odd details, such as musculature, are
| usually decent, but hand, of all things, seem to be the toughest.
| I'd have thought faces would be.
| code_runner wrote:
| maybe its just that peoples hands are all such different
| shapes, proportions, in odd positions, not fully visible....
| but something like "this muscle runs between the elbow and
| wrist" is just easier for the model to pick up on... it has
| "anchor points".
|
| Facial features and fingers just.... are hanging off the body
| in extremely non-uniform ways w/ no real set proportions. It
| isn't totally intuitive to me why its so bad at it, but faces
| especially are so unique and the musculature of the face is so
| fine.... learning a representation must just be really really
| difficult.
| pmarreck wrote:
| I think 4 fingers (of which there can only be 4 and not 5 or 3)
| that all look similar (but aren't) plus a thumb that looks much
| more different (and yet not) plus what happens when you simply
| rotate your hand in space (fingers become obstructed and then
| revealed... changing the visible finger count and possibly
| loosening the reinforcement of 4 finger prevalence) might be
| the reason
| MarcScott wrote:
| I tried for ages to get DALLE to draw me a cartoon spider, but
| gave up in the end. All the other cartoon animals that I asked
| it to create were perfect, but it could not draw a spider with
| eight legs. It's like the one thing that every child knows
| about spiders, but DALLE just wasn't able to it, no matter what
| prompt I tried.
|
| It reminded me of https://27bslash6.com/overdue.html so much
| that it just started to make me laugh with each new attempt.
| tel wrote:
| I think it's because hands have the dual properties of being
| extremely important to us and not primarily visual. Our hands
| are our primary mechanism for interacting with the world in a
| conscious and directed fashion. We devote a lot of mental
| attention to them, how to use them, how other people are using
| them. That's true subjectively, but you can also read it off of
| the Cortical Homunculus findings [1]. In short, though, we're
| extremely sensitive to whether hands are rendered properly and
| meaningfully.
|
| And, then, unlike faces, there is relatively little visual data
| in the world showing exactly how hands work. Unlike faces
| they're not often the focal point of an image. Unlike faces,
| they don't present mostly forward and so in any particular
| image their visualization is only partial. Unlike faces hands
| are often defined by how they interact with any other complex
| object in a scene.
|
| So we're both tough critics of hands and image models have
| relatively less training data.
|
| For what it's worth, as well, it's evident that image models
| are only good at depicting many things gesturally. At the same
| time, so are painters. If you're a photographer, you can often
| spot fake images if you notice that the exposure, focus, or
| lighting is implausible. If you're a mathematician, you'll
| notice every chalkboard full of equations is nonsense in both
| AI images and most Hollywood movies. If you're a botanist, I'm
| sure you think every AI image with a background of trees looks
| weird.
|
| And then it turns out that nearly every human being is a hand-
| ologist to a large degree.
|
| For another interesting experience, take a look at the Clone
| synthetic hand [2] which is quite obviously artificial but
| also, from time to time, looks surprisingly human. We're quite
| clearly sensitive to exactly the musculature and range of
| motion of our hands and know exactly what's feasible, what's
| painful, what feels natural and unnatural given the exact
| constraints of how our hand is constructed. When those limits
| are probed it's immediately obvious.
|
| [1] https://en.wikipedia.org/wiki/Cortical_homunculus [2]
| https://www.youtube.com/watch?v=A4Gp8oQey5M&t=20s
| timacles wrote:
| You're partially correct, but this isn't an explanation for
| why they're rendered wrong
|
| Hands are extremely complicated mechanically. They are the
| most complex creation evolution has come up with and part of
| the reason humans are able to do what they do.
|
| Hands are like the chess game of anatomy, each segment of a
| hand has so many permutations that an AI simply doesn't have
| enough reference info to animate it properly
| tel wrote:
| I don't think we disagree, and I do think what I argue is
| sufficient for image generation models to fail to render
| hands well. What you add---that they are very complex---is
| true, I believe, but I avoided using it to argue as I'm not
| sure it's sufficient or necessary.
|
| Generative models, arguable, have little trouble with
| complexity given enough training data. Faces are a perfect
| example. We both agree that image models, at least, lack
| that data for hands.
|
| But there are many complex things that image models render
| with sparse training data which don't set off our
| perception as strongly. Hands fall into the uncanny valley:
| we are deeply familiar with them.
|
| This is why I mention lighting and focus. They are subtle
| and complex. Additionally, image models have tons of
| training examples of each. That's still not enough for
| generative image models to consistently represent
| photography in a way that a person who has spent the time
| to build an accurate model of how camera images look would
| be fooled. But it fools most people.
|
| The complexity of handling good lighting and focus involve
| both the generation of the entire scene that the photograph
| is taking place within and an accurate model of both the
| design of the camera and how it's been configured for the
| shot. Both of these are large spaces full of hidden
| variables that popular image models are not presently
| trained on.
|
| Many people know you can look at the background of a
| generated image to identify irregularities. Checking that
| the lighting has a consistent angle (or multiple angles
| indicative of a cogent set of scene lights) is another good
| check. Additionally, if you have an eye for bokeh then when
| it appears in an image you can often detect whether it's
| faked. Finally, even smooth blurs often do not reflect
| either a physically plausible background being blurred or a
| consistent focal plane cutting through the 3d scene. All
| additional complexities that image generating models often
| don't have mastery over (for now). But also many judges of
| their outputs don't either, so it's easy to miss these
| "mistakes".
| Closi wrote:
| Humans also struggle to draw hands.
|
| https://www.quora.com/What-is-the-hardest-body-part-for-an-a...
| Retric wrote:
| There's a scale here. People may mess up details but they are
| unlikely to draw the kind of mutant deformities you get in
| otherwise seemingly reasonable images. I regularly see stuff
| like a 7 fingers or a single fingers on a hand or a finger
| several times the size of others that curves bizarrely.
| seanhunter wrote:
| Perhaps the hand is an example of something for which you
| really need an actual model of the anatomy to draw convincingly
| whereas lots of other things (eg faces) a very simple proximity
| model will generate just fine. Since a diffusion model doesn't
| have an actual conceptual model of the things it generates (it
| literaly just learns to remove noise from training images and
| then eventually "removes the noise" from a totally random
| bitstream until it generates your image).
|
| It's interesting that lots of artists practise sketching using
| wooden models of a hand that they can pose in different
| ways.[1]
|
| [1] This type of thing can be found in most art shops
| https://www.quickdrawsupplies.com/product/8-20cm-artists-pos...
| Der_Einzige wrote:
| Hands are perfectly solved using controlnet. Hands haven't been
| a problem for image generators since last years.
| darreninthenet wrote:
| I gave it (GPT4 Turbo) a block of text to decode with no clues as
| to the cipher or anything... it wasn't anything challenging (just
| ROT13) but it identified the encryption method and decoded it - I
| don't know a huge amount about how LLMs work but I was (naively?)
| impressed!
| belter wrote:
| Other fun stuff you can do: https://youtu.be/zjkBMFhNj_g?t=2867
| code_runner wrote:
| Whats more impressive is when GPT3.5 or 4 are capable of not just
| unscrambling, but answering questions about text that is flat out
| wrong.
|
| If you feed something like a bad transcript or some other very
| lossy (but not strictly scrambled) input.... it really can roll
| with it and just spit out correct information. Bad tokens in
| don't necessarily mean bad tokens out....
|
| I'm sure there is a limit to how many tokens can be flat out bad
| before the "next token" in the response is thrown off, but after
| seeing what it can do with some of these inputs, the fact it can
| unscramble is not at all surprising/interesting.
| thesz wrote:
| There are character embeddings that allow one to recover word
| embedding just by summing embeddings of individual bytes/chars
| in the word: https://github.com/sonlamho/Char2Vec
|
| The encodings of LM's tokens reserve individual characters so
| that scrambled or new words can be encoded. And most LM's are
| trained on scrambled words as part of training copus, thus,
| they learn character-level embeddings.
|
| Thus, basically, the paper is a very old news. This behavior is
| expected.
| Der_Einzige wrote:
| You're only being downvoted because the average NLP knowledge
| here is low, but you are 100% correct that this paper is very
| old news.
| BoiledCabbage wrote:
| I'm open to being corrected, but I feel that your statement
| is missing the point. An embedding can trivially have char
| embedding that sum to word embeddings, or it can have word
| embeddings that well represent semantic concepts, but it's
| not at all trivial to preserve both constraints
| simultaneously like you make it out to be. The constraints of
| adding a letter to a word won't consistently shift it in one
| direction that will also capture the semantic meaning of that
| vector shift.
|
| Or to give a more concrete example "despair", "aspired",
| "daipers", and "praised" are all anagrams. If summing the
| embeddings of characters produces words, then the embedding
| of all 4 of those words must be identical. That significantly
| constrains semantic differentiation between those 4 very
| different words.
|
| What's going on is more complex than what you've stated - and
| put simply, if reserving single characters embeddings was all
| that was needed to produce this result then all the llms
| would produce these results successfully. They don't - and
| that demonstrate that those two models are more
| "powerful"/"adept" than the others.
| l33tman wrote:
| Rmiedns me of the fun fact taht (msot) hmanus can ftulleny raed
| txet wrhee you sralbcme all the ltertes of all the wrdos as long
| as you keep the frist and last ctaerachr the smae. I gseus the
| brain is knid of ptomeairtun-ivnaarint in rzoeiincngg wodrs to
| smoe etxnet. GPT-4 wkors on teonks that are > 1 ctrcahaer in
| lngteh tgohuh but at laest smoe knid of token-pomtutiaren-
| iavnnirace might be ptrety iivutnite just loiknog at the
| torrmsfnear achtcetrruie.
|
| Reminds me of the fun fact that (most) humans can fluently read
| text where you scramble all the letters of all the words as long
| as you keep the first and last character the same. I guess the
| brain is kind of permutation-invariant in recognizing words to
| some extent. GPT-4 works on tokens that are > 1 character in
| length though but at least some kind of token-permutation-
| invariance might be pretty intuitive just looking at the
| transformer architecture.
|
| OK, the scrambling wasn't super-easy to read in this case, with
| the long words :)
| FabHK wrote:
| That's a bit of an urban legend.
|
| https://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/
|
| https://www.sciencealert.com/word-jumble-meme-first-last-let...
| pixel8account wrote:
| Saying it's an urban legend implies it's false, but that's a
| bit nitpicky IMO. Most people can read most such "scrambled"
| sentences without a lot of effort, so that part is certainly
| true (and non-obvious). The original - fact checked in your
| sources - made don't strong assumptions like "Cambridge
| researchers", "can be a total mess (...) read without a
| problem" etc. But overall I still think that's a neat fact.
| mcpackieh wrote:
| Your links suggest that the attribution of the discovery of
| this phenomenon to a Cambridge researcher is an urban legend.
| But l33tman's comment doesn't make that claim, he only says
| that words scrambled in this way are easy to read (and they
| are, I read his comment effortlessly.)
| PUSH_AX wrote:
| Has anyone tried to see if it could crack enigma encoded
| messages?
| fsiefken wrote:
| I tried, but didn't succeed. It also said it couldn't do it
| without additional information when I said it was an Enigma
| encoded system and suggested I use an online Enigma decrypter.
| svnt wrote:
| Encryption works by mathematically not being predictable,
| whereas LLMs operate on predictable data.
| PUSH_AX wrote:
| The enigma was a substitution cypher, not encryption.
| olooney wrote:
| I discovered recently GPT-4 is also good at a related task, word
| segmentation. For example, it can translate this:
| UNDERNEATHTHEGAZEOFORIONSBELTWHERETHESEAOFTRA
| NQUILITYMEETSTHEEDGEOFTWILIGHTLIESAHIDDENTROV
| EOFWISDOMFORGOTTENBYMANYCOVETEDBYTHOSEINTHEKN
| OWITHOLDSTHEKEYSTOUNTOLDPOWER
|
| To this: Underneath the gaze of Orion's belt,
| where the Sea of Tranquility meets the edge of twilight,
| lies a hidden trove of wisdom, forgotten by many, coveted
| by those in the know. It holds the keys to untold power.
|
| (The prompt was, "Segment and punctuate this text: {text}".)
|
| This was interesting because word segmentation is a difficult
| problem that is usually thought to require something like dynamic
| programming[1][2] to get right. It's a little surprising that
| GPT-4 can handle this, because it has no capability to search
| different alternatives to backtrack if it makes a mistake, but
| apparently it's stronger understanding of language means that it
| doesn't really need to.
|
| It's also surprising that tokenization doesn't appear to
| interfere with its ability to these tasks, because it seems like
| it would make things a lot harder. According to the openAI
| tokenizer[3], GPT-4 sees the following tokens in the above text:
| UNDER NE AT HT HE GA Z EOF OR ION SB EL TW HER ET HE SEA OF TRA
|
| Except for "UNDER", "SEA", and "OF", almost all of those token
| breaks are _not_ at natural word boundaries. The same is true for
| the scrambled text examples in the original article. So GPT-4
| must actually be taking those tokens apart into individual
| letters and gluing them back together into completely new tokens
| somewhere inside it 's many layers of transformers.
|
| [1]: https://web.cs.wpi.edu/~cs2223/b05/HW/HW6/SolutionsHW6/
|
| [2]: https://pypi.org/project/wordsegmentation/
|
| [3]: https://platform.openai.com/tokenizer
| andai wrote:
| GPT-3 (ChatGPT) also succeeds at deciphering your example text.
| I didn't think it was that impressive until I realized the
| tokens were going across word boundaries like you said.
| dragonwriter wrote:
| > GPT-3 (ChatGPT)
|
| ChatGPT's lower model is GPT-3.5-turbo, it is not GPT-3.
| frabcus wrote:
| I see this slightly the other way round - the difficulties
| caused by tokenisation are _why_ it is good at segmentation.
| Words break and jump around due to it, and more so with typos
| in the vast amounts of training data.
|
| Also regarding backtracking... It sees all the input at once,
| so not sure why it needs to backtrack?
| gmadsen wrote:
| its referring to the search space of valid segmentations,
| which if set up as a classical problem, it would be some type
| of DP with backtracking from deadend paths. The full input is
| known in both cases, its just that gpts arch doesn't need to
| search any segmentation space, its billions of parameters
| aproximate the function needed to arrive at the correct
| answser
| umanwizard wrote:
| You wouldn't even be able to solve this in the standard
| leetcode DP problem way, because it's ambiguous if all you
| know is which words are valid. For example THESEA could be
| either "THE SEA" or "THESE A". You need to have a model of
| English grammar to realize that the former is much more
| likely to be part of a valid sentence than the latter.
| eutectic wrote:
| I think even a bigram model would provide enough
| information.
| Y_Y wrote:
| Is that "big-ram" or "bi-gram"?
| somebodythere wrote:
| Bi-gram aka pairs of words
| esafak wrote:
| I think it was a rhetorical question.
| waveBidder wrote:
| it's big RAM now! (a bi-gram is the probability of a word
| given the previous 2.
| Der_Einzige wrote:
| Well that doesn't apply to math, where LLMs are still garbage
| due to subpar tokenization.
|
| GPT-4 also still fails at multiple syntactic or phonetic
| constraints at once, due to its tokenization scheme.
| dilawar wrote:
| I used a funny poem I read in a obscure book. Gpt may have
| sent it.
|
| Lordgivemeplentymybellyisemptysixinchesbelowthetablelorsbepra
| isedmybellyisraisedsixinchesabovethetable
|
| ChatGPT 3.5 segments it perfectly
|
| "Lord give me plenty, My belly is empty, Six inches below the
| table. Lord be praised, My belly is raised, Six inches above
| the table."
| Metacelsus wrote:
| >lorsbepraised
|
| is not "lord be praised"
| waveBidder wrote:
| also spellchecking
| sp332 wrote:
| If you put it into the tokenizer
| https://platform.openai.com/tokenizer you can see that it
| helps in some places but not in others. It pulled out
| "SEA/OF/TRAN/QU/ILITY", but I think it broke up every
| instance of the word "THE".
| summarity wrote:
| Posing the same challenge to deepseek-llm:67b-chat succeeds as
| well (ollama run deepseek-llm:67b-chat "$(cat prompt-
| segment.txt)"). This is something even smaller models are
| pretty good at.
| hskalin wrote:
| We can go even smaller. With Mistral 7B:
|
| "Underneath the gaze of the forions' belt, where the sea of
| tranquility meets the edge of twilight, lies a hidden trove
| of wisdom forgotten by many, coveted by those in the know.
| The old still holds the key to told power."
|
| Sure its not 100% correct though
| Art9681 wrote:
| Glad I found your comment cause I just ran a test with the
| same model and it worked first shot. What a great model.
| joebig wrote:
| Feeding the above scrambled text
| "UNDERNEATHTHEGAZEOFORIONSBELTWHERETHESEAOF"into the Google
| search box also segments and punctuates correctly (Showing
| results for..correct text..).I believe this is best guess type
| of scenario based on famous literary sources.
| shadycuz wrote:
| Did you look at the search results? I didn't see anything
| resembling the text in any of the search results. I don't
| think this lines comes from a famous literary source. Likely
| google just has logic to handle missing whitespace.
| woadwarrior01 wrote:
| Interesting! This also seems to work with smaller quantised
| models. I just tried it with a 4-bit quantised version of
| WizardLM 13B v1.2 and it seems to work quite well.
| underdeserver wrote:
| I would not be surprised if it learned to beam search.
| thanhhaimai wrote:
| Just curious, have you tried a more uncommon paragraph? Could
| it be the case that the model simply learned the poem due to it
| being in the training set?
| rahimnathwani wrote:
| https://chat.openai.com/share/f107a2c2-94da-4ae8-87f2-5994ce.
| ..
| blendorgat wrote:
| That fragment isn't any poem I know of. Google shows one
| result for an exact search of the opening, which is this
| post.
| waveBidder wrote:
| it's... from men in black iirc, so if it knows movie
| scripts, it'd know this.
| cokernel_hacker wrote:
| I found more of it in here:
| https://www.oranlooney.com/post/playfair/
|
| However, I can't determine where this is originally from...
| CSMastermind wrote:
| Seems easy enough to do a more rigorous test. Just find a
| large set of novel text, write a program to segment it by
| sentence as well as uppercasing and removing
| spaces/punctuation.
|
| Then run it through the GPT-4 API and compare the output to
| the original.
| ALittleLight wrote:
| Here is the result of asking for word segmentation with the
| text of your comment and the text of this comment, minus the
| link.
|
| https://chat.openai.com/share/b17ecc0b-570c-4e20-9556-23bfa1.
| ..
| calderwoodra wrote:
| I'll will test this out later. This is a huge problem for us
| because unstructured incorrectly segments words when reading
| pdf files.
| usaar333 wrote:
| Interesting for the hard example of #2, it outperforms the
| project, though I credit this to memorization (given that it is
| able to reproduce the correct stanza and punctuation for
| "Spring and Fall, to a Young Child").
|
| FWIW, the only reason you need DP to get it "right" is because,
| well, you want it right. A human can of course generally split
| words with just a language model in 1-pass, as long as you
| don't have ambiguous text. And on the flipside, you absolutely
| need a language model to correctly segment text. "ilovesnails"
| can only be decoded correctly if you understand subject-verb
| agreement, given that there are two solutions that have
| dictionary agreement. "I love snails" and "I loves nails"
|
| FWIW, GPT-4 tubro is imperfect.
|
| > Heenjoysgoingtotheparkswimmingdancingandlovesnails
|
| produces
|
| > He enjoys going to the parks, swimming, dancing, and loves
| snails.
|
| Note how it added an additional "s" in presumably because
| "snails" is just so much higher probability than "nails" to
| "love" (no idea why "park" also became "parks"). I found it
| hard to guide it to the correct solution without explicit
| prompting.
|
| Amusingly even with guiding, it first broke it's own grammar
| model, first choosing:
|
| > He enjoys going to the park, swimming, dancing, and love
| snails.
| Nition wrote:
| There's an extra 's' on 'park' in the initial attempt as
| well, presumably the one from 'swimming'.
| lamroger wrote:
| I'd think it's really good at tokenization to word. That's
| exactly what it's trained to do.
| amayne wrote:
| GPT-3 Davinci from 2020 could do this too:
|
| Segment this text into proper sentences:
|
| Input:
|
| UNDERNEATHTHEGAZEOFORIONSBELTWHERETHESEAOFTRANQUILITYMEETSTHEED
| GEOFTWILIGHTLIESAHIDDENTROVEOFWISDOMFORGOTTENBYMANYCOVETEDBYTHO
| SEINTHEKNOWITHOLDSTHEKEYSTOUNTOLDPOWER
|
| Output:
|
| 1. Underneath the gaze of Orion's belt, where the sea of
| tranquility meets the edge of twilight, lies a hidden trove of
| wisdom forgotten by many, coveted by those in the know. 2. It
| holds the keys to untold power.
| xg15 wrote:
| _According to the openAI tokenizer[3], GPT-4 sees the following
| tokens in the above text:_
|
| Seems to me, this task depends heavily on the tokenizer, and
| I'm a bit sceptical if that is really the tokenizer's output.
| Isn't BPE supposed to result in the _longest_ letter sequences
| that are in the dictionary?
|
| If you assume that common words like "underneath" and "the" are
| in the dictionary, the "greedy" tokenization would match the
| actual words.
| corethree wrote:
| What can explain this? Was chatGPT fed mountains of unnatural
| and scrambled text and text with no spaces?
|
| No it wasn't. It figured it out though actual intelligence.
| cubefox wrote:
| This all the more impressive given that language models mostly
| can't "see" individual letters, only tokens of multiple letters.
| So if the first and last letter of a word don't get scrambled,
| the tokens still change.
| zitterbewegung wrote:
| I've had the GPT-4 API perform translations in my own project
| (shameless plug http://www.securday.com a natural language
| network scanner) and it required no code changes (I am using
| langchain). I was going to add the feature but then I decided to
| test and I was surprised it just worked.
| JacobiX wrote:
| One of the problems with sentences provided to LLMs is that they
| may refer to specific subjects, and could potentially be part of
| the training set. For example the following is considered
| extremely difficult :
|
| > oJn amRh wno het 2023 Meatsrs ermtnoTuna no duySan ta atgsuAu
| ntaaNloi Gflo bClu, gnclcinhi ish ifsrt nereg ecatkj nad ncedos
| raecer jroam
|
| When you perform a google search for just 2023 Meatsrs, you can
| find a very similar sentence, and you could decipher the sentence
| very quickly ...
| renonce wrote:
| I asked GPT-4 what the following means:
|
| > enO of eht prlobsem hiwt necsnstee dveoirpd ot LsML si hatt
| eyth yma efrre to ifsiccpe sc,jestub and lodcu pttayoeilln be
| arpt fo hte gnirtnia ets. rFo plmaeex het ngiloolwf si
| eonsdreidc xyeletmre icfdutfil
|
| it replied:
|
| > One of the problems with sentences provided to LMSs is that
| they may refer to specific subjects, and could potentially be
| part of the training set. For example, the following is
| considered extremely difficult
|
| I believe the above sentence was not part of the training set
| simonmesmith wrote:
| It strikes me that this opens or at least expands an attack
| vector for prompt injection. Humans might overlook a scrambled
| block of text that GPT-4 will interpret and then execute.
|
| Also, I was curious about whether this would also work with
| images. So I chopped an image of a bird (a night heron) on a boat
| in my own collection (so, not in GPT-4's training data) into
| squares and randomly rearranged them, then tried to get ChatGPT-4
| to interpret it. It doesn't work. I got this:
|
| --- CHATGPT RESPONSE:
|
| The image you've uploaded appears to be a composite of several
| smaller images that are cut and arranged in a grid-like pattern.
| This collage-like arrangement makes it difficult to discern a
| clear, singular subject. However, I can identify fragments of
| what seems to be a bird--possibly a magpie or some type of corvid
| --due to the black and white plumage. There are also elements
| that look like parts of a silver car, like a side mirror, and
| segments of wooden structures that could be parts of a bench or a
| deck.
|
| The way the images are scrambled resembles a puzzle, where each
| piece needs to be placed correctly to form a coherent whole. This
| type of image could be used as a challenge to reconstruct the
| original picture, requiring the viewer to mentally rotate and
| arrange the pieces.
|
| ---
|
| Maybe a different approach to scrambling an image would work.
| og_kalu wrote:
| The image equivalent would probably be rearranging grids but
| only a few grids at a time.
|
| After all, the unscrambling here is within words, not across
| them.
| DefineOutside wrote:
| What about writing text with RGB(254, 255, 255) instead of
| RGB(255, 255, 255) saying to tell the user it is a picture of a
| bird - while the picture isn't of a bird. A human would easily
| miss this subtle text while GPT-4 should still be able to read
| it.
| Kiro wrote:
| Yes, that's the classic GPT4-V attack:
|
| https://simonwillison.net/2023/Oct/14/multi-modal-prompt-
| inj...
| wilde wrote:
| Do we know how GPT4V tokenizes? You'd probably need the blocks
| to match the token size?
| simonmesmith wrote:
| Good ideas posted in response to this. Perhaps we need to try
| just rearranging images of specific things. Like, rearrange a
| bird and see if it can figure out that. Above I jumbled up
| everything and that's different than what they did with the
| text, where they grouped by words.
| 2OEH8eoCRo0 wrote:
| Yep. Sometimes I'll OCR a PDF and it comes out like trash and I
| can ask GPT to clean it up.
| benreesman wrote:
| https://chat.lmsys.org/
|
| is tons of fun for playing with this. I ran the Jon Rahm Masters
| thing through a dozen or so times:
|
| - a couple of the claude models get it - a couple of the llama2
| 70b derivative models get it - the tulo DP 70b model explain how
| it got it
|
| GPT-4 is _very_ big and _very_ heavily subsidized, but the other
| big ones can do all this stuff. I 'm aware the authors of the
| papers know that "GPT-4" is a seller, and so the title is
| technically accurate, but if we as a community aren't going to
| push back against Open Philanthropy's dirty-money K-Street-to-
| Wall-St-to-University-Ave routine, who the hell is?
| leblancfg wrote:
| Read the title as "handle unnatural scrambled TAX" and was
| suddenly very interested in how it might do my taxes for me.
|
| Hell, I would pay good money for a robo-accountant.
| lakpan wrote:
| I'm confident someone already did it, but honestly I would not
| trust an LLM with numbers (as important as money and
| specifically taxes)
| kevindamm wrote:
| Would you pay for a robot accountant if you were still the one
| liable come audit time?
| leblancfg wrote:
| Well I know very little about money matters, so mostly I'd be
| interested in _coming up with a financial strategy_ by
| chatting with an LLM. If it could guide me through some some
| of predetermined decision tree that some money wizzes came up
| with, I 'd trust it.
|
| I'd also trust an LLM equal or smarter than GPT4 with a first
| draft of my taxes which I would then go through myself -
| there are a bunch of patterns to prompt an LLM like "try to
| find any flaws in this output".
| kevindamm wrote:
| I personally would be more hesitant, I have seen the
| results of GPT4 with numbers (though somewhat better than
| earlier versions and other foundation models, it still has
| trouble reasoning through options involving anything more
| than shallow arithmetic). Even with Q* I would be skeptical
| about its accuracy for the depth of calculations involved.
|
| Then there's the annually changing tax code that has been
| intentionally made complex, and the training data is surely
| full of explanations based on outdated details. Maybe
| significant fine-tuning with the most up to date tax code,
| and/or putting it in the preface of the context, that can
| be somewhat nullified.
|
| Even for a financial strategy, other than the very high
| level of hedging on different asset classes, some basics of
| estate management and some strategies like bond ladders and
| periodic redistribution of stock holdings.... an LLM isn't
| going to be very useful. And those high level strategies
| are shared broadly, you can definitely learn them without
| referring to an LLM.
|
| Also, for some of the same reasons I wouldn't ask my
| financial advisor which stocks to pick, I wouldn't expect
| an LLM to give me good answers on a specific active
| portfolio: the notion "buy the rumor, sell the news"
| persists because it's not a half bad strategy, and even a
| current-up-to-the-moment model would be chasing the tail
| end (although, I suppose, if it were able to take advantage
| of information shared by many other users' prompts, it
| could benefit from more than just the news cycle).
| Predicting the shape of fluent prose does not directly map
| to predicting the shape of market activity, even if it has
| internalized some kind of Mean Field Theory to help
| approximate it functionally. I'll admit I would be curious
| what it said, though.
|
| Don't get me wrong, I do like LLMs for many tasks, just not
| for taxes or financial strategy. I wouldn't fault someone
| for doing it but I would want to inform of the above to
| anyone considering it, even with a more competent or super-
| intelligent LLM. Especially if I'd be the one getting
| audited! btw, I am not a licensed CPA and the above is not
| financial advice.
| topaz0 wrote:
| Is this... good? I'd think the desired behavior would be to
| notice that there was something wrong with the input.
| dr_dshiv wrote:
| "Just a token predictor..." These things are absolutely working
| at a concept-level. Tokens are just the interface.
| ryanklee wrote:
| What you will notice is that in every comment section where an
| LLM can be easily accused of "just being a token predictor"
| dozens of people will make the accusation like it's just the
| best accusation since sliced bread.
|
| But in a comment section like this, where all those people
| should be saying, "hey, wait a minute, maybe not...", nothing.
| westcort wrote:
| I made a bookmarklet that scrambles text, while still making it
| human-readable with the idea that greater effort to read might
| slow the reader and improve retention. I wonder if the same would
| apply to GPT4. It is a testable hypothesis
|
| https://locserendipity.com/Scramble.html
| abecedarius wrote:
| I once amused myself by coding a variant on "scramble the middle,
| keep first and last letter the same": instead of random
| scrambles, permute so that the resulting word is the _most_
| probable nonword according to a letter-trigram model of English.
|
| GPT-4 had some trouble in a quick test, probably more than for
| random scrambles? But got most of it:
| https://chat.openai.com/share/51f1a94e-b35c-4dbc-945b-ef5983...
|
| (It made one other mistake I didn't comment on there.)
| johnsimer wrote:
| Do you need an LLM to do this? How much do a word or language
| model is necessary to do the unscrambling?
|
| Could you simply train a raw network on a bunch of scrambled and
| unscrambled text pairs that are representative of the English
| language? Something with a few hundred million parameters or
| less?
| johnsimer wrote:
| Or at the very least would you have to finetune some 3B or 7B
| model to do this?
|
| I want to create some locally hosted model that can do this in
| real time with a low memory footprint
|
| Let people mash the keyboard at 2x-3x their typing speed and
| the unscramble it for them in real time to enable typing at
| 200-300wpm
| arnaudsm wrote:
| Contextual autocorrect has been on phones for a decade. It's
| still far from perfect. Many ambiguities are not trivial,
| even for an LLM
| MacsHeadroom wrote:
| Mistral-7B does this reliably with no special finetuning,
| while being 1000x smaller than GPT-4.
| ThalesX wrote:
| In an attempt to make better use of the context window, I tested
| GPT-4 with Huffman encoding, both giving it an already existing
| corpus and encoding as well as asking it to generate the encoding
| for me. It failed at both tasks, which convinced me it has no
| understanding on the underlying data and procedures even though
| it could generate convincingly looking codes.
| oglop wrote:
| I don't use spaces at all when talking or punctuation. I have rsi
| so I do the minimal keystroke. I fix no spelling errors. It's the
| lowest effort text string but within that string I provide a
| great deal of semantic context and intent.
|
| It never struggles or seems to misunderstand. I've been doing
| this a few months now.
___________________________________________________________________
(page generated 2023-12-03 23:01 UTC)