[HN Gopher] Searching for DeepSeek's glitch tokens
___________________________________________________________________
Searching for DeepSeek's glitch tokens
Author : arithmoquine
Score : 192 points
Date : 2025-01-25 20:19 UTC (1 days ago)
(HTM) web link (outsidetext.substack.com)
(TXT) w3m dump (outsidetext.substack.com)
| amluto wrote:
| > The most obvious thing differentiating DeepSeek's tokenizer
| from other's is a substantial fraction of the training data being
| Chinese. This makes things much more difficult to work with --
| tokenizations are learned at the byte level, but UTF-8 Chinese
| characters are usually several bytes long.
|
| I realize that these models are more than powerful enough to deal
| with this nonsense, but it seems like, especially for smaller
| models, it might make sense to try using the Unicode input as
| such instead of treating it as bytes.
| pama wrote:
| Not sure what you mean here--care to elaborate? The eventual
| input to these models are integer token IDs (128k different
| ones for DeepSeek). The tokenizers do the conversions from
| Unicode streams to streams of token IDs.
| cchance wrote:
| I still wonder why we're training models on all these
| languages especially when they have different alphabets etc,
| we've got solid translators, wouldn't it be more parameter
| dense to target one language for all data and tokens, and
| then have a layer specifically for input and output
| translation?
| eightysixfour wrote:
| I'd be interested to know if adding more languages makes
| them more or less performant. It is my understanding that
| you have to add code for the models to perform well, for
| example.
| ijustlovemath wrote:
| more languages gives deeper semantic understanding; I think
| it only helps with diversity of data, which ultimately
| improves outputs
| sva_ wrote:
| You'll have a lower bound on the quality of the translator
| you're using.
|
| There's an idea that you can generalize concepts among
| different languages, and that you'll benefit from the
| extended training corpus. As in, talking about an idea from
| different perspectives helps the model carve it out. But I
| don't have anything concrete to back that claim up.
| econ wrote:
| Each language has unique tools. If you have a word for
| something or even better a whole set of words the
| conversation works a lot better than in a language that
| has nothing of the kind. English elaborately talks about
| all kinds of Communities. Dutch does have a word like it
| but it is almost never used. Or, how should we talk about
| the kind of snow if we have only one word?
| https://watchingtheswedes.com/2018/02/28/50-words-for-
| snow/
| cma wrote:
| More languages helps it at novel translation tasks, models
| have been tested with languages not in/barely in the corpus
| and a translation book in context and were able to do an ok
| job. You'll also have things like mulimodal where you want
| to preserve all the tonality and emphasis in the input
| language.
| astrange wrote:
| Bitter lesson says any kind of specialization is not worth
| it[0]. Also, you want to be able to have mixed language
| conversations, like defining a Chinese word in English.
|
| [0] but it might be worth it if you need a smaller model
| because then there are tradeoffs again.
| amluto wrote:
| From the OP, it sounds like those tokens are generated from
| the UTF-8 _bytes_ instead of from the Unicode code points.
| And those bytes are, taken in isolation, complete nonsense.
| Imagine a token that represented the right side of the letter
| d followed by the left side of the letter e but could also
| represent other mishmashes of characters.
|
| I bet the first layer of the model is mostly stuck
| reconstructing something resembling actual words.
|
| (UTF-8 is locally decidable. I bet that a bit of work on the
| token list could cause it to avoid tokens that do not align
| with code point boundaries.)
| pama wrote:
| To be clear these tokenizers use byte-pair encoding
| (subword tokens) so an individual token index typically
| corresponds to a piece of a word; this index does not
| depend on any intermediate decoding of the byte stream as
| long as the start of the stream is a start of your input.
| The decoding always works left to right and always starts
| at the start of the stream. You could write a tokenizer
| that uses plain bytes and one that uses unicode code points
| if your tokenizer was trained on unicode and forced to keep
| unicode codes together (almost all are), and the results
| would be identical for all practical purposes.
| mmoskal wrote:
| llama2, llama3, gpt3, and gpt4 tokenizers are all trained
| on UTF8 bytes and all include invalid (partial) UTF8
| tokens. For llama3 it's only 256 tokens, one for each
| byte, but for the others it's more interesting (eg., 1361
| tokens with UTF8 fragments in llama3).
|
| There is over 1M possible Unicode code points, and 150k
| actually defined. Thus, you can't really encode all of
| them with splitting.
| pama wrote:
| Right. I stand corrected and it makes sense. I may have
| misunderstood the OP. It would not make sense to encode
| each unicode point as a token and then start training the
| subword tokenizer on top unless we go to the vocabularies
| of many millions.
| yorwba wrote:
| There might be better ways to split than simply using
| bytes, though. Normalizing Unicode to NFD form and
| replacing CJK characters with their Ideographic
| Description Sequences gets me down to slightly more than
| 50k codepoints, and visual inspection indicates that the
| IDS library I used is missing data for quite a few
| characters, so maybe 40k or so is possible.
|
| Then you could have the "how many r in strawberry"
| equivalent of "how many Yue in Ming Yue Qing Feng "! On
| the negative side, a model trained on such a
| representation could make up CJK characters not in
| Unicode and you would need a procedural font to display
| them properly.
| mmoskal wrote:
| You essentially have to run a byte regular expression that
| enforces valid UTF8. When you take into account exclusion
| for surrogate pairs and overlongs, you end up with about 14
| states in the corresponding automaton.
|
| This is one thing among many done by our llguidance [0]
| library.
|
| [0] https://github.com/microsoft/llguidance
|
| edit: if anyone's interested:
|
| (([C2-DF] [80-BF]) | (E0 [A0-BF] [80-BF]) | ([E1-EC]
| [80-BF] [80-BF]) | (ED [80-9F] [80-BF]) | ([EE-EF] [80-BF]
| [80-BF]) | (F0 [90-BF] [80-BF] [80-BF]) | ([F1-F3] [80-BF]
| [80-BF] [80-BF]) | (F4 [80-8F] [80-BF] [80-BF]) | [00-7F])
| pama wrote:
| Cool repo--thanks!
| singularity2001 wrote:
| probably something as byte latent tokenization, or get rid of
| organization altogether as kaparthy suggested
| brookst wrote:
| "Tokenizations are learned at the byte level" seems wrong.
| Tokens are integer representations of one or more characters,
| which themselves can be multiple bytes.
|
| When you tokenize "ant" to 0x38 0xF9, it doesn't matter if the
| original was three bytes of ascii or 0x00 0x00 0x00 0x61 0x00
| 0x00 0x00 0x6E 0x00 0x00 0x00 0x74
| mmoskal wrote:
| Tokens are in fact sequences of _bytes_ not characters. For
| example, llama3 tokenizer (128k tokens) includes 1361 tokens
| that are invalid UTF8 (or rather they are partial UTF8).
|
| Models will generally only produce valid UTF8 (that is when
| bytes of tokens are concatenated they are valid UTF8), unless
| really confused.
| petters wrote:
| They are, but they should not be
| bhuztez wrote:
| Being a fanboy of Universal(Tong Yi ) Token(Wen Zi ), I think
| Chinese is the most easy one to work with. Since Chinese has no
| characters, it just have a few thousand tokens. Unicode code
| point is good starting point for Chinese.
|
| What about English? Just as there is no natural boundary
| between tokens in English, there is no natural boundary between
| words in Chinese. Before LLM became popular, people had
| invented many ways to do Chinese word segmentation, just like
| nowadays people are inventing many ways to do tokenization.
|
| However in the past, most of the time, you would end up with
| ngrams. If we learn that from history, ngrams should be a good
| starting point for English. For example, word "token" should be
| 3 tokens, "tok", "oke", "ken". Once add Chinese, everything
| should be just fine.
|
| To be more controversial, I would say there is no such a
| language called Chinese. They are a group of languages who
| adopted Universal Token. Now it is time for English to jump on
| the bandwagon.
| petters wrote:
| I completely agree! This is an oversight that should be fixed
| bn-l wrote:
| Could this be used to poison scrapers that don't respect robots?
| minimaxir wrote:
| In order to do that, you would need a) _massive_ amount of spam
| of a glitch token and b) no LLM developer to notice and
| sanitize it.
| HeatrayEnjoyer wrote:
| Hey it's easier than establishing an entire business selling
| secretly explosive pagers!
|
| Makes you ponder what's coming in the next high effort
| nation-state scheme.
| singularity2001 wrote:
| I think OP means now that the glitch tokens are known if one
| can use them in the second run for the next version to
| disturb it
| robertclaus wrote:
| This was really interesting to me as someone who knows a bit
| about LLMs, but not a ton.
| anonymousiam wrote:
| I saw no attempts to make DeepSeek regurgitate content that is
| unspeakable in China, such as May 35th, Winnie The Pooh, etc.
|
| Such content seems ripe for glitch exploration.
|
| https://en.wiktionary.org/wiki/May_35th
|
| https://en.wikipedia.org/wiki/Censorship_of_Winnie-the-Pooh_...
| whoknowsidont wrote:
| No it doesn't? That's not how glitch tokens work.
| None4U wrote:
| I doubt any of those are short enough to have their own tokens
| spacecadet wrote:
| Just used the glitch token attack in a CTF a week ago. The paper
| is worth a read and there is a repo out there as well that makes
| the attack straight forward- but implementing it yourself is also
| something worth doing.
|
| I will add that the author thinking no one had done this with
| deepseek is unlikely, I run this against models every week out of
| curiosity or for work, not deepseek yet- but considering the
| adversarial ML community is pretty packed, someone likely had and
| just didn't write about it.
|
| https://arxiv.org/abs/2404.09894 https://arxiv.org/pdf/2410.15052
| https://github.com/wooozihui/GlitchMiner
| godelski wrote:
| > DeepSeek censors its own response in realtime as soon as Xi
| Jinping is mentioned
|
| https://x.com/wongmjane/status/1882881778974937524
|
| This censorship is pretty interesting. Reading the post it also
| makes me wonder, are different censors provided depending on
| input language? Different models served to different regions?
| This can also get complicated due to the stochastic nature of
| model output, though the linked tweet appears to be post
| generation filtering. It's much harder to determine generation
| based filtering especially if done in subtle ways like just
| reducing the probability.
|
| I don't think this behavior is just limited to Chinese based
| models fwiw. A lack of transparency makes this space difficult to
| navigate. Though maybe the saving grace is that filtering is very
| hard, even meaning that it is hard to entirely remove certain
| subjects from pretraining data. (Have fun going through 10s of
| trillions of tokens)
| brookst wrote:
| I believe the censorship is at the semantic level, not the
| token level. Same way RL allows training model responses
| independent of eventual input/output languages.
|
| I'm sure the goal is to remove stuff in pre training, but it is
| sufficient to RL it away. Same way OpenAI models doubtlessly
| have training data relating to bio weapons or pedophilia, but
| it is pretty effectively suppressed via RL.
| hntemp787609084 wrote:
| Only played with DeepSeek-R1-Distill-Qwen-14B, but the
| knowledge is definitely still there.
|
| https://pastebin.com/H2UTdi78
|
| Seems more than happy to talk about Tienanmen, Xi, etc.
| starting at line 170 with the very primitive method of
| wrapping the query in its own "<think>...</think>" syntax
| even though it's the user role. Uyghurs are more strictly
| forbidden as a topic, as are its actual system prompts. None
| of this is serious jailbreaking, it was just interesting to
| see where and when it drew lines and that it switched to
| simplified Chinese at the end of the last scenario.
| geewee wrote:
| That was an incredibly interesting read, thank you for
| sharing!
| isoprophlex wrote:
| That was intense, well done!
| joshstrange wrote:
| Incredibly fascinating to read through. I don't follow
| jailbreaking closely so maybe the tricks you used are well-
| known (I've seen 1-2 of them before I think) but I really
| enjoyed seeing how you tricked it. The user-written
| "<think>" blocks were genius as was stopping execution part
| way so you could inject stuff the LLM "thought" it said.
| dangoodmanUT wrote:
| How do you extract the possible tokens from the model weights?
| janalsncm wrote:
| Model weights contain a mapping between logit index and the
| string representation of that token.
| singularity2001 wrote:
| at OP: another one posted the link to the tokens on githup
| two hours ago it's not part of the model but of the pre-
| processing
| ofou wrote:
| Where can I get the actual tokenizer data?
|
| Nevermind, it's here
|
| https://api-docs.deepseek.com/quick_start/token_usage
| singularity2001 wrote:
| Tokenization is one of the reminders that we are far from
| reaching the optimal architecture, But something akin to the
| recent byte latent tokenization gives hope
___________________________________________________________________
(page generated 2025-01-26 23:01 UTC)