[HN Gopher] Show HN: Tiny Diffusion - A character-level text dif...
___________________________________________________________________
Show HN: Tiny Diffusion - A character-level text diffusion model
from scratch
This is a character-level language diffusion model for text
generation. The model is a modified version of Nanochat's GPT
implementation and is trained on Tiny Shakespeare! It is only 10.7
million parameters, so you can try it out locally.
Author : nathan-barry
Score : 152 points
Date : 2025-11-10 15:13 UTC (5 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| yugretcx wrote:
| Why do these text diffusion demos always look like the number of
| allowed tokens is fixed for a specific unfilled region?
|
| Is this the case?
|
| Ie. if the region only has four tokens(here characters) but
| calculates the best word is "forget" does it just abandon the
| best fit or truncate it to fit?
|
| Are there text diffusion models with lax infill directives?
| rand0mwalk wrote:
| Tokens start as a special [MASK] token. Then as the diffusion
| process runs they are "unmasked" i.e. sampled.
|
| So yes, you define a sequence of [MASK] tokens with some length
| ahead of time.
|
| In practice, if a model wants to write a shorter sequence,
| it'll just fill the remaining tokens with empty content. If it
| wants to write a longer sequence, you'll have to identify this
| and extend the sequence with more [MASK] tokens. This is
| typically obvious since there's no "end of sequence" token
| present if the model wants to generate more.
| nathan-barry wrote:
| Yes, this is the case. During training, the model will get a
| sequence of text (ex, 512 tokens long) with a percentage of
| them masked out (with a special <MASK> token). It learns how to
| unmask those tokens to construct the original text.
|
| In the case that you mentioned, if we had 4 <MASK> tokens in a
| row, all we are doing for decoding is predicting what those 4
| tokens should be.
|
| Generally, this does not seem to be a significant problem, as
| there are usually multiple ways to express an idea in varying
| lengths. Also, with confidence-aware parallel decoding, it can
| usually avoid the scenario you mentioned, as focusing on
| decoding the highest confident tokens will generally avoid such
| scenarios with a well trained model.
| simonw wrote:
| This is really neat.
|
| I noticed the diffusion-process.py demo was using matplotlib in a
| window, but I figured it would be cute if it used a terminal UI
| instead - so I had Claude Code convert it to use curses. Code and
| demo GIF here:
| https://gist.github.com/simonw/9033ebd8dd17b4c0ad101ddda7a54...
| Majromax wrote:
| The basic MLP block in this model uses a ReLU^2 activation
| function (x <- ReLU(x)^2). That seems to be copied from the
| nanochat project, and it's not present in nanoGPT. Is there some
| documentation on the choice of this activation function?
| throwaway2027 wrote:
| Isn't it because ReLU is cheap and ^2 is squared loss?
| kouteiheika wrote:
| When it comes to compute cost the choice of activation
| function makes little difference nowadays (and it can often
| be fused with whatever operation comes before it, which makes
| it effectively free).
|
| The real reason is simple: it was inherited.
|
| The relu^2 was used in the nanogpt speedrun[1] because it
| produced the best empirical results, then Andrej based his
| nanochat on the nanogpt speedrun without changing the
| activation function, and then this project was based on
| nanochat.
|
| [1] -- https://github.com/KellerJordan/modded-nanogpt
| macleginn wrote:
| There has been some experimentation with the use of ReLU^2 in
| language models in recent years, e.g., here: https://proceedi
| ngs.neurips.cc/paper_files/paper/2021/file/2...
| mlmonkey wrote:
| I'm curious: has there been any work done on generating embedding
| vectors instead of discrete tokens via diffusion? What would that
| look like? Please point me to some references. Thanks!
| volodia wrote:
| There is also this one that was released in October:
| https://github.com/kuleshov/char-mdlm
| embedding-shape wrote:
| Fun project, easy to understand and nice looking results,
| everything one could ask for! I played around with it locally,
| did some optimizations of low hanging fruits without making it
| much more complicated, and was gonna send over a PR. But then I
| noticed there is no license attached to the project. What are
| your plans regarding the licensing for this?
| nathan-barry wrote:
| Hey, I'll add the MIT licenses later today!
| tell_me_whai wrote:
| Looks fun, thanks for sharing. I see you're implementing game of
| life sampling, what's the reasoning for using this logic?
| gdiamos wrote:
| One year later and there is still no inference engine for
| diffusion LLMs
|
| Students looking for a project to break into AI - please!
| nathan-barry wrote:
| Actually NVIDIA made one earlier this year, check out their
| Fast-dLLM paper
| gdiamos wrote:
| Thanks I'll check it out!
| gdiamos wrote:
| Did I miss something? https://github.com/NVlabs/Fast-
| dLLM/blob/main/llada/chat.py
|
| That's inference code, but where is the high perf web
| server?
| tough wrote:
| training inspired on nanochat for diffusion models:
| https://github.com/ZHZisZZ/dllm
|
| now someone needs to make it work with vllm or something
| doppelgunner wrote:
| This is impressive. Can it run on mobile?
___________________________________________________________________
(page generated 2025-11-15 23:01 UTC)