[HN Gopher] Show HN: I invented a new generative model and got a...
___________________________________________________________________
Show HN: I invented a new generative model and got accepted to ICLR
I invented Discrete Distribution Networks, a novel generative model
with simple principles and unique properties, and the paper has
been accepted to ICLR2025! Modeling data distribution is
challenging; DDN adopts a simple yet fundamentally different
approach compared to mainstream generative models (Diffusion, GAN,
VAE, autoregressive model): 1. The model generates multiple
outputs simultaneously in a single forward pass, rather than just
one output. 2. It uses these multiple outputs to approximate the
target distribution of the training data. 3. These outputs together
represent a discrete distribution. This is why we named it
"Discrete Distribution Networks". Every generative model has its
unique properties, and DDN is no exception. Here, we highlight
three characteristics of DDN: - Zero-Shot Conditional Generation
(ZSCG). - One-dimensional discrete latent representation organized
in a tree structure. - Fully end-to-end differentiable. Reviews
from ICLR: > I find the method novel and elegant. The novelty is
very strong, and this should not be overlooked. This is a whole new
method, very different from any of the existing generative models.
> This is a very good paper that can open a door to new directions
in generative modeling.
Author : diyer22
Score : 475 points
Date : 2025-10-10 09:01 UTC (13 hours ago)
(HTM) web link (discrete-distribution-networks.github.io)
(TXT) w3m dump (discrete-distribution-networks.github.io)
| VoidWhisperer wrote:
| I don't have a super deep understanding of the underlying
| algorithms involved, but going off the demo and that page, is
| this mainly a model for image related tasks, or could it also be
| trained to do things like what GPT/Claude/etc does (chat
| conversations)?
| booli wrote:
| The posts mentions that: https://github.com/Discrete-
| Distribution-Networks/Discrete-D...
| diyer22 wrote:
| Yes, it's absolutely possible--just like how diffusion LLMs
| work, we can do the same with DDN LLMs.
|
| I made an initial attempt to combine [DDN with
| GPT](https://github.com/Discrete-Distribution-
| Networks/Discrete-D...), aiming to remove tokenizers and let
| LLMs directly model binary strings. In each forward pass, the
| model adaptively adjusts the byte length of generated content
| based on generation difficulty (naturally supporting
| speculative sampling).
| vintermann wrote:
| This is what I find most impressive, that it's a natural
| hierarchial method which seems so general, yet is actually
| quite competitive. I feel like the machine learning community
| has been looking for that for a long time. Non-generative
| uses (like hierarchial embeddings, maybe? Making Dewey's
| decimal like embeddings for anything!) are even more
| exciting.
| diyer22 wrote:
| Exactly! The paragraph on _Efficient Data Compression
| Capability_ in the original paper also highlights:
|
| > To our knowledge, Taiji-DDN is the first generative model
| capable of directly transforming data into a semantically
| meaningful binary string which represents a leaf node on a
| balanced binary tree.
|
| This property excites me just as much.
| f_devd wrote:
| Pretty interesting architecture, seems very easy to debug, but as
| a downside you effectively discard K-1 computations at each layer
| since it's using a sampler rather than a MoE-style router.
|
| The best way I can summarize it is a Mixture-of-Experts combined
| with an 'x0-target' latent diffusion model. The main innovation
| is the guided sampler (rather than router) & split-and-prune
| optimizer; making it easier to train.
| yorwba wrote:
| Since the sampling probability is 1/K independent of the input,
| you don't need to compute K different intermediate outputs at
| each layer during inference, you can instead decide ahead of
| time which of the outputs you want to use and only compute that
| one.
|
| (This is mentioned in Q1 in the "Common Questions About DDN"
| section at the bottom.)
| kevmo314 wrote:
| This is a very clever insight, nice work!
| crondee wrote:
| you dont get to do that for conditional generation though.
| When we have a target then we have to generate multiple, pick
| closest to target, and discard the rest.
| ActivePattern wrote:
| I don't think you've understood the paper.
|
| - There are no experts. The outputs are approximating random
| samples from the distribution.
|
| - There is no latent diffusion going on. It's using
| convolutions similar to a GAN.
|
| - At inference time, you select ahead-of-time the sample index,
| so you don't discard any computations.
| diyer22 wrote:
| I agree with @ActivePattern and thank you for your help in
| answering.
|
| Supplement for @f_devd:
|
| During training, the K outputs share the stem feature from
| the NN blocks, so generating the K outputs costs only a small
| amount of extra computation. After L2-distance sampling,
| discarding the other K-1 outputs therefore incurs a
| negligible cost and is not comparable to discarding K-1 MoE
| experts (which would be very expensive).
| p1esk wrote:
| How does it compare to state of the art models? Does it scale?
| diyer22 wrote:
| The first version of DDN was developed in less than three
| months, almost entirely by one person. Consequently, the
| experiments were preliminary and the results far from SoTA.
|
| The current goal in research is scaling up. Here are some
| thoughts in blog about future directions:
| https://github.com/Discrete-Distribution-Networks/Discrete-D...
| aDyslecticCrow wrote:
| It has other advantages and properties to diffusion models. I
| doubt it will generate "art" anytime soon better than
| diffusion... But it's zero-shot and relative* shallow structure
| could make it amazing at edge compute or image/data analysis
| (as another comment discusses) at limited datasets or compute.
|
| Even one of the examples is a very effective re-colorized that
| beat other approaches I've seen with less risk of modifying the
| subject. It's clever, and simple.
|
| it's compared more with GAN in the article than Diffusion, and
| that excites me. GAN are badly behaved, but are really powerful
| reinforcement learners. If this method can compensate for the
| greatest bane of GAN (mode collapse), it can be very useful.
| serf wrote:
| isn't this kind of like an 80% vq-vae?
| diyer22 wrote:
| No, DDN and VQ-VAE are clearly different.
|
| Similarities: - Both map data to a discrete latent space.
|
| Differences: - VQ-VAE needs an external prior over code indices
| (e.g. PixelCNN or a hierarchical prior) to model distribution.
| DDN builds its own hierarchical discrete distribution and can
| even act as the prior for a VQ-VAE-like system. - DDN's K
| outputs are features that change with the input; VQ-VAE's
| codebook is a set of independent parameters (embeddings) that
| remain fixed regardless of the input. - VQ-VAE produces a 2-D
| grid of code indices; DDN yields a 1-D/tree-structured latent.
| - VQ-VAE needs Straight-Through Estimator. - DDN supports zero-
| shot conditional generation.
|
| So I'd call them complementary rather than "80 % the same."
| (See the paper's "Connections to VQ-VAE.")
| FitchApps wrote:
| Can you train this model to detect objects (e.g detect a fish in
| the picture)?
| diyer22 wrote:
| I believe DDN is exceptionally well-suited to the "generative
| models for discriminative tasks" paradigm for object detection.
|
| Much like DiffusionDet, which applies diffusion models to
| detection, DDN can adopt the same philosophy. I expect DDN to
| offer several advantages over diffusion-based approaches: -
| Single forward pass to obtain results, no iterative denoising
| required. - If multiple samples are needed (e.g., for
| uncertainty estimation), DDN can directly produce multiple
| outputs in one forward pass. - Easy to impose constraints
| during generation due to DDN's Zero-Shot Conditional Generation
| capability. - DDN supports more efficient end-to-end
| optimization, thus more suitable for integration with
| discriminative models and reinforcement learning.
| porridgeraisin wrote:
| Yep, the mental model I have from a cursory read of the paper
| is "generative decision tree".
| nvr219 wrote:
| Congrats!! Very cool.
| curtistyr wrote:
| I've been thinking about this too--how different DDN is from
| other generative models. The idea of generating multiple
| outputs at once in a single pass sounds like it could really
| speed things up, especially for tasks where you need a bunch of
| samples quickly. I'm curious how this compares to something
| like GANs, which can also generate multiple samples but often
| struggle with mode collapse.
|
| The zero-shot conditional generation part is wild. Most methods
| rely on gradients or fine-tuning, so I wonder what makes DDN
| tick there. Maybe the tree structure of the latent space helps
| navigate to specific conditions without needing retraining?
| Also, I'm intrigued by the 1D discrete representation--how does
| that even work in practice? Does it make the model more
| interpretable?
|
| The Split-and-Prune optimizer sounds new--I'd love to see how
| it performs against Adam or SGD on similar tasks. And the fact
| that it's fully differentiable end-to-end is a big plus for
| training stability.
|
| I also wonder about scalability--can this handle high-res
| images without blowing up computationally? The hierarchical
| approach seems promising, but I'm not sure how it holds up when
| moving from simple distributions to something complex like
| natural images.
|
| Overall though, this feels like one of those papers that could
| really shift the direction of generative models. Excited to dig
| into the code and see what kind of results people get with it!
| diyer22 wrote:
| Thank you very much for your interest.
|
| 1. The comparison with GANs and the issue of mode collapse
| are addressed in Q2 at the end of the blog:
| https://github.com/Discrete-Distribution-
| Networks/Discrete-D...
|
| 2. Regarding scalability, please see "Future Research
| Directions" in the same blog: https://github.com/Discrete-
| Distribution-Networks/Discrete-D...
|
| 3. Answers or relevant explanations to any other questions
| can be found directly in the original paper
| (https://arxiv.org/abs/2401.00036), so I won't restate them
| here.
| Der_Einzige wrote:
| Wtf, iclr reviews are happening right now. Did you get accepted
| into a workshop? How do you know it's been accepted?
| albertzeyer wrote:
| ICLR 2026 reviews are happening now (or soon). This paper here
| was accepted at ICLR 2025.
| moconnor wrote:
| Super cool, I spent a lot of time playing with representation
| learning back in the day and the grids of MNIST digits took me
| right back :)
|
| A genuinely interesting and novel approach, I'm very curious how
| it will perform when scaled up and applied to non-image domains!
| Where's the best place to follow your work?
| diyer22 wrote:
| Thank you for your appreciation. I will update the future work
| on both GitHub and Twitter.
|
| https://github.com/DIYer22 https://x.com/diyerxx
| GaggiX wrote:
| It's so cool to see the hierarchical generation of the model, on
| their Github page they have one with L=4: https://discrete-
| distribution-networks.github.io/img/tree-la...
|
| The one shown on their page is L=3.
| BrokenCogs wrote:
| This is a great figure
| gurtinator wrote:
| How did this get accepted without any baseline comparisons? They
| should have compared this to VQ-VAE, diffusion inpainting and a
| lot more.
| diyer22 wrote:
| I believe it is the novelty. Here I would like to quote
| Reviewer r4YK's original words:
|
| > Many high rated papers would have been done by someone else
| if their authors never published them or were rejected.
| However, if this paper is not published, it is not likely that
| anyone would come up with this approach. This is real
| publication value. I am reminding again the original diffusion
| paper from 2015 (Sohl-Dickstein) that was almost not noticed
| for 5 years. Had it not been published, would we have had the
| amazing generative models we have today?
|
| Cite from:
| https://openreview.net/forum?id=xNsIfzlefG¬eId=Dl4bXmujh1
|
| Besides, we compared DDN with other approaches in the Table 1
| of original paper, including VQ-VAE.
| michaeldoron wrote:
| Very impressive to see a single author paper in ICLR, especially
| for an innovative method. Well done!
| Lerc wrote:
| It's not often you read a title like that and expect it to pan
| out, but from a quick browse, it looks pretty good.
|
| Now I just need a time-turner.
| CuriouslyC wrote:
| Pretty interesting. I was just doing research on diffusion using
| symbolic transform matrices to try and parallelize a deep graph
| reactive system a few days ago, seems to be a general direction
| that people are going, I wouldn't be surprised to see diffusion
| adjacent models take over for codegen in the next year or two.
| highd wrote:
| Do you have any details on the experiment procedures? E.g.
| hardware, training time, loss curves? It is difficult to
| confidently reproduce research without at least some of these
| details.
| diyer22 wrote:
| We provide the source code and weights along with a Docker
| environment to facilitate reproducing the experimental results.
| The original paper's EXPERIMENTS section mentions the hardware
| configuration (8x RTX 2080 Ti).
| Zacharias030 wrote:
| Impressive setup :)
| kaiokendev wrote:
| very interesting stuff! great work and congratulations on the
| ICLR acceptance!
| intalentive wrote:
| I built something similar in structure, if not in method, using a
| hierarchy of cross attention and learned queries, made sparse by
| applying L1 to the attention matrices.
|
| Discrete hierarchical representations are super cool. The pattern
| of activations across layers amounts to a "parse tree" for each
| input. You have effectively compressed the image into a short
| sequence of integers.
| 0xdeadbeefbabe wrote:
| The part about pruning and selecting sounds similar to genetic
| algorithms from before the popularity of nn.
| diyer22 wrote:
| That's right! The second paragraph on _OPTIMIZATION WITH SPLIT-
| AND-PRUNE_ in the original paper:
|
| > Inspired by the theory of *evolution and genetic algorithms*,
| we propose the Split-and-Prune algorithm to address the above
| issues, as outlined in algorithm 1.
| cs702 wrote:
| This looks like great work.
|
| I've added it to my reading list.
|
| Thank you for sharing it on HN.
| nothrowaways wrote:
| Impressive, congrats.
| qazxcvbnm wrote:
| An uninformed question: If the network is fully composed of 1x1
| convolutions, doesn't that mean no information mixing between
| pixels occur? Would that not imply that each pixel is independent
| of each other? How can that not lead to incoherent results?
| diyer22 wrote:
| In DDN, 1x1 convolutions are used only in the output layers of
| the Discrete Distribution Layer (DDL). The NN blocks between
| DDLs, which supply the fundamental computational power and
| parameter count, adopt standard 3x3 convolutions.
| randomNumber7 wrote:
| Was there a specific reason for this choice?
| gwern wrote:
| This apparently doesn't apply here, but in fact, pixels _can_
| be generated independently of each other. There are
| architectures where you can generate an arbitrary pixel or
| element of the image without generating the others; they are
| just implicit. See NeRFs or 'single-pixel GANs' or MAEs: eg
| https://arxiv.org/abs/2003.08934
| https://arxiv.org/abs/2011.13775
| https://arxiv.org/abs/2401.14391
|
| Why is this possible? I tend to think of it as reflecting the
| ability to 'memorize' all possible data, and the independent
| generation is just when you 'remember' a specific part of a
| memory. The latent space is a Platonic object which doesn't
| change, so why should your generative process for materializing
| any specific point in the latent space have to? It's not
| surprising if you could generate arbitrary points from a
| function like 'y = mx + b' without generating every other
| point, right? It's just an atemporal mathematical object.
| Similarly with 'generating images from a random seed'. They too
| are just (complicated) functions mapping one number to another
| number.
|
| (You might wonder if this is limited to images? It is not. In
| fact, you can generate even natural language like this to some
| degree: https://github.com/ethan-w-roland/AUNN based on my
| proposal for taking the 'independent generation' idea to a
| pathological extreme: https://gwern.net/aunn )
| cellis wrote:
| Could this be used to train a text -> audio model? I'm thinking
| of an architecture that uses RVQ. Would RVQ still be necessary?
| elchananHaas wrote:
| First, I think this is really cool. Its great to see novel
| generative architectures.
|
| Here are my thoughts on the statistics behind this. First, let D
| be the data sample. Start with the expectation of -Log[P(D)]
| (standard generative model objective).
|
| We then condition on the model output at step N.
|
| - Expectation of Log[Sum over model outputs at step N{P(D | model
| output at step N) * P(model output at step N)}]
|
| Now use Jensen's inequality to transform this to
|
| <= - expectation of Sum over model outputs at step N{Log[P(D |
| model output at step N) * P(model output at step N)]}
|
| Apply Log product to sum rule
|
| = - expectation of Sum over model outputs at step N {Log(P(D |
| model output at step N)) + Log(P(model output at step N))}
|
| If we assume there is some normally distributed noise we can
| transform the first term into the standard L2 objective.
|
| = - expectation of Sum over model outputs at step N {L2
| distance(D, model output at step N) + Log(P(model output at step
| N))}
|
| Apply linearity of expectation
|
| = Sum over model outputs at step N [expectation of{L2 distance(D,
| model output at step N)}] - Sum over model outputs at step N
| [expectation of {Log(P(model output at step N))}]
|
| and the summations can be replaced with sampling
|
| = expectation of {L2 distance(D model output at step N)} -
| expectation of {Log(P(model output at step N))}]
|
| Now, focusing on just the - expectation of Log(P(sampled model
| output at step N)) term.
|
| = - expectation of Log[P(model output at step N)]
|
| and condition on the prior step to get
|
| = - expectation of Log[Sum over possible samples at N-1 of
| (P(sample output at step N| sample at step N - 1) * P(sample at
| step N - 1))]
|
| Now, for each P(sample at step T | sample at step T - 1) this is
| approximately equal to 1/K. This is enforced by the Split-and-
| Prune operations which try to keep each output sampled at roughly
| equal frequencies.
|
| So this is approximately equal to
|
| [?] - expectation of Log[Sum over possible samples at N-1 of (1/K
| * P(possible sample at step N - 1))]
|
| And you get an upper bound by only considering the actual sample.
|
| <= -Log[1/K * expectation of P(actual sample at step N - 1))]
|
| And applying some log rules you get
|
| = Log(K) - expectation of Log[P(sample at step N - 1)]
|
| Now, you have (approximately) expectation of -Log[P(sample at
| step N)] <= Log(K) - expectation of Log[P(sample at step N - 1)].
| You can repeatedly apply this transformation until step 0 to get
|
| (approximately) expectation of -Log[P(sample at step N)] <= N *
| Log(K) - expectation of Log[P(sample at step 0)]
|
| and WLOG assume that expectation of P(sample at step 0) is 1 to
| get
|
| expectation of -Log[P(sample at step N)] <= N * Log(K)
|
| Plugging this back into the main objective, we get (assuming the
| Split-and-Prune is perfect)
|
| expectation of -Log[P(D)] <= expectation of {L2 distance(D,
| sampled model output at step N)} + N * Log(K)
|
| And this makes sense. You are providing the model with an
| additional Log_2(K) bits of information every time you perform an
| argmin operation, so in total you have provided the model with N
| * Log_2(K) bits for information. However, this is constant so you
| can ignore it from the gradient based optimizer.
|
| So, given this analysis my conclusions are:
|
| 1) The Split-and-Merge is a load-bearing component of the
| architecture with regards to its statistical correctness. I'm not
| entirely sure about how this fits with the gradient based
| optimizer. Is it working with the gradient based optimizer,
| fighting the gradient based optimizer, or somewhere in the
| middle? I think the answer to this question will strongly affect
| this approaches scalability. This will also need a more in-depth
| analysis to study how deviations from perfect splitting affect
| the upper bound on loss.
|
| 2) With regards to statistical correctness, the L2 distance
| between the output at step N and D is the only one that is
| important. The L2 losses in the middle layers can be considered
| auxiliary losses. Maybe the final L2 loss / L2 losses deeper in
| the model should be weighted more heavily? In final evaluation
| the intermediate L2 losses can be ignored.
|
| 3) Future possibilities could include some sort of RL to
| determine the number of samples K and depth N on a dynamic basis.
| Even a split with K=2 increases NLL loss by Log_2(2) = 1. For
| many samples after a given depth the increase in loss due to the
| additional information outweighs the decrease in L2 loss. This
| also points to another difficulty, it is hard to give fractional
| information in this Discrete Distribution Network architecture.
| In contrast, diffusion models and autoregressive models can
| handle fractional bits. This could be another point of future
| development.
| elchananHaas wrote:
| A thought on why the intermediate L2 losses are important: In
| the early layers there is little information so the L2 loss
| will be high and images blurry. In much deeper layers the
| information from the argmins will dominate and there will be
| little information left to learn. The L2 losses from the
| intermediate layers help this by providing a good training
| signal when there is some information known about the target,
| but there are still large unknowns.
|
| The model can be thought of as N Discrete Distribution
| Networks, one of each depth 1 to N, that are stacked on each
| other and are being trained simultaneously.
| elchananHaas wrote:
| One more concern I noticed: This generative approach needs not
| only for each layer to select each output with uniform
| probability, but also for each layer to select each output with
| uniform probability regardless of the input.
|
| This is the bad case I am concerned about.
|
| Layer 1 -> (A, B) Layer 2 -> (C, D)
|
| Lets say Layer 1 outputs A and B each with probability 1/2
| (perfect split). Now, Layer 2 outputs C when it gets A as an
| input and D when it gets B as an input. Layer 2 is then
| outputting each output with probability 1/2, but it is not
| outputting each output with probability 1/2 when conditioned on
| the output of layer 1.
|
| If this happens, the claim of exponential increase in diversity
| each layer breaks down.
|
| It could be that the first-order approximation provided by
| Split-and-Prune is good enough. My guess though is that the
| gradient and the split-and-prune are helping each other to keep
| the outputs reasonably balanced on the datasets you are working
| on. The split and prune lets the optimization process "tunnel"
| though regions of the loss landscape that would make it hard to
| balance the classes.
| throwaway314155 wrote:
| Deeply uninformed person here:
|
| Is the inference cost of generating this tree to be pruned
| something of a hindrance? In particular I'm watching your MNIST
| example and thinking - does each cell in that video require a
| full inference? Or is this done in parallel at least? In any
| case, you're basically memory for "faster" runtime (for more
| correct outputs), no?
| mellosouls wrote:
| fwiw ICLR:
|
| International Conference on Learning Representations
|
| https://en.wikipedia.org/wiki/International_Conference_on_Le...
| Invictus0 wrote:
| Why do you refer to yourself as "we" in the paper?
| Uehreka wrote:
| When I heled author a paper in undergrad, one of the professors
| told me its just the style all papers are written in: first
| person plural and present tense.
| wordglyph wrote:
| Amazing! So basically the statistical LLM concept for imaging.
| aseg wrote:
| Slightly meta-level: I'm glad the authors finds the ICLR reviews
| useful, and this illustrates one of the successes of ICLR's
| policy of always open sourcing the reviews (regardless of whether
| the paper is accepted or rejected).
|
| The authors benefit from having "testimonials" of how anonymous
| reviewers interpreted their works, and it also allows opens the
| door to people outside of the classic academic pipeline to see
| the behind the scenes arguments to accept/reject a paper.
|
| Here are the reviews for this paper btw:
| https://openreview.net/forum?id=xNsIfzlefG
|
| And here's a list of all the rejected papers:
| https://openreview.net/group?id=ICLR.cc/2025/Conference#tab-...
| 3abiton wrote:
| Absolutely, whenever I got ICLR rejections, at least I could
| always ppint out to that reviewer who didn't understand core
| concepts of the paper.
| 3abiton wrote:
| Absolutely, whenever I got ICLR rejections, at least I could
| always ppint out to that reviewer who didn't understand core
| concepts of the paper.
| hatthew wrote:
| Super cool concept!
|
| Looking at the examples below the abstract, there's several
| details that surprise me with how correct the model is. For
| examples: hairline in row 2 column 3; shirt color in row 2
| columns 7, 8, 9, 11; lipstick throughout rows 4 and 6; face/hair
| position and shape in row 6 column 4. Of particular note is the
| red in the bottom left of row 6 column 4. It's a bit surprising--
| but still understandable--that the model realized there is
| something red there, but it's very surprising that it chose to
| put the red blob in exactly the right spot.
|
| I think some of this can be explained by bias in the dataset
| (e.g. lipstick) and cherry picking on my part (I'm ignoring the
| ones it got wildly wrong), but I can't explain the red shoulder
| strap/blob. Is there any possibility of data leakage and/or
| overfitting of a biased dataset, or are these just coincidences?
___________________________________________________________________
(page generated 2025-10-10 23:00 UTC)