[HN Gopher] Compiling a neural net to C for a speedup
___________________________________________________________________
Compiling a neural net to C for a speedup
Author : todsacerdoti
Score : 154 points
Date : 2025-05-28 17:22 UTC (5 hours ago)
(HTM) web link (slightknack.dev)
(TXT) w3m dump (slightknack.dev)
| memming wrote:
| pretty cool write up. the interesting bits were before what the
| title indicated though.
| enricozb wrote:
| Differentiable Logic Gate Networks [0] are super interesting.
| However, I still don't like that the wiring is fixed initially
| rather than learned.
|
| I did some extremely rough research into doing learnable wirings
| [1], but couldn't get past even learning ~4-bit addition.
|
| [0]: https://arxiv.org/abs/2210.08277
|
| [1]:
| https://ezb.io/thoughts/program_synthesis/boolean_circuits/2...
| jimkoen wrote:
| To ruin it for everyone: They're also patented :)
| https://patents.google.com/patent/WO2023143707A1/en?inventor...
| Lerc wrote:
| What's the innovation here?
|
| Using logic operators? Picking something from a range of
| options with SoftMax? Having a distribution to pick from?
|
| I remember reading about adaptive boolean logic networks in
| the 90's. I remember a paper about them using the phrase
| "Just say no to backpropagation". It probably goes back
| considerably earlier.
|
| Fuzzy logic was all the rage in the 90's too. Almost at the
| level of marketers sticking the label on everything the way
| AI is done today. Most of that was just 'may contain traces
| of stochasticity' but the academic field used actual defined
| logical operators for interpolated values from zero to one.
|
| A quick look on picking from a selection found
| https://psycnet.apa.org/record/1960-03588-000 but these days
| softmax is just about ubiquitous.
| jimkoen wrote:
| > What's the innovation here? > Having a distribution to
| pick from?
|
| As I understand it, it's exactly this. Specifically,
| representing neurons in a neural network via a probability
| distribution of logic gates and then collapsing the
| distribution into the optimal logic gate for a given neuron
| via hyper-parameter tuning in the form of gradient descent.
| The author has a few more details in their thesis:
|
| https://arxiv.org/abs/2209.00616
|
| Specifically it's the training approach that's patented.
| I'm glad to see that people are trying to improve on his
| method, so the patent will likely become irrelevant in the
| future as better methods emerge.
|
| The author also published an approach on applying their
| idea onto convolutional kernels in CNN's:
|
| https://arxiv.org/abs/2411.04732
|
| In the paper they promise to update their difflogic library
| with the resulting code, but apparently they seem to have
| conveniently forgotten to do this.
|
| I also think their patent is too broad, but I guess it
| speaks for the entire ML community that we haven't seen
| more patents in this area. I could also imagine that, given
| that the approach promises some very impressive performance
| improvements, they're somewhat afraid that this will be
| used for embedded military applications.
| mattdesl wrote:
| I think the techniques in "Weight Agnostic Neural Networks"
| should be applicable here, too. It uses a variant of NEAT I
| believe. This would allow for learning the topology and wiring
| rather than just gates. But, in practice it is probably pretty
| slow, and may not be all that different than a pruned and
| optimized DLGN..
|
| https://weightagnostic.github.io/
| mochomocha wrote:
| Ha! I have spent the last 2 years on this idea as a pet
| research project and have recently found a way of learning the
| wiring in a scalable fashion (arbitrary number of input bits,
| arbitray number of output bits). Would love to chat with
| someone also obsessed with this idea.
| UncleOxidant wrote:
| Also very interested. Do you have any code on github?
| AmazingTurtle wrote:
| I recently read about DLGAs on HN and instantly thought: damn
| thats some hot take. But I was too stupid to implement it from
| the paper. Glad you got it working and documented it! Thanks!
| isaacimagine wrote:
| Author here. Any questions, ask away.
| djmips wrote:
| Was this result surprising?
| isaacimagine wrote:
| Yes and no. I wasn't expecting to be able to reproduce the
| work, so I'm just content that it works. I was very surprised
| by how much hyperparameter finagling I had to do to get the
| DLGN converging; the tiny relu network I trained at the
| beginning, in comparison, converged with dead-simple SGD in a
| third of the epochs.
|
| The speedup was surprising in the sense that the bit-level
| parallelism fell out naturally: that 64x speedup alone was
| unexpected and pretty sweet. There's likely still a lot of
| speed left on the table. I just did the bare minimum to get
| the C code working: it's single-threaded, there's no
| vectorization, lots of register spilling, etc. Imagine the
| speedup you'd get running the circuit on e.g. an FPGA.
|
| But no, it was not surprising in the sense that yeah,
| multiplying billions of floats is going to be much slower
| than a handful of parallel bitwise ops. Physics is physics,
| doesn't matter how good your optimizer is.
| jgord wrote:
| what percentage of ops were passthru ?
|
| ps. superb writeup and project
| isaacimagine wrote:
| Thank you! Good question, Here are the NN stats, before
| lowering to C: total gates |
| 2303 | 100.0% -------------------+------+-------
| passthrough | 2134 | 92.7% gates w/ no
| effect | 1476 | 64.1%
|
| Note the rows aren't mutually exclusive.
| Twirrim wrote:
| You've made some mistakes with the Game of Life rules. You've
| missed out the overpopulation rule:
|
| Any live cell with more than three live neighbours dies
|
| Nit: > I guess there's a harsh third rule which is, "if the
| cell is dead, it stays dead".
|
| That phrasing is inaccurate, if a dead cell stayed dead, the
| first rule wouldn't work. I'm not sure that particular sentence
| adds much to the flow, honestly.
| nightpool wrote:
| You're thinking about the cells as toggles on a stateful
| grid, TFA is thinking about them as pure functions that take
| in an input state and output a new state (with "off" being
| the default).
|
| From that perspective, there's no point in "killing" a cell,
| it's simpler to only write out the 0 -> 1 and 1 -> 1
| transition cases and leave all of the other cases as
| implicitly 0
| GloamingNiblets wrote:
| Thank you for the excellent writeup of some extremely
| interesting work! Do you have any opinions on whether binary
| networks and/or differentiable circuits will play a large role
| in the future of AI? I've long had this hunch that we'll look
| back on current dense vector representations as an inferior way
| of encoding information.
| randomtoast wrote:
| Given the complexity of modern compiler optimizations,
| integrating a small neural network into a C compiler like GCC
| might help generate faster executable code by guiding
| optimization decisions.
| nurettin wrote:
| -O3 -march=native is pretty much all you need and the rest is
| marginal or circumstantial.
| godelski wrote:
| > I tried something new for the first time, which was to keep a
| journal during development.
|
| DO THIS!!!
|
| I cannot stress this enough!
|
| If you work in a professional science lab, say, physics, biology,
| chemistry, you are expected to keep an experiment journal. It
| provides more help to you than the company too (knowledge dump,
| liability, etc). I can't tell you how many times some stupid ass
| seemingly benign comment saved my behind. They're worth their
| weight in gold.
|
| For ML experiments I use wandb and hydra[0]. Put all your configs
| into hydra. _Be fucking pedantic._ You should log your seeds,
| versions, the date, and I mean everything. It only takes a few
| extra minutes to set this up but the one time you need it it 'll
| save you hours. Dump all that into wandb AND your model
| checkpoints. You will forget what that checkpoint corresponds to.
| Make liberal use of wandb tags and comments (through hydra you
| can make these cli arguments to automate even if launching from
| slurm scripts). Turn on wandb's code saving.
|
| Most importantly, use those notebooks wandb gives you. Don't
| worry if it gets messy. It's a experiment notebook, it'll get
| messy. You'll get better with experience and as you find your
| style.
|
| It sounds like a lot of work but it really isn't. You can get
| this all done under 20 minutes and if you write it right you can
| just copy paste it moving forward (i.e. yeah, make a personal
| library). I can PROMISE you that one mishap will far outweigh
| this extra work. You look like a pretentious perfectionist but
| really I'm a lazy piece of shit rust doesn't want to spend hours
| or days debugging some stupid mistake I'm too dumb or tired to
| catch. The extra benefit is when shit world you can spin up some
| (wandb) sweeps and go do some other thing that's always behind.
|
| (On topic, stop using personal wandb accounts for your work
| experiments. They're like the best company out there, get your
| boss to pay. They provide an amazing service and are a delight to
| work with. I cannot speak highly enough about them. They're not
| the company you want to mooch from. I've literally seen this
| happen while working for a top 3 market cap which was already
| paying for seats and you just needed to send a slack message to
| one dude... not cool guys... not cool...)
|
| [0] https://hydra.cc/docs/intro/
| 0cf8612b2e1e wrote:
| From my two minute skim of the docs, not encouraging that hydra
| only officially supports up to Python 3.11.
| godelski wrote:
| I use it in python 3.12, and 3.12 just got out of bug fix. I
| haven't tried 3.13 but I would be surprised if there was a
| break. Most of it works through OmegaConf[0].
|
| Idk why they haven't pushed an update in 2 years but neither
| has this been a problem. FWIW, they're still updating the
| repo[1]
|
| [0] https://omegaconf.readthedocs.io
|
| [1] https://github.com/facebookresearch/hydra
| nine_k wrote:
| The problem with keeping a journal is that the distraction of
| doing so may break the state of flow.
|
| OTOH there are natural breaks in the process of working;
| writing things down during these works fine. The fidelity is a
| bit lower, but it's still much better than nothing.
| isaacimagine wrote:
| Agree, it's much better to write up a journal at times when
| your colleagues would be https://xkcd.com/303
| godelski wrote:
| > the distraction of doing so may break the state of flow.
|
| Sure, but like you said, don't do it when in the state of
| flow.
|
| Or better, make it part of your flow state. To me, it is part
| of my flow state, so not a real issue.
|
| I mean whatever works for you. You gotta time manage and I
| can't manage for you. I'm sure your boss is asking for more
| writeups than I am and just send them your notes. They don't
| care whats in it half the time, they just don't know how to
| figure out if you're working or not and just want something.
|
| Hell, we're on HN on a workday... I can guarantee you aren't
| in a flow state the whole time and can't be bothered with a
| few minutes to write some stuff down. I mean you have to eat
| and go to the bathroom, right?
| isaacimagine wrote:
| Chaotic energy haha, I like it. Thanks for the tips re: keeping
| a journal, I will do this more in the future. I usually keep
| development notes, though normally in markdown files scattered
| across the codebase or in comments, never by date in the
| README. In the future, I might make JOURNAL.md a standard
| practice in my projects? re:w&b, I used w&b when it first came
| out and I liked it but I'm sure it's come a _lot_ further in
| the time since then. I will have to take a look!
|
| Also lol "pretentious perfectionist" I'm glad to finally have
| some words to describe my design aesthetic. I like crisp fonts,
| what can I say.
| godelski wrote:
| > Chaotic energy haha, I like it
|
| My boss says I'm eccentric. I say that's just a nice word for
| crazy lol
|
| > normally in markdown files scattered across the codebase or
| in comments
|
| I used to do that too but they didn't end up helping because
| I could never find them. So I moved back to using a physical
| book. The wandb reports was the first time I really had
| something where I felt like I got more out of it than a
| physical book. Even my iPad just results in a lot of lost
| stuff and more time trying to figure out why I can't just
| zoom in on the notes app. I mean what is an iPad even for if
| it isn't really good for writing?
|
| But the most important part of the process I talked about is
| the logging of all the parameters and options. Those are the
| details you tend to lose and go hunting for. So even if you
| never write a word you'll see huge benefits from this.
| > re:w&b
|
| Wandb's best feature is that you can email them requesting a
| feature and they'll implement it or help you implement it.
| It's literally their business model. I love it. I swear, they
| have a support agent assigned to me (thanks Art! And if wandb
| sees this, give the man a raise. Just look at what crazy
| people he has to deal with) > lol
| "pretentious perfectionist" I'm glad to finally have some
| words to describe my design aesthetic
|
| To be clear, I'm actually not. Too chaotic lol. Besides,
| perfectionism doesn't even exist. It's more a question about
| personal tastes and where we draw the line for what is good
| enough. I wish we'd stop saying "don't let perfectionism get
| in the way of good" because it assumes like there's universal
| agreement about what good enough is.
| isaacimagine wrote:
| Parameters and options, got it. I try to keep all
| configuration declarative and make building and running as
| deterministic as possible. Then I can commit whenever I do
| something interesting, that I can just checkout to revisit.
| godelski wrote:
| I think these are the two main headaches with
| experimenting. No matter what kind of experiment you're
| doing (computation, physics, chem, bio, whatever)
| - Why the fuck aren't things working - Why the fuck
| are things working
|
| The second is far more frustrating. The goal is to
| understand and explain _why_ things are the way they are.
| To find that causal structure, right? So in
| experimenting, getting things working means you 're not
| even half way done.
|
| So if you are "organized" and flexible, you can quickly
| test different hypotheses. Is it the seed? The model
| depth? The activation layers? What?
|
| Without the flexibility it gets too easy to test multiple
| things simultaneously and lose track. You want to isolate
| variables as much as possible. Variable interplay throws
| a wrench into that so you should make multiple
| modifications at once to optimally search through
| configuration space but how can you do any actual
| analysis if you don't record this stuff. And I guarantee
| you'll have some hunch and be like "wait, I did something
| earlier that would be affected by that!" and you can go
| check to see if you should narrow down on that thing or
| not.
|
| The reason experimenting is hard is because it is the
| little shit that matters. That's why I'm a crazy
| pretentious "perfectionist". Because I'm lazy and don't
| have the budgets or time to be exhaustive. So free up
| your ability so you can quickly launch experiments and
| spend more time working on your hypotheses, because that
| task is hard enough. You don't want to do that while also
| having to be debugging and making big changes to code
| where you're really just going to accidentally introduce
| more errors. At least that's what happens to my dumb ass,
| but I haven't yet met a person that avoids this, so I
| know I'm not alone.
| rvz wrote:
| All of this agreed.
|
| Now in the age of AI, many students entering into CS need do
| this NOW, otherwise any answer they come up in the interview,
| will be assumed that it was from an AI and they need to show
| that they something useful came out of their blogpost or
| research.
|
| It is what it now means to know how to experiment, understand
| and build knowledge, rather than spitting out the answer
| because it it from stack overflow or ChatGPT.
|
| The mistakes are raw, all the learnings in a blog post which is
| what makes us human yet, 90% of candidates do not do this which
| is why most of them cannot explain an AI's mistakes an
| interview if they use it.
| godelski wrote:
| I actually really like this idea. I've often found it odd we
| don't show off reports or how we run experiments during
| interviews. Certainly this has far greater influence over
| your aptitude than leetcode. > 90% of
| candidates do not do this or cannot explain AI's mistakes an
| interview.
|
| I have a growing concern that people do not _see_ mistakes.
| This seems to be a bigger divide than "uses AI to code" vs
| "doesn't".
| mr_toad wrote:
| Same goes for something as simple as setting up a server. You
| will forget, and if you don't write it down you'll have to
| figure it out again.
| godelski wrote:
| Where did I put those configs again? Where did Bob put that
| script? Fuck, why didn't I write an ansible script. It's
| never a one off, and it serves as documentation. I'll
| remember after I make the same mistake next time.
|
| Also, environment modules for the win
|
| https://modules.readthedocs.io/en/latest/
| crubier wrote:
| Work logs are generalized at my company and they are AWESOME
| heavyset_go wrote:
| My current workflow is to keep a wiki, would you say hydra
| would replace/complement especially that if you're used to note
| keeping the wiki way?
| godelski wrote:
| Hydra is part of the documentation process imo. Truthfully,
| the most important stuff that goes in your experiment journal
| is all those pesky parameters and things that can
| surprisingly change results.
|
| So I love that hydra uses OmegaConf and I essentially get 3
| copies: the experiment config yaml, the wandb log, a
| dictionary in the checkpoint. Multiple times my dumbass has
| had to try to match the checkpoint to the wandb log, so the
| redundancy is incredibly helpful. Sometimes just a library
| version has unexpected changes on performance and this makes
| it trivial to trace. The yaml file is more helpful when
| passing off the code to someone else or releasing to public.
|
| So yeah, I would say that it'll benefit no matter how you
| document. Use whatever documentation method works for you.
| Reports can still offer some benefits in just throwing some
| charts together quickly and organizing but I think you'd
| still benefit from hydra. It's too easy to lose track of
| those little things and this helps me automate. But you can
| also just straight up use OmegaConf or even dictionaries.
| Whatever works for you.
|
| The real help is logging. So whatever tools help you log, use
| them. This is just what I benefit from (there's a lot I can
| talk about too and I'd love to see what others do as well)
| lairv wrote:
| Converged to something similar after spending 2 days bissecting
| a repo to reproduce a training run, having to wait 3hr on each
| commit before conclusive results. I couldn't get myself to use
| hydra though, it felt like a lot of bloat vs loading a yaml
| with pydantic
| tomcam wrote:
| OK so I try to do that. But then I'll have some big problem or
| add too many big features and just give up. (My sleep is nearly
| nonexistent so I don't have a lot of time for logging things
| anyway.)
| chairmansteve wrote:
| Anyone use a digital notebook, like the reMarkable, for this
| kind of thing?
| thyristan wrote:
| Started but stopped. most of the things are commands, code
| snippets, urls, all of which are tedious to hand-write and
| just easier to copy&paste. Often I do 'typescript' for shell
| sessions, asciinema or stuff like that, and file those.
|
| Also, use git, commit everything, never care about doing tidy
| commits, just commit commit commit and use tons of branches
| to try out stuff. if you need clean history later on, you can
| always do interactive rebase, squash merge or whatever. but
| having a documentation of all the things tried and failed is
| far more important.
| Xss3 wrote:
| I thought the entire point was that it syncs cross platform
| really quickly and lets you have the best of both worlds?
| thyristan wrote:
| Only handwriting in a proprietary format. It isn't at all
| like one would wish for. It works as a replacement for a
| paper notebook. But it largely ignores the things one
| could do when adding more digital embeddings. In that way
| it is even worse than OneNote.
|
| If you want to do something like that, my recommendation
| would actually be something like OneNote on some Windows
| tablet.
| rolandhvar wrote:
| So here's the thing I struggle with. I do a lot of work in
| jupyter notebooks. I come up with a new model or approach to
| some problem, and I want to fork out and test a hypothesis in
| the background (which might be some set of hyperparameters, and
| might take several minutes, or hours; call it Run A) while
| continuing to work down some other path in the same notebook,
| and maybe kick off a Run B that explores some other change
| (like a restructure of the code that's not "compatible" with
| the hyperparameter search of Run A).
|
| Then at some point when Run A finishes, I want to incorporate
| the changes I made in Run B and kick off Run C, and so on.
|
| The hard/important things are:
|
| 1) Being able to do this while staying in a Jupyter notebook
| context the whole time. Even something as simple as
| multiprocessing sucks because I've found it's too hard to
| manage in a Jupyter context (e.g. how do you handle where
| stdout and stderr go?). It's easier if you move to scripts
| where you have full support for this sort of thing and you are
| expecting to look at multiple log files on disk and whatnot.
|
| Also the sequential nature of notebooks doesn't help when you
| want to occasionally fork out or conditionally run stuff.
|
| 2) Keeping track of all these changes and hypotheses and
| merging the results/code together as you learn. It's like you
| need a VCS for your hypotheses. Maybe hydra & wandb help with
| that, I haven't used them. But this idea of keeping track of
| hypotheses seems like the more fundamental thing.
|
| 3) The main reason I prefer to stay in a notebook context is
| because I have all my objects easily accessible. My models, all
| my dataframes, functions to do some ad-hoc charting etc, all
| super easy to access in a REPL-like form. That is invaluable
| for doing ad-hoc sanity checks or digging/drilling down. So a
| big part of the workflow is you basically have this in-memory
| database of a bunch of relevant objects and you're querying it
| and constructing new objects & visualisations using Python as
| your tool, without having to load things from disk or build up
| the context from scratch. It's all "just there".
|
| 4) And then sometimes you want to take the results X1 of that
| notebook and plot them against some entirely different set of
| data X2 that requires a whole bunch of other code that you've
| defined in some other notebook somewhere, or maybe even as a
| real Python module. Like maybe that data lives in a database
| and you transform it or something. So OK, you call some
| functions to load X2 within your original notebook, but BOOM
| you get an OOM and you're like ok now I have to write some code
| to serialise X1 to disk, and make YET ANOTHER notebook so I can
| go analyze X1 and X2. It all just seems so... unnecessary, if
| only the right tooling existed.
|
| My current best approach is to use semantic versioning on the
| filename, just copy the whole notebook each time I make a
| fundamental change, and try to keep track of my hypotheses,
| preconditions, learnings etc within comments and have a few of
| those on the go running, but it's often hard to engage in
| critical thinking when everything you know is sprawled across
| multiple notebooks.
|
| Maybe a simple global journal is the only thing for this sort
| of use case. And that doesn't even address (4) which is often a
| huge pain point. Can anyone think of something better?
| nine_k wrote:
| The interesting thing here is that it's _not_ a straightforward
| port. JAX is already very fast, for the architecture it
| implements. The point is that the network is heavily contracted
| by removing nodes that only do pass-through, and then hugely
| parallelizing the computations using bitwise operations on 64
| bits at once. Hence this incredible speedup.
| JonChesterfield wrote:
| If you replace the uint64_t cell with an
| attribute((vector_size(32))) and build with march=native, the
| bitwise ops will work exactly as before but you'll light up the
| vector units on the x64 machine.
|
| Good blog post, thanks!
| hermitShell wrote:
| This is very fascinating as a limit case, which always serve as a
| good example of the bound. I think it highlights that "efficiency
| isn't everything" just like in so many other systems like
| healthcare and justice. In this case we could figure out the
| activation functions by analysis, which is impossible for
| problems of higher dimensionality. The magic of AI isn't in it's
| efficiency, it's in making things computable that simply aren't
| by other means.
| gwern wrote:
| How much of a speedup is the C compiler optimization able to
| achieve in terms of compiling it down to a hand-written C
| equivalent vs the -O0 non-optimized assembler? What does the
| optimized C/assembler do which isn't actually necessary and
| accounts for the remaining inefficiency?
| thirtygeo wrote:
| That approach is bananas! I had seen the source inspiration paper
| from Google but it's need to see it replicated and extended so
| shortly after.
___________________________________________________________________
(page generated 2025-05-28 23:00 UTC)