[HN Gopher] A bug that taught me more about PyTorch than years o...
___________________________________________________________________
A bug that taught me more about PyTorch than years of using it
Author : bblcla
Score : 294 points
Date : 2025-10-23 17:06 UTC (3 days ago)
(HTM) web link (elanapearl.github.io)
(TXT) w3m dump (elanapearl.github.io)
| brilee wrote:
| Great write-up, but I admit that I found the interweaving of
| human and AI-written content/headlines/summaries pretty
| distracting. I kept on wanting to scroll past, but had to keep on
| backtracking to find the human thread again.
|
| I think if you want to give your reader a quick intro to, e.g.,
| what is the Adam optimizer, a simple link to Wikipedia is fine.
| No need to copy-paste an AI tutorial on Adam into the blog post.
| CaptainOfCoit wrote:
| To be fair, you can easily click to hide those expanded
| sections. I found it a neat compromise between "Link to
| (usually) obtuse Wikipedia article" which aren't usually
| written for laypersons, and forcing me to read through stuff I
| already know about, I just hid the sections I already
| understood but found value in the others.
| reilly3000 wrote:
| I came here to say the same thing. Claude's voice was pretty
| evident, but became actually grating when the header was "The
| Fix".
| cadamsdotcom wrote:
| Sounds like Placeholder should somehow be split into
| InputPlaceholder and OutputPlaceholder, based on the usage.
|
| Even identical classes could help future folks know copying back
| is platform specific: "hm, we wrote to an OutputPlaceholder but
| didn't read back from it, that seems wrong".
| ramses0 wrote:
| Apps Hungarian v. System Hungarian:
| https://herbsutter.com/2008/07/15/hungarian-notation-is-clea...
| kccqzy wrote:
| This is a minor quibble but I don't really like the author
| calling Placeholder a leaky abstraction. It's just straight up an
| incomplete abstraction that only handles inputs but not outputs.
| As the author says, Placeholder should know about the difference
| and do the copy-back itself.
| airza wrote:
| I too have been insanely burned by an MPS bug. I wish Apple would
| throw an engineer or two at making sure their hardware works with
| PyTorch.
| montebicyclelo wrote:
| Incorrect Pytorch gradients with Apple MPS backend...
|
| Yep this kind of thing can happen. I found and reported incorrect
| gradients for Apple's Metal-backed tensorflow conv2d in 2021 [1].
|
| (Pretty sure I've seen incorrect gradients with another Pytorch
| backend, but that was a few years ago and I don't seem to have
| raised an issue to refer to... )
|
| One might think this class of errors would be caught by a test
| suite. Autodiff can be tested quite comprehensively against
| numerical differentiation [2]. (Although this example is from a
| much simpler lib than Pytorch, so I could be missing something.)
|
| [1] https://github.com/apple/tensorflow_macos/issues/230
|
| [2]
| https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...
| liuliu wrote:
| Yeah, luckily, you can unit tests these and fix them. They are
| not concurrency bugs (again, luckily).
|
| BTW, numeric differentiation can only be tested very limitedly
| (due to algorithmic complexity when you doing big matrix). It
| is much easier / effective to test against multiple
| implementations.
| CaptainOfCoit wrote:
| Only slightly related, but how common are bugs in GPUs and/or
| CUDA? I'm currently on Day 5 of trying to debug why my GPT-OSS
| implementation (not using PyTorch) I've made from scratch isn't
| working correctly, and while I have it somewhat working with some
| naive and slow methods, I'm now doing an implementation of the
| tensor cores and have been just stuck for 2-3 days because of
| some small numerical difference I can't understand why it's
| happening.
|
| Every day I'm getting closer to believing this is some sort of
| hardware bug in Blackwell or in CUDA itself, but as we know, the
| bug is (almost) never in the compiler or in the hardware. Until
| it is...
| saagarjha wrote:
| How big is the numerical difference? If it's small it might be
| within the precision of the operation itself.
| CaptainOfCoit wrote:
| Magnitudes away (maybe "small numerical difference" was an
| understatement), my current hypothesis is that I'm doing
| scaling wrong somewhere, but I can't help but sometimes slide
| into the "maybe there is something deeper wrong" territory in
| the evening after another day...
| QuadmasterXLII wrote:
| You may be running into jensen (huang)'s inequality,
|
| E(loss).cuda() <= E(loss.cuda())
| CaptainOfCoit wrote:
| Would make sense I suppose if I was using two different GPUs
| for the same thing and get two different outcomes. But
| instead I have two implementations (one naive, one tensor
| cores) running on the same GPU, but getting different
| outcomes, where they should be the same.
|
| But then this joke might be flying above my head as well.
| p1esk wrote:
| Tensor cores use lower precision, so small numerical
| differences should be expected.
| hansvm wrote:
| They exist, but they're not that common (give or take the
| "expected" numerical deviations based on the order of summation
| and whatnot, which can both be nontrivial and propagate error
| further).
|
| Something I recommend doing, the best time being the start of
| the project and the second best time being now, is adding
| numerical gradient checking tests to all operations. You will
| make mistakes in your kernels from time to time, and it's
| valuable to know at a glance where those mistakes are.
|
| Mind you, it's possible to write both the forward pass and the
| backward pass in a way that's wrong but compatible. An
| additional layer of checks I like to add is a dead-simple
| implementation of all algorithms -- no vectorization, no fancy
| blocking or re-orderings, nothing. Compare results to the
| simple implementation.
|
| It sounds like a lot of work, but writing an optimized kernel
| is much slower than the numerical gradient checking and the
| simple kernel, and given how in numerical code it's basically
| impossible to identify the source of a bug without doing the
| equivalent of all of those checks, it only takes one bug in the
| whole project for the effort to pay off.
| CaptainOfCoit wrote:
| Thanks a lot for the pointers, I think I've done a similar
| approach to what you suggest, lots of tiny (relative) tests
| for each step in the process, and doing sort of sanity
| checking between the naive stuff I first wrote which works
| and which does inference correctly, and the new kernel which
| is a lot more performant, but currently incorrect and
| produces incoherent outputs.
|
| I'll try to replace bits by simplified versions though,
| probably could help at least getting closer to knowing where
| the issue is.
|
| Anyone have more debugging tips I'd greatly appreciate it!
| Nothing is too small or "obvious", as I'm about to lose my
| mind more or less.
| jjmarr wrote:
| Consumer-visible hardware bugs are extremely uncommon nowadays.
| There's approximately 10x as many people working in design
| verification as actual hardware design.
|
| I say "consumer-visible" because the bugs still exist and
| people who can catch them early get promoted quickly and paid a
| lot. It's very exciting work if you can get it, since you
| really have to understand the full GPU to break it.
|
| Good luck!!
| gugagore wrote:
| This is the first time I see "SGD" to mean "standard gradient
| descent" and not "stochastic gradient descent".
| tavianator wrote:
| Presumably that's just a mistake. The author calls it
| "stochastic gradient descent" correctly elsewhere in the
| article
| elanapearl wrote:
| haha oops yeah the other comment is correct- that was just a
| mistake
|
| I originally wrote "vanilla" there but didn't want to repeat
| that word twice in a row so swapped it for "standard" without
| realizing it now looked like the SGD acronym
|
| just fixed that to avoid confusion- thanks for pointing it out!
| saagarjha wrote:
| Non-contiguous tensors have to be the #1 source of bugs in
| PyTorch lol
| jebarker wrote:
| This is a great write up and I'd love to see more like it.
| Debugging this sort of thing in the megatron->pytorch->CUDA stack
| is what my team spends more than half of their time on as an ML
| research team.
| hobom wrote:
| What a fantastic way to write a post mortem, pedagogically very
| useful.
| dangoodmanUT wrote:
| The tinygrad folks talk about this a lot.
|
| Not that I understand much of what they say, but it appears there
| are a lot of correctness bugs in pytorch that are flying under
| the radar, probably having a measurable impact on the results of
| model quality.
|
| It would be interesting to see model weights comparison of the
| same model trained with the two to see if they exhibit
| meaningfully different behavior.
| CaptainOfCoit wrote:
| > Not that I understand much of what they say, but it appears
| there are a lot of correctness bugs in pytorch that are flying
| under the radar, probably having a measurable impact on the
| results of model quality.
|
| Do you have any links to public thoughts about this? As if it
| was true, could mean _a lot_ of research could be invalidated,
| so obviously would make huge news.
|
| Also feels like something that would be relatively easy to make
| reproducible test cases from, so easy to prove if that's true
| or not.
|
| And finally if something is easy to validate, and would make
| huge news, I feel like someone would already have attempted to
| prove this, and if it was true, would have published something
| a long time ago.
| dangoodmanUT wrote:
| Check their Twitter, I saw something either yesterday or
| earlier today iirc
| 3abiton wrote:
| That's why project like nanochat are really cool, you can get
| around the limitations of such gigantic libraries, while at the
| same time understanding the underlying architecture.
| woodson wrote:
| Nanochat is using PyTorch under the hood. I don't understand
| your point.
| CamperBob2 wrote:
| They might be referring to Karpathy's earlier micrograd
| tutorial, where the whole thing is built from scratch. That
| was how I learned the basics myself.
| coredog64 wrote:
| When we update Torch versions, we're required to run a test
| where the only change is the library change and compare the
| outputs. We saw a measurable improvement in accuracy by
| upgrading from torch 2.4.x to 2.7.x.
| doctorpangloss wrote:
| I see another commenter highlighted this:
|
| > The exact same float32 code updates weights on CPU but fails
| on MPS
|
| It's MPS... Exactly zero research is being impacted. Why
| doesn't the $3.9T corporation contribute more to torch?
| ACCount37 wrote:
| I mean, some researchers clearly use Apple Silicon for their
| "cheap and cheerful" runs.
| dapperdrake wrote:
| https://moyix.blogspot.com/2022/09/someones-been-messing-wit...
|
| TLDR: Python gevent compiled with -Ofast messes up x87 floating
| point unit state. Bad for PyTorch.
| woodson wrote:
| I thought that the effect of these compiler flags was widely
| known in numerical computing. It allows e.g., reordering of
| floating point computations and in general disregards IEEE
| 754. As such, these results are expected, I'd think.
| tempay wrote:
| Widely know amongst very niche groups, most of whom have
| either been burnt by the issue or heard about someone who
| has and have it ingrained in their mind out of fear of
| debugging such a thing.
|
| I'd bet the majority of ML people are unaware, including
| those doing lower level stuff.
| pm215 wrote:
| The unexpected thing in that particular case is that even
| if you were well aware and avoided the flag when building
| your numeric code, the way some other non-numeric-computing
| person compiled some unrelated non-numeric module like
| "gevent" could result in the fast-math behaviour being
| applied to your code too. (Happily gcc has now fixed this.)
| Q6T46nT668w6i3m wrote:
| The tinygrad folks talk too much
| dataflow wrote:
| Dumb question: why isn't there some kind of assertion to sanity-
| check some bits of the GPU results against CPU's?
| nraynaud wrote:
| Naive question: ML tensor libraries don't use a Z-order memory
| layout like textures do? It's not beneficial like it is for
| textures?
| hinkley wrote:
| Reminds me of the largest AJAX app I worked on, back when jquery
| was still hot and IE6 still existed as a problem.
|
| The landing page in our app used jqueryUI's drag and drop
| support, back around the time they declared bankruptcy on the
| confusing buggy code and wouldn't even accept bug fixes because
| they were replacing it component by component (which was taking
| almost 3x as long as predicted). We had columns you could drag
| items between but they had a max height and scroll bars and it
| turned out jqueryUI would let you drag items into different rows
| if the overflow area for adjacent drag targets overlapped your
| row.
|
| The person who found it couldn't fix it. The other fixer couldn't
| fix it. I diagnosed it but the spaghetti code was a recursive
| mess and I could not find a spot where I could fix it. Especially
| given I couldn't send in a patch to them.
|
| So I spent half of my free time the last day of every (2 week)
| sprint for almost six months before I finally found a small
| function I could monkey patch to wrap it in a short circuit check
| for clipping region. I spent maybe 20,30 hours on this, a lot of
| it just getting back to the same situation to debug. But it felt
| like it took forever to fix it.
|
| The short circuit also made drag and drop faster, which was just
| getting in the edge of distracting. Particularly on a crowded
| page.
| CaptainOfCoit wrote:
| I remember many similar cycles of having different browsers
| open side-by-side, and trying to pinpoint (without the
| developer tools we know and love today) the exact reason why
| one border was one pixel in one browser, and two pixels in the
| other, throwing the whole layout off.
|
| Also remembering when Firebug for Firefox appeared, and made so
| many things so much easier. Suddenly things that took hours
| took days, and it was so much easier when you had some
| introspection tools.
| yard2010 wrote:
| * { border: red 1px solid } Remember when IE6 was a thing?
| The kids today are angry at chrome for good reasons and yet,
| there was a time in which the most popular browser didn't
| implement jack shit from the specs. And it was the kind of
| browser that ships with the OS.
|
| God the bad karma for working with this crap. I'm glad it's
| over.
| hinkley wrote:
| I had to do a reflow reordering trick on a sibling page in
| that app and it doubled or tripled the speed on FF and
| safari, but on IE6 the test case went from 30s to 3.5s.
| Good Christ.
| hinkley wrote:
| That bug took me on a whirlwind tour of that code and I
| understand why they wanted to start over. Woof.
| tosapple wrote:
| >> and made so many things so much easier. >> Suddenly things
| that took hours took days
|
| Inverse? Shouldn't it be things that took days took hours ?
| ipsum2 wrote:
| Apple used to contribute to the PyTorch MPS backend, but decided
| to create their own framework (MLX) instead, fragmenting the
| ecosystem for very little gain. (MLX is basically PyTorch, but
| invented-at-apple)
|
| Meta, the creator and main contributor to PyTorch, does not use
| Macs for their day-to-day ML work (they focus on GPUs and CPUs),
| so the MPS backend is sadly incomplete and has errors like the
| one you see here.
| almostgotcaught wrote:
| none of this is correct (except the part where FB doesn't use
| apple in prod).
|
| EDIT: for the downvoters - i'll repeat, this is not a correct
| assessment of the relationship between Apple and PyTorch. but
| you can keep downvoting if you want <shrug>
| ipsum2 wrote:
| Please be specific if you have anything to say. By the way,
| the co-creator and core maintainer of PyTorch has the same
| opinion as me.
|
| https://x.com/soumithchintala/status/1978848796953161754
|
| "MacStudio you ask?
|
| Apple Engineering's *actual* time spent on PyTorch support
| has't given me confidence that PyTorch Mac experience would
| get anywhere close to NVIDIA's any time soon, if ever.
|
| The Meta engineers continue to do a huge amount of heavy-
| lifting for improving the MPS backend, including feeling the
| responsibility for the Mac experience. Apple's priorities
| keep changing, the number of engineering hours they
| contribute keeps changing and their interest in actually and
| wholly owning the PyTorch MPS backend keeps varying.
|
| If Apple wants MacStudio to become an actual AI devbox, and
| not just an AI inference machine, then prioritizing software
| support for PyTorch (>90% marketshare in AI) would probably
| be a good idea."
| hedgehog wrote:
| Apple has never cared about ML research on their hardware.
| I've never been able to pin down a specific reason why,
| best I can figure out is they don't see it bringing enough
| additional hardware sales to be a focus.
| almostgotcaught wrote:
| lol @ quoting soumith - the guy's sole job responsibility
| is tweeting.
| ipsum2 wrote:
| If you have more knowledge than the core maintainer of
| pytorch, why are you unwilling to share, instead of
| snarking?
| Q6T46nT668w6i3m wrote:
| Everyone I know at Meta uses a Mac
| ipsum2 wrote:
| No one at Meta runs local inference on a Mac, unless its
| for fun.
| sampton wrote:
| MLX and MPS are 2 completely different teams within Apple. It's
| more like MPS team doesn't have control or visibility into
| PyTorch roadmap and can only contribute so much from their
| side.
| mirekrusin wrote:
| Nice work, surprising, I'd imagine implementations are cross
| tested all the time and this kind of bugs have no way of
| appearing?
| modeless wrote:
| Another reason people use Nvidia. You know that Nvidia is the
| most used backend and the most likely to have this kind of bug
| found and fixed before you encounter it.
| cryber wrote:
| this is a great writeup! methodical without being pedantic.
| hershyb_ wrote:
| awesome read!
| anal_reactor wrote:
| If I understand correctly, the root cause of the bug was improper
| use of object-oriented programming. A `Placeholder` object
| behaves differently depending on how it was created, and requires
| the user to have this awareness. The check `if is_continuous`
| should only ever exist inside the code of the `Placeholder`
| class.
| albertzeyer wrote:
| The bug was with non-contiguous data in tensors.
|
| I also had a very similar bug a while ago, broken gradients due
| to non-contiguous data for masked_select:
| https://github.com/pytorch/pytorch/issues/99638
|
| In my case, it was easier to identify: I had another
| implementation of my loss function before that did not use
| masked_select. But then I thought I can be clever and use
| masked_select to take out the non-masked frames and calculate the
| loss only on those. But it wasn't working. Also, it only happened
| for some models, not for all. It turns out, it was always
| happening when the data coming out of the model was non-
| contiguous.
|
| I think the bugs with non-contiguous data are not so uncommon. I
| wonder how much of that we still have.
| dcl wrote:
| Is this why I cannot seem to fine tune YOLO models on a Apple M4?
| The loss hits nan after a few batches. Same code using Windows PC
| and Google Colab CPU and GPU is fine...
___________________________________________________________________
(page generated 2025-10-26 23:00 UTC)