[HN Gopher] Five Years of Tinygrad
___________________________________________________________________
Five Years of Tinygrad
Author : iyaja
Score : 170 points
Date : 2025-12-29 17:12 UTC (1 days ago)
(HTM) web link (geohot.github.io)
(TXT) w3m dump (geohot.github.io)
| mika6996 wrote:
| What would tinygrad replace if they continue to proceed like
| this?
| spiderfarmer wrote:
| Potentially PyTorch and Tensorflow.
| cyberax wrote:
| I think it has great potential for deployments on edge systems.
| piskov wrote:
| It is already used in comma.ai's openpilot hardware
| vessenes wrote:
| But that is an inside deal - same founder, I believe
| vileain wrote:
| [flagged]
| spiderfarmer wrote:
| There are lots of bubbles where Elon is still king. Those
| bubbles are often void of deodorant.
| vileain wrote:
| Based on the response it appears HN is one such bubble.
| spiderfarmer wrote:
| Elon spent billions to buy a platform and promote his
| tweets. He spent billions more to create a tweaked AI model
| that praised him like a mad king.
|
| He only has to spend a couple thousand a month to influence
| comment ranking on HN.
| mycodendral wrote:
| the value is the directness, not implied origination
|
| not everyone cares about playing voldemort
| vileain wrote:
| What is so aggrandizingly 'direct' about calling the system
| you are attempting to improve 'dumb'?
| mycodendral wrote:
| you have a more direct way?
| dang wrote:
| " _Please don 't pick the most provocative thing in an article
| or post to complain about in the thread. Find something
| interesting to respond to instead._"
|
| https://news.ycombinator.com/newsguidelines.html
| pa7ch wrote:
| Very weird to market this as subscribing to "Elon process for
| software"
|
| I remember when defcon ctf would play Geohot's PlayStation rap
| video every year on the wall.
| spiderfarmer wrote:
| I hate it when 'inspirational' quotes are attributed to the
| person with the largest audience and not the people who came up
| with it, like in this case, the engineers at Lockheed's Skunk
| Works.
| piskov wrote:
| You would be amazed
|
| https://en.wikipedia.org/wiki/TRIZ
| ramesh31 wrote:
| It's an apocryphal quote.
|
| "A designer knows he has achieved perfection not when there
| is nothing left to add, but when there is nothing left to
| take away."
|
| - Antoine de Saint-Exupery
| vessenes wrote:
| Or Colin Chapman at Lotus: "Simplify, then add lightness."
| Could just be wise advice :)
| deburo wrote:
| So this is all python? I bet Chris Lattner probably approached
| them.
| zephen wrote:
| Lattner is a smart guy, but I think Mojo might be the wrong
| direction.
|
| Time will tell.
|
| History has not so far been kind to projects which attempt to
| supplant cPython, whether they are other Python variants such
| as PyPy, or other languages such as julia.
|
| Python has a lot of detractors, but (despite some huge missteps
| with the 2-3 transition) the core team keeps churning out stuff
| that people want to use.
|
| Mojo is being positioned "as a member of the Python family"
| but, like Pyrex/Cython, it has special syntax, and even worse,
| the calling convention is both different than Python, and
| depends on the type of variable being passed. And the
| introspection is completely missing.
| tucnak wrote:
| Honestly, I feel like Julia might as well beat Mojo or sommat
| to the punch, sooner or later. It has some facilities and
| supporting infrastructure for a lot of scientific and data-
| handling tasks surrounding ML, if not for compiling and
| dispatching kernels (where XLA reins supreme to anything in
| the CUDA ecosystem!) For example, Bayesian programming like
| Turing.jl is virtually unmatched in Python. It's been a while
| since I looked at Lux.jl for XLA integration, but I reckon it
| could be incredibly useful. As long as LLM's and RLVR
| training thereof should continue to improve, we may be able
| to translate loads of exiting Pytorch code eventually.
| eli_gottlieb wrote:
| > For example, Bayesian programming like Turing.jl is
| virtually unmatched in Python.
|
| What about numpyro?
|
| Disclaimer: I contribute to numpyro occasionally.
| tucnak wrote:
| Honestly, I'm not familiar with it. I had only played
| with RxInfer, if only to try the so-called "message-
| passing" paradigm. My grasp on probability is really
| lacking, in fact I picked up prediction markets and Julia
| to get better at it.
|
| If you don't mind me asking, what's the deal with NumPyro
| why you chose to work it?
| zephen wrote:
| I dunno. This sort of thing gives me pause:
|
| https://danluu.com/julialang/
|
| But the first thing that gave me pause about Julia? They
| sort of pivoted to say "we're general purpose" but the
| whole index-starting-at-one thing really belies that --
| these days, that's pretty much the province of specialty
| languages.
| tucnak wrote:
| You're not supposed to admit it, but I never cared for
| Dijkstra's arguments on the matter. The same goes for his
| GOTO tirade, although that has been distorted by time
| somewhat. Pascal is using 1-ord, Fortran, R, Mathematica.
| If anything, it seems there's a longer tradition of 1-ord
| in scientific computing. In this view, I must agree
| insofar I don't think Julia people are serious about
| their "general purpose" stance whatsoever. But hey, these
| are merely idiosyncrasies. People say multiple dispatch
| is the shit, but it's just one bit of the puzzle with
| Julia. How they managed to engineer a solid foundation,
| semantics like that, without unnecessarily sacrificing
| performance--I don't think they get enough credit for
| that from programming guys.
| drnick1 wrote:
| This 100% concerning Julia. Until recently at least,
| there was a ton of fundamental correctness issues. The
| issues may have been fixed, but it certainly does not
| inspire trust.
| paxys wrote:
| Lots of words and weird analogies to say basically nothing.
|
| What is the status of the project? What can it do? What has it
| achieved in 5 years?
|
| But no, let's highlight how we follow the "Elon process".
|
| As a side note, whenever someone incessantly focuses on lines of
| code as a metric (in either direction), I immediately start to
| take them less seriously.
| jszymborski wrote:
| From [0]:
|
| "When we can reproduce a common set of papers on 1 NVIDIA GPU
| 2x faster than PyTorch. We also want the speed to be good on
| the M1. ETA, Q2 next year."
|
| [0] https://tinygrad.org/#tinybox
| dewey wrote:
| Using lines of code as a metric for productivity is bad. Using
| it to show how simple something is, or how a refactor removed x
| lines of code that doesn't need to be maintained any more isn't
| such a bad thing I'd say.
| whilenot-dev wrote:
| TFA includes a time measurement though, and 5 years for
| 18'935 SLOC doesn't scream quite "how simple something is".
| alphazard wrote:
| Yeah this is exactly right, if you can trust the contributors
| to not code-golf or otherwise Goodhart the LoC metric, then
| it's a reasonable measure of complexity.
|
| It doesn't work as well when you start mixing languages, or
| generating code.
| selkin wrote:
| Less LOC also doesn't imply simplicity: just look at the
| demoscene, which often has the former but not the latter.
| dewey wrote:
| Demo scene or other competitions that aim at using little
| space / shortest way to achieve something are not really a
| good counter example.
| piskov wrote:
| He was able to run nvidia gpu on mac via thunderbolt with
| tinygrad.
|
| https://www.tomshardware.com/pc-components/gpus/tiny-corp-su...
|
| --
|
| Check tinygrad's twitter account for specifics if you want to
| catch up on progress
| JoeDohn wrote:
| making things less dumb is not a elon's process, if it's the
| case then we are saying everything that elon is not involved
| in/with is dumb !
| bko wrote:
| Did we read the same article?
|
| The third sentence is "The codebase is 18,935 lines not
| including tests." You can go to the repo and see what it does.
| Look at issues, PRs, etc. He wasn't saying it to brag about its
| size, but the opposite.
|
| The mission is to commoditize the petaflop. Basically allow
| LLMs to be trained efficiently on commodity non-NVIDIA GPUs.
| Would you prefer some bullshit mission like Meta of "connecting
| people" or whatever the hell it is?
|
| He said he has a hardware division that makes $2m a year. You
| can click on the link and buy a computer. He tells you the
| revenue.
|
| He said he has a deal with AMD which is also public and on X.
|
| The "Elon process" relies specifically to the goal of getting
| rid of all dependencies. Musk has spoken extensively about
| building things from the ground up and not relying on other
| vendors (in this example complex software dependencies). He
| says he wouldn't be able to build SpaceX competitively if he
| had just bought rockets or components.
|
| I wish more people focused on public code and shipping
| publicly. Can I see Toyotas stack? Why does the touch screen
| perform worse than a 2012 iPad. What the hell is going on
|
| I don't understand this hate someone like Hotz, a true engineer
| running an ambitious very open company, receives on a ...
| checks notes ... engineering forum? The whole setup is like a
| wet dream for engineers from a decade ago.
|
| Almost completely open source
|
| Interview is completing a paid bounty
|
| CEO is engineer
|
| Has a clear mission, goals, and timeline
|
| What is your issue?
| grumbelbart wrote:
| > The "Elon process" relies specifically to the goal of
| getting rid of all dependencies. Musk has spoken extensively
| about building things from the ground up and not relying on
| other vendors (in this example complex software
| dependencies). He says he wouldn't be able to build SpaceX
| competitively if he had just bought rockets or components.
|
| That I cannot believe. He might have shifted the make-or-buy
| decisions, but both Tesla and SpaceX do a lot of outsourcing.
| timzaman wrote:
| Fell bad for geohotz. Such a lovely guy, i hope he strikes it
| right soon
| still-learning wrote:
| Seems like he's doing fine, why do you feel bad for him?
| still-learning wrote:
| >People get hired by contributing to the repo. It's a very self
| directed job, with one meeting a week and a goal of making
| tinygrad better
|
| I find this organizational structure compelling, probably the
| closest to reaching 100% productivity in a week as you can get.
| ttul wrote:
| I wonder what happened to George's old policy of requiring
| everyone to move to San Diego?
| georgehotz wrote:
| That's comma.ai's policy since they make hardware and solve
| physical problems. The tiny corp has been hybrid (remote-
| first) since day 1 because it primarily writes open source
| software, and there's a long track record of success with
| remote for this kind of task.
|
| We have a few whole-team meetups in Hong Kong each year for
| 2-4 weeks, and there's a San Diego or Hong Kong office that
| anyone can work from as they choose. We also have a wide
| array of fancy multi GPU boxes that everyone on the team gets
| full access to (known external contributors can get some
| access also).
|
| I think many companies that were quick to embrace remote have
| walked it back, not everyone is capable of working
| productively remotely, nor are all types of work amenable to
| remote.
| dustbunny wrote:
| Why Hong Kong? I guess you have a bunch of contributors
| near there?
| octoberfranklin wrote:
| Have you run into problems with contributors who can't
| enter the PRC?
| ttul wrote:
| An answer from the man himself. Thank you.
| measurablefunc wrote:
| Is it really "Complex"? Or did we just make it "Complicated"? -
| https://www.youtube.com/watch?v=ubaX1Smg6pY
| alphazard wrote:
| Programming a GPU in 2025 is complex, that might be because it
| has been made complicated, but regardless, it is not complexity
| that this project can control.
|
| The fact that it competes with PyTorch in so few lines speaks
| to the incredibly low incidental complexity imposed by
| Tinygrad.
| fragmede wrote:
| If you think you can do better, the code's right there and
| you're welcome to try! There's a giant pile of money at the end
| of the rainbow for you if you can uncomplexify it.
|
| (Love that talk tho)
| alphazard wrote:
| > To fund the operation, we have a computer sales division that
| makes about $2M revenue a year.
|
| What's the margin on that? Do 5 software engineers really subsist
| on the spread from moving $2M/yr in hardware?
| piskov wrote:
| George raised $5.1M in 2023 for Tinygrad
| vessenes wrote:
| Looking at their boxes, I'd guess 20-ish%, once the labor costs
| of designing the boxes is out of the picture. I imagine George
| doesn't take a salary, or at least takes only a small one. They
| have a contract with AMD as well. I'd bet the company is
| modestly profitable.
| piskov wrote:
| > tinygrad is following the Elon process for software. Make the
| requirements less dumb. The best part is no part.
|
| That's not Elon. See Russian TRIZ
|
| https://en.wikipedia.org/wiki/TRIZ
| gessha wrote:
| Not sure where it was from but I remember reading a comment
| about how every modern rocket scientist needs to come to terms
| with the fact that every novel idea they might have has already
| been proposed and tried by a Russian scientist in the 20th
| century.
| wiseowise wrote:
| Soviet scientist, let me correct you a bit, comrade.
| geremiiah wrote:
| The risk for Tinygrad is that PyTorch will create a new backend
| for Inductor, plug in their AMD codegen stuff and walala, PyTorch
| still king. I mean, they could have easily just taken that route
| themselves instead of bothering with a new ML framework and AD
| engine. 99% of the work is just the AMD codegen part of the
| compiler.
|
| Either way, super cool project and I wish them the best.
| ellis0n wrote:
| The main risk is that an LLM will rewrite itself and
| programmers will no longer be needed. I worked a bit with
| tinygrad and it looks quite amusing I managed to run it right
| away and make fixes in one of the tasks, but I decided not to
| commit because I was afraid of rejection. For example, the
| tasks are strange: $500 for two months, optimizing H.265,
| something that only a small group of people in the world can
| do.
|
| The SV is a unique place where you can meet Geo and get $5M,
| maintain a bunch of hardware, build a framework in 20,000 LOC
| and everything works well.
| peter_d_sherman wrote:
| >"We also have a contract with AMD to get MI350X on MLPerf for
| Llama 405B training."
|
| Anything to help AMD (and potentially other GPU/NPU/IPU etc. chip
| makers) catch up with NVidia/CUDA is potentially worth money,
| potentially worth a lot of money, potentially worth up to
| _Billion$_...
|
| Why?
|
| If we have
|
| a) Market worth _Billion$_
|
| and
|
| b) A _competitive_ race in that Market...
|
| then
|
| c) We have _VALUE_ in _anything_ (product, service, ?, ???) that
| helps any given participant capture more of that market than
| their competitors...
|
| (AMD (and the other lesser known GPU/NPU/IPU etc. chip vendors)
| are currently lagging behind NVidia's CUDA AI market dominance --
| so anything that helps the others advance in this area should,
| generally speaking, be beneficial for all technology users in
| general, and be potentially profitable (if the correct deals
| could be struck!) by those that have the skills to do such
| assisting...)
|
| Anyway, wishing you well in your endeavors, Tinygrad!
| semiquaver wrote:
| Is this the guy who talked a big game about all the things he was
| going to fix at Twitter, then utterly failed when confronted with
| a real world codebase and gave up having done nothing of use?
| piskov wrote:
| He left after realizing nothing was going to change (not like
| he's in the leadership).
|
| Also half-joked how the good food went away.
|
| --
|
| George is many things but not a quitter (see comma ai for
| example).
|
| If someone could pull this, it's him due to "never give up,
| never surrender" attitude.
|
| The shit with nvidia just needs to stop
| rent0pat wrote:
| I've looked at the "only 18,935 lines of code" python code and it
| made me want to poke my eyes out. Not sure what's the point of
| this extreme code-golfing.
| Barrin92 wrote:
| yes, it's really crazy, if people think you're exaggerating
| look at this:
|
| https://github.com/tinygrad/tinygrad/blob/master/tinygrad/co...
|
| One of the cases why I think obsession with lines of code is
| one of the most counterproductive metrics, it always produces
| code like this.
| nighthawk454 wrote:
| That doesn't look super awful to me? Hardly extreme code
| golfing.
|
| The far more interesting part is the order of magnitude. If
| they can pull off a 20k LOC with zero dependencies (implying
| a pretty concise project size) and it still works well on
| meaningful applications, that's pretty neat. A 1000x
| reduction in code size and matching/exceeding in perf is
| worth looking at. Probably also implying a better
| architecture as code golf isn't gonna get you 1000x less
| code. Again - their claims not mine, so we'll see.
|
| But at that point they can triple the LOC to 60k with nothing
| but white space, new lines, and comments, for all I care. It
| won't even add a zero.
| abacadaba wrote:
| it looks dense but perfectly readable. arguably more
| readable that way than if it had a bunch of extra new
| lines, definitions, and code blocks spreading the logic out
| into a larger visible area.
| chubot wrote:
| It's not crazy at all, but personally I like simple code that
| flows down the page more, not across
| octoberfranklin wrote:
| It's two-dimensional code, not one-dimensional code.
|
| Declarations flow down the page, definitions flow across.
| vovavili wrote:
| Running a formatter on that codebase would probably at least
| triple it.
| octoberfranklin wrote:
| Looks fine to me.
|
| I'm going to guess that you're used to codebases that use an
| autoformatter.
|
| There's definitely a case to be made for autoformatters in
| projects with a very large (>100) number of developers, but
| the price you pay for that is much poorer information density
| per screenful of code.
|
| For things like tinygrad or sqlite with a small dev team it's
| possible to use whitespace, style, and formatting to
| communicate important information, just like mathematicians
| do. The AMS journals don't autoformat mathematicians'
| equations!
| eichin wrote:
| to quote an old academic research programmer: "Two-letter
| variable names are fine, as long as both letters are
| meaningful" :-)
| random9749832 wrote:
| If you want to "own" Nvidia, the much more realistic way of doing
| this then trying to compete with all the data centers that are
| already being built with Nvidia chips is obviously with open
| source models. In the case of open source models, inference is
| much more important to most people not training which a maxed out
| macbook already does a good job of.
| agentifysh wrote:
| what have people built with tinygrad?
| piskov wrote:
| Comma ai
| pehtran wrote:
| How are ergonomics compared to pytorch, though? Adoption can be
| also driven by frictionless research (e.g. torch vs. tf comes to
| mind). Repo is missing proper docs aimed at early adopters imho
| Tepix wrote:
| What happened to the tinybox red (v1)? It had _way_ better specs
| than red v2.
___________________________________________________________________
(page generated 2025-12-30 23:00 UTC)