[HN Gopher] Paper Tape Is All You Need - Training a Transformer ...
___________________________________________________________________
Paper Tape Is All You Need - Training a Transformer on a 1976
Minicomputer
Author : rahen
Score : 127 points
Date : 2026-03-25 15:20 UTC (3 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| arglebarnacle wrote:
| Fascinating. We hear that the leaps in AI have been made possible
| by orders of magnitude increases in compute and data
| availability, and of course that's substantially true--but
| exactly how true? It's a nice exercise in perspective to see how
| much or how little modern machine learning methods would have
| been capable of if you brought them by time machine to the 70's
| and optimized them for that environment.
| AnimalMuppet wrote:
| Woah. Dude has a running PDP-11/34 in 2026? Personally, I find
| that more impressive than the program.
| rahen wrote:
| That thing is a Tamagochi though, it constantly needs
| attention, pardon the pun. I did most of the development and
| tuning on ll-34 for that reason.
| budman1 wrote:
| I am a bit surprised, but I guess everything eventually wears
| out.
|
| In the 1980's I worked as a field engineer that supported a
| lot of pdp-11's. They were very reliable for the time; tape
| drives and disks were the #1 maintenance items. To actually
| have to open up the processor and change a board was not a
| regular activity.
|
| Other machines of that era, like those from Gould or
| Perkin/Elmer or DG gave regular practice in the art of
| repairing processors.
|
| Guess I expect them to work forever. Like a Toyota.
| rahen wrote:
| I encouter two main failure modes. First, the bipolar PROMs
| degrade at the atomic level, the metal ions in the fuses
| tend to migrate or 'regrow' over decades, causing bit rot.
| Second, the backplanes suffer from mechanical fatigue.
| After forty years of thermal expansion and structural
| flexing, especially when inserting boards, the traces and
| solder joints develop stress cracks. Both are a pain to
| repair.
|
| https://retrocmp.com/articles/trying-to-fix-a-dec-
| pdp-1134-b...
| budman1 wrote:
| Excellent work.
|
| The feeling of accomplishment when the machine boots
| after a major repair (almost) makes it all worth while.
|
| (i think i would have found a used backplane...fixing it
| was crazy clever)
| adrian_b wrote:
| Not only that, but the author has also written a cycle-accurate
| PDP-11/34 simulator for the benefit of those who do not have
| such hardware.
|
| https://github.com/dbrll/ll-34
| rahen wrote:
| The WASM GUI is probably the easiest way to see the
| Transformer in action on this machine:
| https://dbrll.github.io/ll-34/
|
| There's also the original Tetris from 1984 to play.
| rahen wrote:
| Thanks for reposting! I'm the author of ATTN-11. Happy to answer
| any questions about the fixed-point arithmetic, the PDP-11
| hardware, or the training process.
| functional_dev wrote:
| Incredible work! Fitting transformer into 32KB RAM is crazy
|
| For those who read this project and do not know PDP-11 it could
| be hard to understand that working with these memory limits is
| difficult. Here is visual guide for PDP11 architecture -
| https://vectree.io/c/pdp-11-hardware-architecture
|
| Thanks for this amazing project!
| PaulHoule wrote:
| That PDP-11 was the most fun minicomputer of the late 1970s
| in my opinion. Growing up in NH about an hour north of
| Digital's HQ all sorts of schools from primary to secondary
| as well as museums had PDP-8, PDP-10, PDP-11 and later VAX
| machines.
|
| The PDP-11 had a timesharing OS called RSTS/E which could
| give maybe 10 people a BASIC programming experience a little
| bit better than an Apple ][. If you were messing with 8-bit
| microcomputers in 1981 you might think a 16-bit future would
| look like the PDP-11 but the 1970 design was long in the
| tooth by 1980 -- like 8-bit micros it was limited to a 64kb
| logical address space. Virtual memory let it offer 64k
| environments to more users, but not let a user have a bigger
| environment.
| dare944 wrote:
| Fun stuff! At one point I wondered about building something
| similar. But I lack the AI chops, and have too many other
| projects going on anyway.
|
| I'm curious as to the type of memory in the 11/34. I also have
| a working PDP-11, an 11/05 with 32KW of actual core. I wonder
| what performance would be like with EIS emulation grafted in.
| Stunningly slow, I imagine.
|
| Thanks for publishing this.
| McGlockenshire wrote:
| Thank you for the inspiration, I now have a practical-
| impractical assembly project for my TI TMS99105A homebrew! The
| 64k barrier is a real pain.
| rahen wrote:
| I also have a working design for a small Transformer on the
| original Game Boy. It has around 4000 parameters fitting in
| the 8 KB cartridge SRAM, where the "saved game" is the
| trained model. A TI-82 with its 32 KB of RAM would be even
| more comfortable.
| kristopolous wrote:
| I like how the author's "modern" machine to connect to it is
| still 20 years old.
|
| With a concave trackpoint, respect.
|
| BTW, I nag Framework at every conference I go to that people want
| this shell and keyboard. It's been years. I think it's time to go
| through the effort to figure out how to do the production run of
| the case myself. Framework actually wants people to do things
| like this but you know, manufacturing is hard. Anyone wanna help?
| kmoser wrote:
| > I don't have an actual paper tape reader, so the object code is
| directly deposited in memory through the console.
|
| So, really, a Turing Machine is all you need?
| thyrsus wrote:
| I dealt with physical paper tape on only three or four
| occasions in the early 1980's, each time terrified of a jam or
| tear. It seems in this case it's a read-once operation, which
| is plausible. Read-many, not so much. Punch cards are orders of
| magnitude more reliable.
| ashwinnair99 wrote:
| The fact that it is possible at all says more about how simple
| transformers actually are underneath than it does about the
| hardware.
| mkagenius wrote:
| I had the exact same idea but for AI agent harnesses.
|
| I even created an app to explain it -
| https://news.ycombinator.com/item?id=47381803 (deleted the app as
| got no traction whatsoever)
|
| Idea was that, the ai models like opus 4.6 and codex 5.4 have
| become so good at trying new ways to attack a problem, that even
| just Bash() tool is enough.
|
| Continuing the idea, infact even File() operations are enough.
|
| Again continuing the same line of thought, even just a Tape is
| enough. Given enough time, codex and opus will achieve your
| target.
| tcdent wrote:
| 5.5 min to train on a PDP/11 you mean to tell me we could have
| been doing this all along???
| rahen wrote:
| Yes. The Cray supercomputers from the 80s were crazy good
| matmul machines in particular. The quad-CPU Cray X-MP (1984)
| could sustain 800 MFLOPS to 1 GFLOPS, and with a 1 GB SSD, had
| enough computer power and bandwidth to train a 7-10M-parameter
| language model in about six months, and infer at 18-25 tok/sec.
|
| A mid-90s Cray T3E could have handled GPT-2 124M, 24 years
| before OpenAI.
|
| I also had a punch-card computer from 1965 learn XOR with
| backpropagation.
|
| The hardware was never the bottleneck, the ideas were.
___________________________________________________________________
(page generated 2026-03-28 23:00 UTC)