[HN Gopher] Petaflops to the People: From Personal Compute Clust...
___________________________________________________________________
Petaflops to the People: From Personal Compute Cluster to Person of
Compute
Author : tosh
Score : 52 points
Date : 2023-06-20 17:16 UTC (5 hours ago)
(HTM) web link (www.latent.space)
(TXT) w3m dump (www.latent.space)
| croes wrote:
| I see George Hotz and I think another interesting idea that gets
| another half-baked implementation.
| jxy wrote:
| They are clearly scared of ggml.
| swyx wrote:
| thanks for posting tosh!
|
| we were really honored that George granted us the first interview
| on tiny corp. We even set up a dedicated podcast youtube for the
| first time (Alessio and I are relative youtube newbs) and George
| was kind enough to come back to Twitter just to help us kick it
| off:
| https://twitter.com/realGeorgeHotz/status/167120786704580608...
|
| Personal highlights for the HN crowd:
|
| - the three theses of tiny are all interesting: the CISC/RISC
| analogy, why you should start with software for commodity GPUs
| rather than taping out chip design, why to avoid Turing complete
| layers in the ML stack
|
| - I loved comparing his natural debugging and CI style to John
| Carmack - developers who assume geohot is a cowboy coder might be
| surprised at how seriously he takes developer experience
|
| - Dishing on tinygrad vs ggml and Mojo (i'll just leave it there,
| its good)
|
| - how everyone is in this mad quantization rush but nobody's
| putting up benchmarks to show that it works (tinybox is
| resolutely supporting non quantized LLaMA)
|
| - hardware design constraint choices for tinybox (<60 decibels,
| single power plug, etc)
|
| and then we of course indulged in the array of offtopic convos
| for fan service:
|
| - GPT4 is just a mixture of experts - 8 GPT's in a trenchcoat
|
| - Elon vs e/acc
|
| - Rich Sutton's Bitter Lesson
|
| - The Goddess of Everything Else (go listen to him read it
| https://mixtape.swyx.io/episodes/the-goddess-of-everything-e...)
|
| and of course, he's doing all this so that you'll consider
| working on/for tiny. job application details on
| https://tinygrad.org/
| swyx wrote:
| personal reflections from interviewing him:
|
| - Idk how to say this but surprisingly not arrogant. we both
| prepped by watching his lex fridman and prior livestreams, so
| its not like we were dealing with an unknown quantity, but I've
| never seen him interact with "normal" (technical, but not his
| level of technical) people like us before. he was pretty
| patient I think, except I definitely got roasted for suggesting
| you could jailbreak the tinybox. instantly regretted it but he
| only joked once aboout it which was super cool :)
|
| - quite reasonable expectations for his AMD frustrations. the
| fact that Lisa Su got personally involved and they listened was
| also fantastic. this isn't a "tech diva flipping the table
| because they stubbed their toe" story - AMD was found wanting
| in basic documentation requirements, he raised the issue, they
| took a while but eventually fixed it. Great partnership all
| around imo
|
| - hiring in the age of AI is really going to get strange. he
| thinks we're ~20 years out from developer jobs being gone. Yes
| he is essentially asking people to do work for small bounties.
| in some circles, that's a no go. but it basically guarantees
| that the candidate will be a fit, and is fine if you want to
| keep the team tiny anyway. he also changed his mind on remote
| work which is cool.
|
| - this was the first time i heard anything materially different
| about GPT4 architecture being a mixture of experts rather than
| a 1 trillion super model. does raise the question about what
| GPT5 will be built on if we have indeed run into the limits of
| scaling model size.
|
| - notes to future interviewers: george is ultra open (he
| identifies strongly as a streamer, is very comfortable with
| what he is saying and doesnt need a ton of prep or post
| recording review) - just be super attentive to what he does and
| does not want to talk about and let him riff and he will
| indulge. i think we were maybe like a 7 out of 10 on getting
| the best out of him, but hopefully we will be a lot better by
| the time we go down to San Diego for the tinybox launch
| :fingerscrossed:
|
| - He gave us advice on whether or not we wanted to go big and
| be the next Lex Fridman or stay niche and I think we were
| pretty convicted on wanting to stay niche and technical. To my
| knowledge there's no other podcast in the world that could have
| had this deep-but-still-kinda-accessible conversation with
| George and I strongly feel like something like this has to
| exist.
| lhl wrote:
| > how everyone is in this mad quantization rush but nobody's
| putting up benchmarks to show that it works (tinybox is
| resolutely supporting non quantized LLaMA)
|
| I don't think this is true. llama.cpp has historically been
| very conscientious about benchmarking perplexity. Here's a
| detailed chart of baseline FP16 vs the new k-quants:
| https://github.com/ggerganov/llama.cpp/pull/1684
|
| While most evals aren't currently evaluating performance
| between quantized models, there are two evals that are:
|
| * Gotzmann LLM Score:
| https://docs.google.com/spreadsheets/d/1ikqqIaptv2P4_15Ytzro...
|
| * llm-jeopardy: https://github.com/aigoopy/llm-jeopardy - You
| can see that the same Airoboros 65B model goes from a score of
| 81.62% to 80.00% going from an 8_0 to 5_1 quant, and 5_1
| solidly beats out the 33B 8_0, as expected.
|
| Also, GPTQ, SPQR, AWQ, SqueezeLLM all have arXiv papers and
| every single team is running their own perplexity tests.
|
| Now, that being said, every code base seems to be calculating
| perplexity _slightly_ differently. I recently have been working
| on trying to decode them all for apples-to-apples comparisons
| between implementations.
| georgehotz wrote:
| I question if perplexity on wikitext is enough, I wonder if
| someone has checked how well perplexity correlates with
| https://github.com/google/BIG-bench
|
| Fundamentally, I'm not sure if perplexity on a dataset that's
| pretty much leaked everywhere will properly measure the task
| performance of LLMs. But open to being wrong on this.
| lhl wrote:
| Forgot to mention that Fabrice Bellard has run lm-eval on a
| range of functional tests (lambada, hellaswag, winogrande,
| piqa, coqa) for q4, q8 (and unquantized on smaller) models:
| https://bellard.org/ts_server/
|
| Losses appear to be in-line (possibly less) than increases
| in perplexity.
|
| My intuition is that the dataset you use for testing
| perplexity (c4, wikitext2, wikitext103, whatever) shouldn't
| matter so much as long as its a representative sample of
| data it's read. You're calculating cross entropy loss, so
| you _want_ to have data you know it 's read to see how much
| each quantization gets more wrong.
___________________________________________________________________
(page generated 2023-06-20 23:02 UTC)