[HN Gopher] GPT in 500 Lines of SQL
___________________________________________________________________
GPT in 500 Lines of SQL
Author : thunderbong
Score : 1056 points
Date : 2024-02-24 02:45 UTC (1 days ago)
(HTM) web link (explainextended.com)
(TXT) w3m dump (explainextended.com)
| seasonalnull wrote:
| This should be illegal
| fifilura wrote:
| Can you elaborate?
| behnamoh wrote:
| What is this sorcery?
| jer0me wrote:
| Related:
|
| _A GPT in 60 Lines of NumPy_ -
| https://news.ycombinator.com/item?id=34726115 - February 2023
| (146 comments)
| sunnybeetroot wrote:
| This is mentioned in the article near the top.
| chrsig wrote:
| This is beautiful. I'd actually been going down this same rabbit
| hole with sqlite, I hadn't gotten far enough to bring a neural
| net into it.
|
| I'd been inspired by the makemore lecture series[0]. At the 1hr
| mark or so, he switches from counting to using a nn, which is
| about as far as I've gotten. Breaking it down into a relational
| model is actually a really great exercise.
|
| [0] https://www.youtube.com/watch?v=PaCmpygFfXo
| pests wrote:
| If you keep watching you will see the NN derive the same exact
| table as the counting method and even gives the exact same
| results when generating.
| brainless wrote:
| I think, I _think_ , GPT creating GPT... creating GPT will be a
| thing soon. GPTception.
| incahoots wrote:
| Personally I like the AI dogman angle. AI trained to beat other
| AI (resumes tailored to beat ATS algorithms)
| codetiger wrote:
| GPT creating a better algo than itself is what's even more
| interesting
| mavamaarten wrote:
| It might. But also, all decisions and knowledge is pretty
| much based on a resampling of our own language, conversations
| and internet data. It might puzzle together some existing
| ideas that have never been combined. Or it might hallucinate
| a better solution by accident. But we're definitely not at
| the level yet where it will actively build something great.
| erwincoumans wrote:
| Self-play GPT (by bots in a rich simulation) similar to
| Alpha Go Zero?
| codetiger wrote:
| Games like Alpha Go have very limited(or known) end state
| so reinforcements learning or similar methods work great.
| However, I wonder how will AI train itself in learning
| human languages without being judged by humans. It's just
| a matter of time before someone figures out
| erwincoumans wrote:
| Right, a rich simulator with humans for feedback: an
| evolved version of online worlds with a mix of AI NPC's
| and real people, with the task: find the NPC's. The NPC's
| can train in rooms with exclusive NPC's or mixed with
| people, without knowing.
| wizzwizz4 wrote:
| Self-play works for Go, because the "world" (for lack of
| a better term) can be fully simulated. Human language
| talks about the _real_ world, which we cannot simulate,
| so self-play wouldn 't be able to learn new things about
| the world.
|
| We might end up with more regularised language, and a
| more _consistent_ model of the world, but that would come
| at the expense of accuracy and faithfulness (two things
| which are already lacking).
| slt2021 wrote:
| I can feel the SQL-Force in this young jedi, Midichlorian level
| is on another level
| swasheck wrote:
| alex is a genius. he's worth a follow.
| B1FF_PSUVM wrote:
| "It's full of jewels", as someone almost said.
|
| E.g. The Sultan's Riddle in SQL
| https://explainextended.com/2016/12/31/happy-new-year-8/
| hsbauauvhabzb wrote:
| I've completely avoided GPT and LLMs. This looks like it would
| generate some level of fluidity in text output, but not be able
| to parse and answer a question.
|
| Is there any simplistic blog posts / training courses which go
| through how they work, or expose a toy engine in python or
| similar that? All the training I've seen so far seems oriented at
| how to use the platforms rather than how they actually work.
| zaptrem wrote:
| Strap in, this is by far the best resource:
| https://www.youtube.com/watch?v=kCc8FmEb1nY
| ford wrote:
| Jay Alammar has my favorite sequence of tutorials from basic
| neural network math to GPT2.
|
| Particularly [0], [1], and [2]
|
| [0] http://jalammar.github.io/illustrated-transformer/
|
| [1] http://jalammar.github.io/illustrated-gpt2/
|
| [2] https://jalammar.github.io/visualizing-neural-machine-
| transl...
| ianand wrote:
| This is great. In a similar vein, I implemented GPT entirely in
| spreadsheet functions with accompanying video tutorials
|
| https://spreadsheets-are-all-you-need.ai/
| danielmarkbruce wrote:
| Nice job. Spreadsheets are a natural way to explain an LLM. I
| suspect that you can show training well too by calculating the
| derivatives for each parameter under each training example and
| showing it all explicitly mapped to the relevant parameter etc
| etc
| ianand wrote:
| Thank you. Validating to hear others feel spreadsheets are a
| natural and accessible way to explain LLMs. Someone asks
| "what's do they mean by a parameter in a model" or "what is
| the attention matrix" and you can just pull it up graphically
| laid out. Then they can fiddle with it and get a visceral
| feel for things. It also becomes easier for non coders do
| things like logit lens which is just a few extra simple
| spreadsheet functions.
|
| I actually plan to do what you describe after I do the
| embeddings video (but only for a "toy" neural net as a proof-
| of-concept introduction to gradient descent).
| airstrike wrote:
| Not only is that amazing, your video was so well done. Superb
| crowd work! Color me double impressed.
| ianand wrote:
| Thanks! Each one takes a surprisingly long time to make. Even
| figuring out how make the explanation accessible and
| compelling yet still accurate takes awhile and then there's
| still the actual video to do.
| alisonatwork wrote:
| Your first video is fantastic. As someone who thinks LLMs are
| pretty nifty but never had a professional need to learn how
| they actually work, that 10 minute video taught me more than
| several years of reading esoteric HN comments and fluffy
| mainstream media on the topic. Seeing a bazillion floating
| point numbers all stacked up ready to be calculated across also
| makes it much more intuitive why this tech devours GPUs, which
| never really occurred to me before. Thanks for sharing.
| int_19h wrote:
| Great approach and great delivery; looking forward to the next
| video in the series!
| wanderingmind wrote:
| These marvels need to be preserved. Just posting the archive link
| here in case the blog is not maintained in future.
|
| https://archive.is/VAGzF
| rawgabbit wrote:
| This is very cool
| zarkenfrood wrote:
| Thanks, this is a fantastic article and it would be a shame to
| be lost.
| neonate wrote:
| and
| https://web.archive.org/web/20240224015308/https://explainex...
| lagniappe wrote:
| This is a great read, I didn't expect to scroll right back to the
| top as soon as I finished it the first time.
| namnhocq8 wrote:
| Tai xiu
| ksarw wrote:
| Great write up, I enjoyed the reading the explanations for each
| piece and found them to be clear and quite thorough.
|
| I did make the mistake though of clicking "+ expand source", and
| after seeing the (remarkable) abomination I can sympathize with
| ChatGPT's "SQL is not suitable for implementing large language
| model..." :)
| deskamess wrote:
| I did that too and could not find a way to collapse it.
| sigmoid10 wrote:
| It's a nice demo. Unfortunately the article mixes up things in
| the explanation for causal masking, because it seems the author
| conflates aspects from training and inference. First, causal
| masking exists to prevent the model from "peeking" at future
| tokens during training, and second (at least for GPT-like
| architectures) for enforcing the autoregressive aspect during
| inference. During inference we only use the last token anyways,
| so it will attend the entire input sequence. So this token is
| definitely not decided only from the last token's embedding.
| jakjak123 wrote:
| This is a very good article and introduction.
| Alifatisk wrote:
| I love this, something that starts off as some kind of sorcery a
| year ago is now being explained so well and almost in a childish
| way.
| maelito wrote:
| > and almost in a childish way.
|
| No. You've got to have a solid background in computer science
| to even start to understand fully this article.
|
| Even the title itself is not accessible to 99% of humans.
| bookofjoe wrote:
| Count me in as one of the 99%
| coldtea wrote:
| Parent obviously means childish as in "simple and easy
| manner" not as "something 10 year olds will understand".
| pedrosorio wrote:
| This sorcery didn't start a year ago. The model being described
| in the article is GPT-2 which was released in early 2019.
| huqedato wrote:
| Fantastic article. It kept my eyes on the screen for 2 hours,
| without interruption The author is a genius.
| Hendrikto wrote:
| > Plain Unicode, however, doesn't really work well with neural
| networks.
|
| That is not true. See ByT5, for example.
|
| > As an illustration, let's take the word "PostgreSQL". If we
| were to encode it (convert to an array of numbers) using Unicode,
| we would get 10 numbers that could potentially be from 1 to
| 149186. It means that our neural network would need to store a
| matrix with 149186 rows in it and perform a number of
| calculations on 10 rows from this matrix.
|
| What the author calls alphabet here, is typically called
| vocabulary. And you can just use UTF-8 bytes as your vocabulary,
| so you end up with 256 tokens, not 149186. That is what ByT5
| does.
| int_19h wrote:
| The point isn't that it doesn't work at all, but that it
| doesn't work as well as other approaches that we have. Which is
| evidenced by the fact that all the best-performing models on
| the market use tokenization. It's not a secret that
| tokenization is fundamentally a hack, and that ideally we'll
| get rid of it eventually one way or another
| (https://twitter.com/karpathy/status/1657949234535211009). And
| in principle, you can compensate for the deficiencies of byte-
| level tokenization with larger models and larger contexts. But
| what this means in practice is that a model that has the same
| level of intelligence (on most tasks; obviously, there are some
| specific tasks, like say counting characters in a word, where
| tokenization is detrimental to intelligence) takes a lot more
| resources to train, hence why we aren't seeing more of those.
| deskamess wrote:
| In the tokenization example for '"Mississippilessly", why is 'si'
| not part of the combined column? It appears twice in the text. My
| speculation is that it got collapsed out with 'iss' (a longer
| token). Is that right?
| remram wrote:
| Yes. At step 1 there are two 'i','s' and two 's','i', 'is' gets
| picked because it comes first. Then at step 7, because 'is' got
| formed, there are two instances of 'is','s' and only one
| instance of 's','i' so 'iss' gets formed.
| deskamess wrote:
| Thanks.
| cdelsolar wrote:
| Serious question, how do I get this smart?
| anthomtb wrote:
| No doubt the author is a super genius ex-child prodigy whizzkid
| who emerged from the womb with the umbilical cord diagramming
| Euler's formula.
|
| For real though, and knowing this is a leading question, the
| author has near-on 15 years of blog posts showing complex
| problems being solved in SQL. Is their brain bigger than yours
| and mine? Maybe a little bit. Do they have a _ton_ of
| experience doing things like this? Most definitely.
| fifilura wrote:
| Also remember he builds these things piece by piece from the
| inside out.
|
| It is easy to get overwhelmed when looking at the end result.
|
| To learn you should probably run one block at a time to
| understand what each piece does. (In "normal" languages those
| pieces would be an isolated function)
| JonChesterfield wrote:
| Is this an accurate representation of the GPT driver loop?
| def generate(prompt: str) -> str: # Transforms a string
| into a list of tokens. tokens = tokenize(prompt) #
| tokenize(prompt: str) -> list[int] while True:
| # Runs the algorithm. # Returns tokens'
| probabilities: a list of 50257 floats, adding up to 1.
| candidates = gpt2(tokens) # gpt2(tokens: list[int]) ->
| list[float] # Selects the next token from
| the list of candidates next_token =
| select_next_token(candidates) #
| select_next_token(candidates: list[float]) -> int
| # Append it to the list of tokens
| tokens.append(next_token) # Decide if we
| want to stop generating. # It can be token counter,
| timeout, stopword or something else. if
| should_stop_generating(): break #
| Transform the list of tokens into a string completion =
| detokenize(tokens) # detokenize(tokens: list[int]) -> str
| return completion
|
| because that looks a lot like a state machine implementing
| Shlemiel the painter's algorithm which throws doubt on the
| intrinsic compute cost of the generative exercise.
| NortySpock wrote:
| I think the "context window" that people refer to with large
| language models means there's a maximum number of tokens that
| are retained, with the oldest being discarded. The window is a
| sliding window.
| canjobear wrote:
| Yes, that is the loop. All the magic is in the gpt2 function
| there.
| creatonez wrote:
| This is a very small section of the algorithm. This is just how
| it collects the tokens it has generated into a sentence.
| ein0p wrote:
| Unexpectedly insightful and answers some of the same questions I
| had early on: not just "how" questions, but "why" as well. You
| see the pattern with softmax quite often. I wish it was taught as
| "differentiable argmax" rather than by giving people a formula
| straight away. That's not all it is, but that's how it's often
| used.
| 101008 wrote:
| I keep reading that GPT is a "smarter" "complex" Markov, in the
| end, just a function spitting out next word with some
| probability.
|
| But from my experience that cannot be true - it has to learn
| somehow. There is an easy example to make. Tell it something that
| happened today and contradicts the past (I used to test this with
| the Qatar World Cup), and then ask questions that are affected by
| that event, and it replied correctly. How is that possible? How a
| simple sentence (the information I provide) changes the
| probabilites for next token by that far?
| Lerc wrote:
| There are two parts of the knowledge at play here.
|
| 1. The trained knowledge included in the parameters of the
| model
|
| 2. The context of the conversation
|
| The 'learning' you are experiencing here is due to the
| conversation context retaining the new facts. Historically the
| context windows were very short and as the conversation
| continues it would quickly forget the new facts.
|
| More recently context windows have grown to rather massive
| lengths.
| pests wrote:
| Mostly #2 of what the other response says. The entire input to
| the NN is what it's learning off - the weights are static, you
| are changing the probabilities based on which context is
| provided and it's length.
| drexlspivey wrote:
| Markov chains also depend on the initial state. In GPTs the
| initial state is your conversation history
| Scene_Cast2 wrote:
| Interestingly, modern ML does not require Turing completeness.
| And yet, people are considering the possibility of AGI - I would
| find it pretty amusing if Turing completeness isn't necessary.
| DennisP wrote:
| Seems to me that Turing completeness is necessary, for the
| simple reason that I can mentally trace through Turing-complete
| code.
| r-w wrote:
| Without a piece of paper to take notes on? ;)
| DennisP wrote:
| My own memory functions as the paper tape as long as the
| program is simple enough :)
| int_19h wrote:
| Token inference by itself isn't Turing complete, but if its
| output can have side effects (e.g. editing the prompt for the
| next iteration), that's a whole different story.
| mikeytown2 wrote:
| There a GitHub page for this?
| pajop wrote:
| Here you go: https://github.com/quassnoi/explain-extended-2024
| haolez wrote:
| Is there a world where this can be made performant? Just my geek
| curiosity kicking in, as I love Postgres and SQL.
| firemelt wrote:
| laugh at people who overhyped it with singularity agi and skynet
| lmao
___________________________________________________________________
(page generated 2024-02-25 23:02 UTC)