[HN Gopher] Transformers from Scratch (2021)
___________________________________________________________________
Transformers from Scratch (2021)
Author : jasim
Score : 535 points
Date : 2023-04-25 07:42 UTC (1 days ago)
(HTM) web link (e2eml.school)
(TXT) w3m dump (e2eml.school)
| toyg wrote:
| MORE THAN MEETS THE EYE!
|
| ... oh, not _those_ Transformers. Meh.
| zabzonk wrote:
| that was my first glance reading too - shape-changing toys
| written in the scratch language! how cool could that be?
| MisterTea wrote:
| I was hoping it was an article on designing and building an
| electrical transformer complete with pictures of a home made,
| hand wound transformer. I was very disappointed.
| sachinkalsi wrote:
| Check this out https://youtu.be/73gTEub2e3I
| dsubburam wrote:
| An early explainer of transformers, which is a quicker read, that
| I found very useful when they were still new to me, is The
| Illustrated Transformer[1], by Jay Alammar.
|
| A more recent academic but high-level explanation of
| transformers, very good for detail on the different flow flavors
| (e.g. encoder-decoder vs decoder only), is Formal Algorithms for
| Transformers[2], from DeepMind.
|
| [1] https://jalammar.github.io/illustrated-transformer/ [2]
| https://arxiv.org/abs/2207.09238
| behnamoh wrote:
| Can anyone please say how much value there is in learning the
| fundamentals of LLMs for someone who uses them in practice?
| m3kw9 wrote:
| A little understand would give you why certain prompts work
| or don't work. A high level should do. It could help you make
| better prompts or trouble shoot, although you don't need it
| for 80% of the cases
| driscoll42 wrote:
| The Illustrated Transformer is fantastic, but I would suggest
| that those going into it really should read the previous
| articles in the series to get a foundation to understand it
| more, plus later articles that go into GPT and BERT, here's the
| list:
|
| A Visual and Interactive Guide to the Basics of Neural Networks
| - https://jalammar.github.io/visual-interactive-guide-
| basics-n...
|
| A Visual And Interactive Look at Basic Neural Network Math -
| https://jalammar.github.io/feedforward-neural-networks-visua...
|
| Visualizing A Neural Machine Translation Model (Mechanics of
| Seq2seq Models With Attention) -
| https://jalammar.github.io/visualizing-neural-machine-transl...
|
| The Illustrated Transformer -
| https://jalammar.github.io/illustrated-transformer/
|
| The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer
| Learning) - https://jalammar.github.io/illustrated-bert/
|
| The Illustrated GPT-2 (Visualizing Transformer Language Models)
| - https://jalammar.github.io/illustrated-gpt2/
|
| How GPT3 Works - Visualizations and Animations -
| https://jalammar.github.io/how-gpt3-works-visualizations-ani...
|
| The Illustrated Retrieval Transformer -
| https://jalammar.github.io/illustrated-retrieval-transformer...
|
| The Illustrated Stable Diffusion -
| https://jalammar.github.io/illustrated-stable-diffusion/
|
| If you want to learn how to code them, this book is great:
| https://d2l.ai/chapter_attention-mechanisms-and-transformers...
| mkw5053 wrote:
| Thanks for sharing!
| noobcoder wrote:
| I remember looking into this article. It was really helpful for
| me to understand transformers. Although the OP's article is
| detailed, this one is concise. Here's the link: https://blue-
| season.github.io/transformer-in-5-minutes
| JackFr wrote:
| Read this as "Transformers in Scratch" at first and was _very_
| curious.
|
| Obviously implementing transformers in Scratch is likely
| impossible, but has anyone built a Scratch-like environment for
| building NN models?
| cuuupid wrote:
| This is cool, I highly recommend Jay Alammar's Illustrated
| Transformer series to anyone wanting to get an understanding of
| the different types of transformers and how self-attention works.
|
| The math behind self-attention is also cool and easy to extend to
| e.g. dual attention
| bambax wrote:
| [flagged]
| erwincoumans wrote:
| Andrej Karpathy's 2 hour video and code is really good to
| understand the details of Transformers:
|
| "Let's build GPT: from scratch, in code, spelled out."
|
| https://youtube.com/watch?v=kCc8FmEb1nY
| pmoriarty wrote:
| So how practical is learning to create your own transformers if
| you can't afford a giant amount of resources to train them?
| almost wrote:
| Understanding how things work is useful and worthwhile on its
| own. Also while you probably can't afford to train your own LLM
| you probably can afford to fine tune an existing one or to join
| one to another mode or lots of other things like that.
| Reason077 wrote:
| [flagged]
| stared wrote:
| Thank you for sharing!
|
| For the "from scratch" version, I recommend "The GPT-3
| Architecture, on a Napkin"
| https://dugas.ch/artificial_curiosity/GPT_architecture.html,
| which was there as well
| (https://news.ycombinator.com/item?id=33942597).
|
| Then, to actually dive into details, "The Annotated Transformer",
| i.e. a walktrough "Attention Is All You Need", with code in
| PyTorch, https://nlp.seas.harvard.edu/2018/04/03/attention.html.
| ziyunli wrote:
| There is a newer version of the second article
| https://nlp.seas.harvard.edu/annotated-transformer/
| Buttons840 wrote:
| This article describes positional encodings based on several sine
| waves with different frequencies, but I've also seen positional
| "embeddings" used, where the position (the position is an integer
| value) is used to select an differentiable embedding from an
| embedding table. Thus, the model learns its own positional
| encoding. Does anyone know how these compare?
|
| I've also wondered why we add the positional encoding to the
| value, rather than concatenating them?
|
| Also, the terms encoding, embedding, projection, and others are
| all starting to sound the same to me. I'm not sure exactly what
| the difference is. Linear projections start to look like
| embeddings start to look like encodings start to look like
| projections, etc. I guess that's just the nature of linear
| algebra? It's all the same? The data is the computation, and the
| computation is the data. Numbers in, numbers out, and if the
| wrong numbers come out then God help you.
|
| I digress. Is there a distinction between encoding, embedding,
| and projection I should be aware of?
|
| I recently read in "The Little Learner" book that finding the
| right parameters _is_ learning. That 's the point. Everything we
| do in deep learning is focused on choosing the right sequence of
| numbers and we call those numbers _parameters_. Every parameter
| has a specific role in our model. _Parameters_ are our choice,
| those are the nobs that we (as a personified machine learning
| algorithm) get to adjust. Ever since then the word "parameters"
| has been much more meaningful to me. I'm hoping for similar
| clarity with these other words.
| Silverback_VII wrote:
| Also, the terms encoding, embedding, projection, and others are
| all starting to sound the same to me.
|
| Well, projection is used to create the embedding with which the
| symbol is encoded.
|
| It greatly reduces computational cost as the encoding carries
| already a lot of information.
| giovannibonetti wrote:
| > I recently read in "The Little Learner" book that finding the
| right parameters is learning. That's the point. Everything we
| do in deep learning is focused on choosing the right sequence
| of numbers and we call those numbers parameters. Every
| parameter has a specific role in our model. Parameters are our
| choice, those are the nobs that we (as a personified machine
| learning algorithm) get to adjust.
|
| Be careful not to mistake parameters for hyperparameters. -
| Parameters are the result of the training phase, as you
| mentioned. They start with random values and are discovered by
| the training algorithm; - Hyperparameters, on the other hand,
| are the knobs you tweak to make the training process arrive at
| the "right" parameters. You can think of them as meta-
| parameters;
|
| Also, it is important to think on the ML architecture -
| transformers, neural networks, random forests and so on - as
| the parameters change completely depending on which one you're
| using.
| Buttons840 wrote:
| Yes, hyper-parameters are parameters about the parameters.
| Parameters we get to choose which control the parameters that
| the learning algorithm chooses.
|
| The other set of data the book called "arguments", which is
| the term they use to describe the data you are training on.
| That seems like an unnecessarily confusing term, and I
| haven't heard it anywhere else.
|
| I didn't learn anything truly new in all this, but it helped
| me sort my own thoughts to realize there is data, parameters,
| and hyperparameters. Data comes from the world and we cannot
| change it. Parameters are chosen by us indirectly through the
| learning algorithm, they are the most important outcome of
| successful learning. Hyperparameters are chosen by us
| directly and control the model and learning algorithm, and
| the resulting parameters.
| ralusek wrote:
| What I don't understand about positional encoding is why use
| sine waves at all? Sine waves have a weird property of
| "accelerating" and "decelerating," such that the distance
| between two points that are linearly the same distance apart in
| a sentence would result in two very different positional
| encodings relative to one another, just arbitrarily dependent
| on where they were in the curve. I'm sure this is somewhat
| counteracted by the fact that you have lots of these sine waves
| offset and at different frequencies, but it still seems like an
| unnecessary feature.
|
| Wouldn't the positional encoding be better served by encoding
| to positions on a triangle waveform like this:
|
| https://upload.wikimedia.org/wikipedia/commons/thumb/7/77/Wa...
|
| Could still do all of the same tricks of overlaying waves of
| different frequencies, but let the words be much more linearly
| related to one another. I suspect the function to plot to a
| wave like this is something like a modulus operation, where you
| adjust the modulus to produce different frequencies. And a
| normal modulus would get you something like the "sawtooth"
| graph, but if you did it so that you could determine if you
| were in an even or an odd period of the modulus, then you 1 - v
| the even ones and you'd get something like a triangle wave.
| ralusek wrote:
| I'm sure there's an even better positional encoding that can
| get around the ugly feature that both triangle and sine waves
| have of having distinct points where they "change direction."
| I suspect there's a way around this by sampling values from a
| 2 dimensional space while moving in a circle, something like
| that.
| zwaps wrote:
| You might be misunderstanding the use of sin-cos in early
| positional embeddings. The waves are shifted such that each
| position gets a unique positional encoding. These encodings
| are not relative in distance to other positions, they are at
| best ordinal. They are meant to provide the same flexibility
| for the model as absolute positional encoding that are
| learned (so, position=2 always gets the same encoding) -
| which is what they achieve. The matter of learning about
| distances between positions is left up to the later stages of
| the model.
|
| This differs from the much more modern approach of relative
| positional embeddings, for instance Alibi or Rotary
| Embeddings. These I think fit your intuition much better, as
| they seek to encode relative distances between tokens
| correctly.
| whoateallthepy wrote:
| This is a great set of comments/questions! To try and answer
| this a bit briefly:
|
| The input string is _tokenized_ into a sequence of _token
| indices_ (integers) as the first step of processing the input.
| For example, "Hello World" is tokenized to:
| [15496, 2159]
|
| The first step in a transformer network is to _embed_ the
| tokens. Each token index is mapped to a (learned or fixed)
| embedding (a vector of floats) via the embeddings table. The
| _Embeddings_ module from PyTorch is commonly used. After
| mapping, the matrix of embeddings will look something like:
| [[-0.147, 2.861, ..., -0.447], [-0.517, -0.698, ...,
| -0.558]]
|
| where the number of columns is the _model dimension_.
|
| A single _transformer block_ takes a matrix of embeddings and
| transforms them to a matrix of identical dimensions. An
| important property of the block is that if you reorder the rows
| of the matrix (which can be done by reordering the input
| tokens), the output will be reordered but otherwise identical
| too. (The formal name for this is _permutation equivariance_ ).
|
| In problems related to language it seems inappropriate to have
| the order of tokens not matter, so to solve for this we need to
| adjust the embeddings of the tokens initially based on their
| position.
|
| There are a few common ways you might see this done, but they
| broadly work by assigning fixed or learned embeddings to each
| position in the input token sequence. These embeddings can be
| added to our matrix above so that the first row gets the
| embedding for the first position added to it, the second row
| gets the embedding for the second position, and so on. Now if
| the tokens are reordered, the combined embedding matrix will
| _not_ be the same. Alternatively, these embeddings can be
| concatenated horizontally to our matrix: this guarantees the
| positional information is kept entirely separate from the
| linguistic (at the cost of having a larger model dimension).
|
| I put together this repository at the end of last year to
| better help visualize the internals of a transformer block when
| applied to a toy problem: https://github.com/rstebbing/workshop
| /tree/main/experiments/.... It is not super long, and the point
| is to try and better distinguish between the quantities you
| referred to by seeing them (which is possible when embeddings
| are in a low dimension).
|
| I hope this helps!
| Buttons840 wrote:
| > Alternatively, these embeddings can be concatenated
| horizontally to our matrix: this guarantees the positional
| information is kept entirely separate from the linguistic (at
| the cost of having a larger model dimension).
|
| Yes, the entire description is helpful, but I especially
| appreciate this validation that concatenating the position
| encoding is a valid option.
|
| I've been thinking a lot about aggregation functions, usually
| summation since it's the most basic aggregation function.
| After adding the token embedding and the positional encoding
| together, it seems information has been lost, because the
| resulting sum cannot be separated back into the original
| values. And yet, that seems to be what they do in most
| transformers, so it must be worth the trade-off.
|
| It reminds me of being a kid, when you first realize that
| zipping a file produces a smaller file and you think "well,
| what if I zip the zip file?" At first you wonder if you can
| eventually compress everything down to a single byte. I
| wonder the same with aggregation / summation, "if I can add
| the position to the embedding, and things still work, can I
| just keep adding things together until I have a single
| number?" Obviously there are some limits, but I'm not sure
| where those are. Maybe nobody knows? I'm hoping to study
| linear algebra more and perhaps I will find some answers
| there?
| zwaps wrote:
| The information might be formally lost for the given token,
| but remember that transformers train on huge amounts of
| data.
|
| The (absolute) positional encoding is an arbitrary but
| fixed bias (push into some direction). The word "cat" at
| position 2 is pushed into the 2-direction. This "cat" might
| be different from a "cat at position 3, such that the model
| can learn about this distinction.
|
| Nevertheless, the model could also still learn to keep
| "cats" at all positions together, for instance such "cats"
| are more similar to "cats" than to "dogs" at any position.
| More importantly, for some words, the model might learn
| that a word at the beginning of the sequence should have an
| entirely different meaning than the same word at the end of
| the sequence.
|
| In other words, since the embeddings are a free parameter
| to be learned (usually both as embeddings, and weight-tied
| in the head), there isn't any loss in flexbility. Rather,
| the model can learn how much mixing is required or whether
| the information added by the positional embedding should be
| seperable (for instance by making embeddings linearly
| independent otherwise)
|
| If you concat, you carry along an otherwise useless and
| static dimension, and mixing it into the embeddings would
| be the very first thing the model learns in layer 1.
| whoateallthepy wrote:
| One thing to bear in mind is that these embedding vectors
| are high dimensional, so that it is entirely possible that
| the token embedding and position embedding are near-
| orthogonal to one another. As a result, information isn't
| necessarily lost.
| dist-epoch wrote:
| > The input string is tokenized into a sequence of token
| indices (integers)
|
| How is this tokenization done? Sometimes a single word can be
| two tokens. My understanding is that the token indices are
| also learned, but by whom? The same transformer? Another
| neural network?
| whoateallthepy wrote:
| The tokenization is done by the tokenizer which can be
| thought of as just a function that maps strings to integers
| _before_ the neural network. Tokenizers can be hand-
| specified or learned, but in either case this is typically
| done separately from training the model. It is also less
| frequently necessary unless you are dealing with an
| entirely new input type /language.
|
| Tokenizers can be quite gnarly internally.
| https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt is
| a good resource on BPE tokenization.
| montebicyclelo wrote:
| Huggingface have good guides on tokenization, and tokenizer
| training. BPE (e.g. used by gpt) and wordpiece (e.g. used
| by bert) are two commonly used methods
| https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt
| adriantam wrote:
| If you want a TensorFlow implementation, here it is:
| https://machinelearningmastery.com/building-transformer-mode...
| metalloid wrote:
| The author of the article should had provided an implementation
| of the transformer using only numpy or pure C++.
| jaymody wrote:
| I wrote a minimal implementation in NumPy here (the forward
| pass code is only 40 lines): https://github.com/jaymody/picoGPT
|
| And also a related blog post:
| https://news.ycombinator.com/item?id=34726115
|
| Although this is for a decoder-only transformer (aka GPT) and
| doesnt include the encoder part.
| newhouseb wrote:
| And I adapted Jay's work to Typescript (without the numpy
| obviously, just raw typescript/javascript):
| https://github.com/newhouseb/potatogpt
| dang wrote:
| Related:
|
| _Transformers from Scratch_ -
| https://news.ycombinator.com/item?id=29315107 - Nov 2021 (17
| comments)
|
| also these, but it was a different article:
|
| _Transformers from Scratch (2019)_ -
| https://news.ycombinator.com/item?id=29280909 - Nov 2021 (9
| comments)
|
| _Transformers from Scratch_ -
| https://news.ycombinator.com/item?id=20773992 - Aug 2019 (28
| comments)
| quickthrower2 wrote:
| so I'm on the same journey of trying to teach myself ML and I do
| find most of the resources go over things very quickly and leave
| a lot you to figure out yourself.
|
| Having had a quick look at this one, it looks very beginner,
| friendly, and also very careful to explain things slowly, so I
| will definitely added to my reading list.
|
| Thanks to the author for this!
| KyeRussell wrote:
| Ultimately because it's such a hot topic the "market" is
| flooded with people that want to crank out content without
| understanding what they're talking about.
| dingosity wrote:
| Did anyone make the obvious "Robots in Smalltalk" joke yet?
|
| Okay... here goes...
|
| When I first read that title I thought the author was talking
| about Robots in Smalltalk.
| lucidrains wrote:
| besides everything that was mentioned here, what made it finally
| click for me early in my journey was running through this
| excellent tutorial by Peter Bloem multiple times
| https://peterbloem.nl/blog/transformers highly recommend
| [deleted]
| leobg wrote:
| Can somebody explain to me the sinus wave positional encoding
| thing? The naive approach would be to just add number indices to
| the tokens, wouldn't it?
| tipsytoad wrote:
| I'm no expert, but I think it's so that the model can learn the
| relative position wrt other tokens.
|
| They use indices for models like vision transformers with a
| fixed number of patches but for variable length context I think
| it's more beneficial to use encodings that can also capture the
| relative distance.
| ftxbro wrote:
| According to
| https://kazemnejad.com/blog/transformer_architecture_positio...
| they explain it as a clever way to satisfy the following
| criteria: - It should output a unique
| encoding for each time-step (word's position in a sentence)
| - Distance between any two time-steps should be consistent
| across sentences with different lengths. - Our model
| should generalize to longer sentences without any efforts. Its
| values should be bounded. - It must be deterministic.
|
| Your example contradicts the 'values should be bounded'
| criterion as it generalizes to longer sentences.
___________________________________________________________________
(page generated 2023-04-26 23:02 UTC)