https://gist.github.com/veekaybee/be375ab33085102f9027853128dc5f0e

Skip to content
 
[                    ]
All gists Back to GitHub Sign in Sign up
 
Sign in Sign up
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session. You switched accounts on another tab or window. Reload
to refresh your session.
{{ message }}

Instantly share code, notes, and snippets.

@veekaybee

veekaybee/normcore-llm.md

Last active August 27, 2023 22:59

  * Star 725 You must be signed in to star a gist
  * Fork 47 You must be signed in to fork a gist

Star You must be signed in to star a gist
Code Revisions 23 Stars 725 Forks 47
Embed
What would you like to do?
">Embed Embed this gist in your website.
Share Copy sharable link for this gist.
Clone via HTTPS Clone with Git or checkout with SVN using the
repository's web address.
 
Learn more about clone URLs
[<script src="https:/]
 
Download ZIP
Normcore LLM Reads
Raw
normcore-llm.md

 Anti-hype LLM reading list

Goals: Add links that are reasonable and good explanations of how
stuff works. No hype and no vendor content if possible. Practical
first-hand accounts and experience preferred (super rare at this
point).

My own notes from a few months back.

 Background

  * Survey of LLMS
  * Self-attention and transformer networks
  * What are embeddings
  * The Illustrated Word2vec - A Gentle Intro to Word Embeddings in
    Machine Learning (YouTube)
  * Catching up on the weird world of LLMS

 Foundational Papers

  * Attention is all you Need
  * Scaling Laws for Neural Language Models
  * BERT
  * Language Models are Unsupervised Multi-Task Learners
  * Training Language Models to Follow Instructions
  * Language Models are Few-Shot Learners

 Training Your Own

  * Why host your own LLM?
  * How to train your own LLMs
  * Training Compute-Optimal Large Language Models
  * Opt-175B Logbook

 Algos

  * The case for GZIP Classifiers and more on nearest neighbors algos
  * Meta Recsys Using and extending Word2Vec
  * The State of GPT (YouTube)
  * What is ChatGPT doing and why does it work
  * How is LlamaCPP Possible?
  * On Prompt Engineering
  * Transformers from Scratch

 Deployment

  * Building LLM Applications for Production
  * Challenges and Applications of Large Language Models
  * All the Hard Stuff Nobody talks about when building products with
    LLMs
  * Scaling Kubernetes to run ChatGPT
  * Numbers every LLM Developer should know

 Evaluation

  * Interpretable Machine Learning
  * Evaluating ChatGPT
  * ChatGPT: Jack of All Trades, Master of None

 UX

  * Generative Interfaces Beyond Chat (YouTube)
  * Why Chatbots are not the Future

Thanks to everyone who added suggestions on Twitter, Mastodon, and
Bluesky.

@untitled01ipynb
Copy link

untitled01ipynb commented Aug 15, 2023

Needs memes

Sorry, something went wrong.

@san7988
Copy link

san7988 commented Aug 16, 2023

Sorry but what is normcore llm?

Sorry, something went wrong.

@NirantK
Copy link

NirantK commented Aug 16, 2023 *
edited

How is Llama.cpp possible perhaps?

Sorry, something went wrong.

@veekaybee
Copy link
Author

veekaybee commented Aug 17, 2023 *
edited

    How is Llama.cpp possible perhaps?

good one!

Sorry, something went wrong.

@veekaybee
Copy link
Author

veekaybee commented Aug 17, 2023

    Needs memes

You up for the challenge?

Sorry, something went wrong.

@veekaybee
Copy link
Author

veekaybee commented Aug 17, 2023

    Sorry but what is normcore llm?

Content that is not hype or vendor knowledge but concrete papers and
blog posts about people's experiences with them.

Sorry, something went wrong.

@paws07
Copy link

paws07 commented Aug 17, 2023

The gist name made me miss normconf

Sorry, something went wrong.

@ktrivedisf
Copy link

ktrivedisf commented Aug 19, 2023

It would be really nice if community adds some resources on
following.

 1. How to debug LLM (or not ?) ?
 2. How to monitor LLM in production ? can you apply traditional
    system KPI like tput and latency to begin with.
    2a. Can we observe effect of prompts on performance of the model
    (again performance of GPU cycle and memory consumption and then
    performance of correctness).
    2b. Are there anything else which would be helpful to make sure
    that performance is steady (both of resources and of actual
    prompt)
 3. How batching works and is there a good way to select right batch
    size ?

Sorry, something went wrong.

@tekumara
Copy link

tekumara commented Aug 20, 2023

Transformer Math 101

    We present basic math related to computation and memory usage for
    transformers

Sorry, something went wrong.

@tekumara
Copy link

tekumara commented Aug 20, 2023

Why Chatbots Are Not the Future

Sorry, something went wrong.

@san7988
Copy link

san7988 commented Aug 20, 2023

Posts from eugene yan are also pretty good read.

Sorry, something went wrong.

@butsugiri
Copy link

butsugiri commented Aug 22, 2023

Hi, thank you for your great work!

    Training Your Own

I wonder if we can add huggingface/llm_training_handbook: An open
collection of methodologies to help with successful training of large
language models. to this section?

Sorry, something went wrong.

@lcrmorin
Copy link

lcrmorin commented Aug 22, 2023

It seems that the gzip approach, altough really cool, was
'optimistic' and thus overhyped, see: https://kenschutte.com/
gzip-knn-paper/ (basiccaly they confused k in k-nn and top-k
accuracy, reporting top-2 accuracy). More recent studies found that
it is, as expected, on 'bag of words' performance level Gzip versus
bag-of-words for text classification.

I don't know if you intend to (or are even interested) but I am on
the look out for "usecases for normies".

Sorry, something went wrong.

@wrhall
Copy link

wrhall commented Aug 22, 2023

Do you think it's worth annotating with dates of the articles /
papers / videos?

Sorry, something went wrong.

@veekaybee
Copy link
Author

veekaybee commented Aug 23, 2023

    Why Chatbots Are Not the Future

Great one!

Sorry, something went wrong.

@veekaybee
Copy link
Author

veekaybee commented Aug 23, 2023

    It seems that the gzip approach, altough really cool, was
    'optimistic' and thus overhyped, see: https://kenschutte.com/
    gzip-knn-paper/ (basiccaly they confused k in k-nn and top-k
    accuracy, reporting top-2 accuracy). More recent studies found
    that it is, as expected, on 'bag of words' performance level Gzip
    versus bag-of-words for text classification.

    I don't know if you intend to (or are even interested) but I am
    on the look out for "usecases for normies".

Yeah that was my read on it as well but I'm also very interested in
it as a general theoretical approach and baseline, even if this
particular implementation doesn't work.

Sorry, something went wrong.

@veekaybee
Copy link
Author

veekaybee commented Aug 23, 2023

    Do you think it's worth annotating with dates of the articles /
    papers / videos?

Maybe would be helpful but I explicitly picked stuff that I thought
wouldn't age and/or where the recency didn't matter because the
fundamentals are timeless.

Sorry, something went wrong.

@janhesse53
Copy link

janhesse53 commented Aug 23, 2023

Patterns for Building LLM-based Systems & Products

In my opinion, this is a super in depth article that covers many of
the categories and deserves a place in the reading list.

Sorry, something went wrong.

@rmitsch
Copy link

rmitsch commented Aug 25, 2023

Against LLM maximalism (disclaimer: I work at Explosion)

Sorry, something went wrong.

@Lykos2
Copy link

Lykos2 commented Aug 27, 2023

    Patterns for Building LLM-based Systems & Products In my opinion,
    this is a super in depth article that covers many of the
    categories and deserves a place in the reading list.

detailed blog

Sorry, something went wrong.

@spmurrayzzz
Copy link

spmurrayzzz commented Aug 27, 2023

The Illustrated Transformer - maybe redundant from some of the other
transformers content here, but is very well-written and strikes a
good balance between prose and visual aids.

Sorry, something went wrong.

@satisfice
Copy link

satisfice commented Aug 27, 2023

ChatGPT Sucks at Being a Testing Expert
https://www.satisfice.com/download/
appendix-chatgpt-sucks-at-being-a-testing-expert?wpdmdl=487569

This is a careful analysis of an attempt to demonstrate ChatGPT's
usefulness to help testers.

Sorry, something went wrong.

@davidzshi
Copy link

davidzshi commented Aug 27, 2023

I liked this article a lot: https://lilianweng.github.io/posts/
2023-06-23-agent/

Sorry, something went wrong.

Sign up for free to join this conversation on GitHub. Already have an
account? Sign in to comment

Footer

 (c) 2023 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact GitHub
  * Pricing
  * API
  * Training
  * Blog
  * About

You can't perform that action at this time.