https://ai.v-gar.de/ml/transformer/timeline/ AI / ML / LLM / Transformer Models Timeline and List Viktor Garske @vemgar, Last update: Sun Apr 30 22:49:10 2023 This is a collection of important papers in the area of Large Language Models and Transformer Models. It focuses on recent development, especially from mid-2022 onwards, and in no way claims to be exhaustive. It is actively updated. See also blog post and paper of Xavier Amatriain which provides an excellent overview over the variety of Transformer models and also arranges the models by their family. Something missing or wrong? Do you want to give feedback? Feel free to email me! Legend: Model Method Dataset Application / Analysis -: Origin (architecture, idea or model), - (dashed): (weaker) origin (e.g. code), - (dotted): Related work Download PDF Download PNG This graph is clickable! (c) 2023 Viktor Garske , CC-BY-SA 4.0, https:// ai.v-gar.de/, Last update: Sun Apr 30 22:48:38 UTC 2023 01/2017 02/ 2017 03/2017 04/2017 05/2017 06/2017 07/2017 08/2017 09/2017 10/2017 11/2017 12/2017 01/2018 02/2018 03/2018 04/2018 05/2018 06/2018 07/ 2018 08/2018 09/2018 10/2018 11/2018 12/2018 01/2019 02/2019 03/2019 04/2019 05/2019 06/2019 07/2019 08/2019 09/2019 10/2019 11/2019 12/ 2019 01/2020 02/2020 03/2020 04/2020 05/2020 06/2020 07/2020 08/2020 09/2020 10/2020 11/2020 12/2020 01/2021 02/2021 03/2021 04/2021 05/ 2021 06/2021 07/2021 08/2021 09/2021 10/2021 11/2021 12/2021 01/2022 02/2022 03/2022 04/2022 05/2022 06/2022 07/2022 08/2022 09/2022 10/ 2022 11/2022 12/2022 01/2023 02/2023 03/2023 04/2023 Megatron-LM GPT-NeoX GPT GPT-2 GPT-3 OpenAI Codex GPT-4 InstructGPT LLaMA DALL-E GPT-3.5 MTF AmbiEnt ChatGPT Sparks of AGI Self-Instruct databricks-dolly-15k LLaMA-Adapter GPT4All Koala Baize OpenAssistant Alpaca RedPajama-Data-1T GPT4All-J Pythia Dolly 2.0 StableLM T5 Flan-T5 TANGO UL2 Flan-UL2 LaMDA Bard Dolly DALL-E 2 Stable Diffusion VideoLDM Whisper BLOOM BLOOMZ BloombergGPT PaLM Chinchilla Vicuna StableVicuna MiniGPT-4 DeepFloyd IF BLIP-2 Flamingo NaturalSpeech 2 Generative Agents GPT-J BERT Survey on ChatGPT and Beyond AudioLDM Attention / Transformers Denoising Diffusion Latent Diffusion Models ControlNet RHLF CLIP LAION-5B LoRA Imagen ROOTS The Pile ScienceQA OASST1 Latest entries Published Name 2023-04-28 DeepFloyd IF 2023-04-28 StableVicuna 2023-04-27 AmbiEnt 2023-04-26 Survey on ChatGPT and Beyond 2023-04-24 TANGO 2023-04-19 StableLM 2023-04-18 NaturalSpeech 2 2023-04-18 VideoLDM 2023-04-17 MiniGPT-4 2023-04-17 RedPajama-Data-1T Curated List of Large Language Models and Models based on Transformers (Index) This list contains models, methods and analyses of Large Language Models (LLM) or other (Seq2Seq) models that use Transformers. The list is mostly limited to causal models. Anything missing? Feel free to email me! I am more than happy to update the list! Name Published Paper Name / Blog Post Name Alpaca 2023-03-13 Alpaca: A Strong, Replicable Instruction-Following Model AmbiEnt 2023-04-27 We're Afraid Language Models Aren't Modeling Ambiguity Attention / 2017-06-12 Attention Is All You Need Transformers AudioLDM 2023-01-29 AudioLDM: Text-to-Audio Generation with Latent Diffusion Models Baize: An Open-Source Chat Model with Baize 2023-04-03 Parameter-Efficient Tuning on Self-Chat Data Bard 2023-03-21 BERT 2018-10 BLIP-2: Bootstrapping Language-Image BLIP-2 2023-01-30 Pre-training with Frozen Image Encoders and Large Language Models BLOOM 2022-11-09 BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BloombergGPT 2023-03-30 BloombergGPT: A Large Language Model for Finance BLOOMZ 2022-11-03 Crosslingual Generalization through Multitask Finetuning ChatGPT 2022-11-30 Introducing ChatGPT Chinchilla 2022-03-29 Training Compute-Optimal Large Language Models CLIP 2021-02-26 Learning Transferable Visual Models From Natural Language Supervision OpenAI Codex 2021-07-07 Evaluating Large Language Models Trained on Code ControlNet 2023-02-10 Adding Conditional Control to Text-to-Image Diffusion Models DALL-E 2021-01 DALL-E 2 2022-04-13 Hierarchical Text-Conditional Image Generation with CLIP Latents Stability AI releases DeepFloyd IF, a DeepFloyd IF 2023-04-28 powerful text-to-image model that can smartly integrate text into images Denoising Diffusion 2020-06-19 Denoising Diffusion Probabilistic Models Dolly 2023-03-24 Hello Dolly: Democratizing the magic of ChatGPT with open models Free Dolly: Introducing the World's databricks-dolly-15k 2023-04-12 First Truly Open Instruction-Tuned LLM Free Dolly: Introducing the World's Dolly 2.0 2023-04-12 First Truly Open Instruction-Tuned LLM Flamingo 2022-04-29 Flamingo: a Visual Language Model for Few-Shot Learning Flan-T5 2022-10-20 Scaling Instruction-Finetuned Language Models Flan-UL2 2023-03-03 A New Open Source Flan 20B with UL2 Generative Agents 2023-04-07 Generative Agents: Interactive Simulacra of Human Behavior GPT 2018-06-11 Improving Language Understanding by Generative Pre-Training GPT-J 2021-06 GPT-NeoX 2021-08 GPT-2 2019-02-14 Language Models are Unsupervised Multitask Learners GPT-3 2020-05-28 Language Models are Few-Shot Learners GPT-3.5 2022-03 GPT-4 2023-03-15 GPT-4 Technical Report GPT4All: Training an Assistant-style GPT4All 2023-03 Chatbot with Large Scale Data Distillation from GPT-3.5-Turbo GPT4All-J 2023-04 GPT4All-J: An Apache-2 Licensed Assistant-Style Chatbot Survey on ChatGPT Harnessing the Power of LLMs in and Beyond 2023-04-26 Practice: A Survey on ChatGPT and Beyond Photorealistic Text-to-Image Imagen 2022-05-22 Diffusion Models with Deep Language Understanding InstructGPT 2022-01-27 Training language models to follow instructions with human feedback Koala 2023-04-03 Koala: A Dialogue Model for Academic Research LAION-5B: An open large-scale dataset LAION-5B 2022-06 for training next generation image-text models LaMDA 2021-05-18 LaMDA: our breakthrough conversation technology Latent Diffusion 2021-12-20 High-Resolution Image Synthesis with Models Latent Diffusion Models LLaMA 2023-02-27 LLaMA: Open and Efficient Foundation Language Models LLaMA-Adapter: Efficient Fine-tuning LLaMA-Adapter 2023-03-28 of Language Models with Zero-init Attention LoRA 2021-06-17 LoRA: Low-Rank Adaptation of Large Language Models Megatron-LM: Training Multi-Billion Megatron-LM 2019-09-17 Parameter Language Models Using Model Parallelism MiniGPT-4: Enhancing Vision-Language MiniGPT-4 2023-04-17 Understanding with Advanced Large Language Models MTF 2021-10-15 Multitask Prompted Training Enables Zero-Shot Task Generalization Latent Diffusion Models are Natural NaturalSpeech 2 2023-04-18 and Zero-Shot Speech and Singing Synthesizers OpenAssistant Conversations - OASST1 2023-04-15 Democratizing Large Language Model Alignment OpenAssistant Conversations - OpenAssistant 2023-04-15 Democratizing Large Language Model Alignment Pathways Language Model (PaLM): PaLM 2022-04-04 Scaling to 540 Billion Parameters for Breakthrough Performance The Pile 2020-12 The Pile: An 800GB Dataset of Diverse Text for Language Modeling Pythia: A Suite for Analyzing Large Pythia 2023-04-03 Language Models Across Training and Scaling RedPajama, a project to create RedPajama-Data-1T 2023-04-17 leading open-source models, starts by reproducing LLaMA training dataset of over 1.2 trillion tokens RHLF 2022-04-04 Training language models to follow instructions with human feedback ROOTS 2023-03-07 The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset Learn to Explain: Multimodal ScienceQA 2022-09-20 Reasoning via Thought Chains for Science Question Answering Self-Instruct: Aligning Language Self-Instruct 2022-12-20 Model with Self Generated Instructions Sparks of Artificial General Sparks of AGI 2023-03-22 Intelligence: Early experiments with GPT-4 Stable Diffusion 2022-08-22 StableLM 2023-04-19 Stability AI Launches the First of its StableLM Suite of Language Models Stability AI releases StableVicuna, StableVicuna 2023-04-28 the AI World's First Open Source RLHF LLM Chatbot Exploring the Limits of Transfer T5 2019-10-23 Learning with a Unified Text-to-Text Transformer Text-to-Audio Generation using TANGO 2023-04-24 Instruction-Tuned LLM and Latent Diffusion Model UL2 2022-05-10 UL2: Unifying Language Learning Paradigms Vicuna: An Open-Source Chatbot Vicuna 2023-03-30 Impressing GPT-4 with 90% ChatGPT Quality Align your Latents: High-Resolution VideoLDM 2023-04-18 Video Synthesis with Latent Diffusion Models Whisper 2022-09-21 Robust Speech Recognition via Large-Scale Weak Supervision Homepage, Imprint / Impressum, Privacy / Datenschutz