https://github.com/huggingface/diffusers Skip to content Sign up * Product + Features + Mobile + Actions + Codespaces + Copilot + Packages + Security + Code review + Issues + Discussions + Integrations + GitHub Sponsors + Customer stories * Team * Enterprise * Explore + Explore GitHub + Learn and contribute + Topics + Collections + Trending + Skills + GitHub Sponsors + Open source guides + Connect with others + The ReadME Project + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing + Plans + Compare plans + Contact Sales + Education [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} huggingface / diffusers Public * Notifications * Fork 38 * Star 760 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch License Apache-2.0 license 760 stars 38 forks Star Notifications * Code * Issues 11 * Pull requests 4 * Actions * Projects 0 * Wiki * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights huggingface/diffusers This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 26 branches 6 tags Code Latest commit @StoneCypher StoneCypher Documentation cross-reference (#127) ... 85244d4 Jul 21, 2022 Documentation cross-reference (#127) In #124 I incorrectly suggested that the image set creation process was undocumented. In reality, I just hadn't located it. @patrickvonplaten did so for me. This PR places a hotlink so that people like me can be shoehorned over where they needed to be. 85244d4 Git stats * 664 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows Docs (#45) Jul 13, 2022 docs/source README improvements: credits and roadmap (#116) Jul 21, 2022 examples Documentation cross-reference (#127) Jul 21, 2022 scripts some more cleaning Jul 21, 2022 src/diffusers fixed URLs broken by bdecc3 folder move (#77) Jul 21, 2022 tests Final fixes (#118) Jul 21, 2022 utils Make style Jun 27, 2022 .gitignore upload some cleaning tools May 31, 2022 LICENSE init upload May 30, 2022 MANIFEST.in Include MANIFEST.in to package the modelcard template Jul 19, 2022 Makefile Make style Jun 27, 2022 README.md Update README.md Jul 21, 2022 pyproject.toml init upload May 30, 2022 setup.cfg init upload May 30, 2022 setup.py Release: 0.1.2 Jul 21, 2022 View code Quickstart Examples Definitions Philosophy Installation In the works Credits README.md [diffusers_library] GitHub GitHub release Contributor Covenant Diffusers provides pretrained diffusion models across multiple modalities, such as vision and audio, and serves as a modular toolbox for inference and training of diffusion models. More precisely, Diffusers offers: * State-of-the-art diffusion pipelines that can be run in inference with just a couple of lines of code (see src/diffusers/pipelines ). * Various noise schedulers that can be used interchangeably for the prefered speed vs. quality trade-off in inference (see src/ diffusers/schedulers). * Multiple types of models, such as UNet, that can be used as building blocks in an end-to-end diffusion system (see src/ diffusers/models). * Training examples to show how to train the most popular diffusion models (see examples). Quickstart In order to get started, we recommend taking a look at two notebooks: * The Getting started with Diffusers notebook, which showcases an end-to-end example of usage for diffusion models, schedulers and pipelines. Take a look at this notebook to learn how to use the pipeline abstraction, which takes care of everything (model, scheduler, noise handling) for you, but also to get an understanding of each independent building blocks in the library. * The Training a diffusers model notebook, which summarizes diffuser model training methods. This notebook takes a step-by-step approach to training your diffuser model on an image dataset, with explanatory graphics. Examples If you want to run the code yourself , you can try out: * Text-to-Image Latent Diffusion # !pip install diffusers transformers from diffusers import DiffusionPipeline model_id = "CompVis/ldm-text2im-large-256" # load model and scheduler ldm = DiffusionPipeline.from_pretrained(model_id) # run pipeline in inference (sample random noise and denoise) prompt = "A painting of a squirrel eating a burger" images = ldm([prompt], num_inference_steps=50, eta=0.3, guidance_scale=6)["sample"] # save images for idx, image in enumerate(images): image.save(f"squirrel-{idx}.png") * Unconditional Diffusion with discrete scheduler # !pip install diffusers from diffusers import DDPMPipeline, DDIMPipeline, PNDMPipeline model_id = "google/ddpm-celebahq-256" # load model and scheduler ddpm = DDPMPipeline.from_pretrained(model_id) # you can replace DDPMPipeline with DDIMPipeline or PNDMPipeline for faster inference # run pipeline in inference (sample random noise and denoise) image = ddpm()["sample"] # save image image[0].save("ddpm_generated_image.png") * Unconditional Latent Diffusion * Unconditional Diffusion with continous scheduler If you just want to play around with some web demos, you can try out the following Spaces: Model Hugging Face Spaces Text-to-Image Latent Diffusion Hugging Face Spaces Faces generator Hugging Face Spaces DDPM with different schedulers Hugging Face Spaces Definitions Models: Neural network that models $p_\theta(\mathbf{x}_{t-1}|\mathbf {x}_t)$ (see image below) and is trained end-to-end to denoise a noisy input to an image. Examples: UNet, Conditioned UNet, 3D UNet, Transformer UNet [174349667-04e9e485-793b-429a-affe-096e8199ad5b] Figure from DDPM paper (https://arxiv.org/abs/2006.11239). Schedulers: Algorithm class for both inference and training. The class provides functionality to compute previous image according to alpha, beta schedule as well as predict noise for training. Examples: DDPM, DDIM, PNDM, DEIS [174349706-53d58acc-a4d1-4cda-b3e8-432d9dc7ad38] Sampling and training algorithms. Figure from DDPM paper (https:// arxiv.org/abs/2006.11239). Diffusion Pipeline: End-to-end pipeline that includes multiple diffusion models, possible text encoders, ... Examples: Glide, Latent-Diffusion, Imagen, DALL-E 2 [174348898-481bd7c2-5457-4830-89bc-f0907756f64c] Figure from ImageGen (https://imagen.research.google/). Philosophy * Readability and clarity is prefered over highly optimized code. A strong importance is put on providing readable, intuitive and elementary code design. E.g., the provided schedulers are separated from the provided models and provide well-commented code that can be read alongside the original paper. * Diffusers is modality independent and focusses on providing pretrained models and tools to build systems that generate continous outputs, e.g. vision and audio. * Diffusion models and schedulers are provided as consise, elementary building blocks whereas diffusion pipelines are a collection of end-to-end diffusion systems that can be used out-of-the-box, should stay as close as possible to their original implementation and can include components of other library, such as text-encoders. Examples for diffusion pipelines are Glide and Latent Diffusion. Installation pip install diffusers # should install diffusers 0.1.2 In the works For the first release, Diffusers focuses on text-to-image diffusion techniques. However, diffusers can be used for much more than that! Over the upcoming releases, we'll be focusing on: * Diffusers for audio * Diffusers for reinforcement learning (initial work happening in # 105). * Diffusers for video generation * Diffusers for molecule generation (initial work happening in #54) A few pipeline components are already being worked on, namely: * BDDMPipeline for spectrogram-to-sound vocoding * GLIDEPipeline to support OpenAI's GLIDE model * Grad-TTS for text to audio generation / conditional audio generation We want diffusers to be a toolbox useful for diffusers models in general; if you find yourself limited in any way by the current API, or would like to see additional models, schedulers, or techniques, please open a GitHub issue mentioning what you would like to see. Credits This library concretizes previous work by many different authors and would not have been possible without their great research and implementations. We'd like to thank, in particular, the following implementations which have helped us in our development and without which the API could not have been as polished today: * @CompVis' latent diffusion models library, available here * @hojonathanho original DDPM implementation, available here as well as the extremely useful translation into PyTorch by @pesser, available here * @ermongroup's DDIM implementation, available here. * @yang-song's Score-VE and Score-VP implementations, available here We also want to thank @heejkoo for the very helpful overview of papers, code and resources on diffusion models, available here as well as @crowsonkb and @rromb for useful discussions and insights. About Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch Topics deep-learning pytorch image-generation diffusion score-based-generative-modeling Resources Readme License Apache-2.0 license Stars 760 stars Watchers 38 watching Forks 38 forks Releases 1 Initial release of Diffusers Latest Jul 21, 2022 Packages 0 No packages published Used by 1 * @creatorrr @creatorrr / golld Contributors 19 * * * * * * * * * * * + 8 contributors Languages * Python 99.5% * Makefile 0.5% Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.