https://github.com/ErwannMillon/Color-diffusion Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} ErwannMillon / Color-diffusion Public * Notifications * Fork 2 * Star 107 A diffusion model to colorize black and white images 107 stars 2 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights ErwannMillon/Color-diffusion This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 6 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/E] Use Git or checkout with SVN using the web URL. [gh repo clone Erwann] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @ErwannMillon ErwannMillon Update README.md ... 8acee88 Jan 17, 2023 Update README.md 8acee88 Git stats * 20 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time configs/default update config January 16, 2023 00:01 input_images Initial commit January 6, 2023 16:53 visualization update inference gifs January 17, 2023 19:48 .gitignore cleaning and refactoring January 15, 2023 23:40 README.md Update README.md January 17, 2023 14:51 app.py update app.py January 16, 2023 02:17 dataset.py update visualization code January 16, 2023 01:45 denoising.py cleaning and refactoring January 15, 2023 23:40 diffusion.py cleaning and refactoring January 15, 2023 23:40 download_dataset.sh cleaning and refactoring January 15, 2023 23:40 dynamic_threshold.py big clean January 14, 2023 14:26 inference.py cleaning and refactoring January 15, 2023 23:40 model.py update visualization code January 16, 2023 01:45 requirements.txt cleaning and teting January 15, 2023 02:10 train.py cleanup, visualization January 15, 2023 15:20 utils.py update visualization code January 16, 2023 01:45 visualization.py update inference gifs January 17, 2023 19:48 View code Color Diffusion Overview Future Work / Ideas References Usage README.md Color Diffusion Using diffusion models to colorize black and white images. [total_1] [total_2] [total_3] [total_4] [total_8] [total_90] Overview This project is a simple example of how we can use diffusion models to colorize black and white images. This implementation uses the LAB color space, a 3 channel alternative to the RGB color space. The "L" (Lightness) channel in this space is equivalent to a greyscale image: it represents the luminous intensity of each pixel. The two other channels are used to represent the color of each pixel. To train the model, we first load color images and convert them to LAB. Then, we add noise only to the color channels, keeping the L channel constant. The model gets this channel "for free" because it doesn't need to learn how to predict the greyscale image: it is always known at train and test time. [forward_diff] Forward Diffusion Process Note that we actually don't need to go through all of the steps of the diffusion process to get to timestep t. Our forward diffusion process is non-Markovian, but the entire diffusion process is shown for illustrative purposes The model is a UNet that takes a 3 channel LAB input (the ground-truth greyscale channel concatenated with noised AB channels) and outputs a 2 channel prediction of the color noise. [total1] [total2] [total3] [total4] Forward diffusion and denoising at train time In addition to receiving the greyscale channel as input, the UNet is also conditioned on features extracted from the greyscale channel. Intermediate feature maps from an encoder (implemented as the first half of a UNet) are concatenated with the features of the main denoising UNet throughout the downsampling stage of the forward pass. Future Work / Ideas This was just a quick proof of concept to satisfy my curiosity and get a feel for training diffusion models from scratch, so the results are very basic. There are many ways this project could be improved, such as: * Using pretrained face recognition networks like ArcFace or FaceNet as feature extractors to get the conditioning features * Implementing cross attention on the embeddings * Pretraining the greyscale feature extractor as the encoder stage of a greyscale autoencoder References A lot of code for the dataset and LAB color operations was adapted from moein-sharitania's colorization project, which used Conditional GANs https://github.com/moein-shariatnia/Deep-Learning I implemented optional dynamic thresholding as in Assembly AI's Minimagen project (the Assembly AI blog posts are excellent for getting a deep understanding of the maths and concepts behind diffusion models) https://www.assemblyai.com/blog/ minimagen-build-your-own-imagen-text-to-image-model/ The UNet architecture was adapted from denoising-diffusion-pytorch https://github.com/lucidrains/denoising-diffusion-pytorch Usage Run bash download_dataset.sh to download the CelebA dataset and extract it Use inference.py for command line colorization. python inference.py --image-path --checkpoint --output Or run python app.py for a simple gradio web UI About A diffusion model to colorize black and white images Resources Readme Activity Stars 107 stars Watchers 3 watching Forks 2 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 99.5% * Shell 0.5% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.