https://github.com/oracle-samples/sd4j Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} oracle-samples / sd4j Public generated from oracle/template-repo * Notifications * Fork 4 * Star 87 Stable diffusion pipeline in Java using ONNX Runtime License UPL-1.0 license 87 stars 4 forks Activity Star Notifications * Code * Issues 1 * Pull requests 1 * Actions * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Security * Insights oracle-samples/sd4j This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 2 branches 1 tag Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/o] Use Git or checkout with SVN using the web URL. [gh repo clone oracle] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @Craigacp Craigacp Fix model type autodetection and provide a mechanism for supplying a ... ... 2932027 Dec 31, 2023 Fix model type autodetection and provide a mechanism for supplying a ... ...default value. (#10) 2932027 Git stats * 10 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time images Initial commit of SD4J. December 12, 2023 09:43 src Fix model type autodetection and provide a mechanism for supplying a ... December 30, 2023 20:33 text_tokenizer Initial commit of SD4J. December 12, 2023 09:43 .gitignore Initial commit of SD4J. December 12, 2023 09:43 CONTRIBUTING.md Initial commit of SD4J. December 12, 2023 09:43 LICENSE.txt Initial commit December 12, 2023 15:43 README.md Fix model type autodetection and provide a mechanism for supplying a ... December 30, 2023 20:33 SECURITY.md Initial commit December 12, 2023 15:43 THIRD_PARTY_LICENSES.txt Initial commit of SD4J. December 12, 2023 09:43 pom.xml Adding execution provider selection and fixing a bug in the save dial... December 14, 2023 22:01 View code [ ] SD4J (Stable Diffusion in Java) Example images Model support Installation Prepare model checkpoint Setup ORT extensions Running the GUI Use in other programs Using a CUDA GPU Implementation details Contributing Security License README.md SD4J (Stable Diffusion in Java) This repo contains an implementation of Stable Diffusion inference running on top of ONNX Runtime, written in Java. It's a modified port of the C# implementation, with a GUI for repeated generations and support for negative text inputs. It is intended to be a demonstration of how to use ONNX Runtime from Java, and best practices for ONNX Runtime to get good performance. We will keep it current with the latest releases of ONNX Runtime, with appropriate updates as new performance related ONNX Runtime features become available through the ONNX Runtime Java API. All the code is subject to change as this is a code sample, any APIs in it should not be considered stable. This repo targets ONNX Runtime 1.14. The version number is in two parts -, and the initial release of sd4j is v1.0-1.14.0. We'll bump the sd4j version number if it gains new features and the ONNX Runtime version number as we depend on newer versions of ONNX Runtime. The project supports txt2img generation, it doesn't currently implement img2img, upscaling or inpainting. By default it uses a fp32 model, and running on a 6 core 2019 16" Intel Macbook Pro each diffusion step takes around 5s. Running on better hardware, or with a CUDA GPU will greatly reduce the time taken to generate an image, as will using an SD-Turbo model. There is experimental support for the CoreML (for macOS) and DirectML (for Windows) backends, but proper utilisation of these may require model changes like quantization which is not yet implemented. Example images These are a few example images generated by this code along with their generation parameters: Generated image from the prompt "Wildlife photograph of an astronaut riding a horse in the desert" Text: "Wildlife photograph of an astronaut riding a horse in the desert", Negative Text: "", Seed: 42, Guidance Scale: 10, Inference Steps: 40, Scheduler: Euler Ancestral, Image Size: 512x512. Generated image from the prompt "Press photo of an America's Cup catamaran sailing through the sands of Mars, high resolution, high quality" Text: "Press photo of an America's Cup catamaran sailing through the sands of Mars, high resolution, high quality", Negative Text: "water, sea, ocean, lake", Seed: 42, Guidance Scale: 10, Inference Steps: 40, Scheduler: Euler Ancestral, Image Size: 512x512. Generated image from the prompt "Professional photograph of the Apollo 11 lunar lander in a field, high quality, 4k" Text: "Professional photograph of the Apollo 11 lunar lander in a field, high quality, 4k", Negative Text: "", Seed: 42, Guidance Scale: 10, Inference Steps: 50, Scheduler: Euler Ancestral, Image Size: 512x512. Generated image from the prompt "Professional photograph of George Washington in his garden grilling steaks, detailed face, high quality, 4k" Text: "Professional photograph of George Washington in his garden grilling steaks, detailed face, high quality, 4k", Negative Text: "painting, drawing, art", Seed: 42, Guidance Scale: 10, Inference Steps: 60, Scheduler: Euler Ancestral, Image Size: 512x512. Model support The SD4J project supports both SD v1.5 and SD v2 style models, the difference is detected automatically on loading. For models which do not support classifier-free guidance or negative prompts, such as SD-Turbo, the guidance scale should be set to a value less than 1.0 which disables that guidance. Models like SD-Turbo can generate acceptable images in as few as two diffusion steps. Usually the model type is autodetected, but depending on how the model is exported to ONNX format this detection can fail causing the image generation process to crash. In such cases supplying the --model-type {SD1.5, SD2} argument with the appropriate parameter will fix the model type. Installation This project requires Apache Maven, Java 17 or newer, a compiled ONNX Runtime extensions binary, and a Stable Diffusion model checkpoint. The other dependencies (ONNX Runtime and Apache Commons Math) are downloaded by Maven automatically. Prepare model checkpoint There are many compatible models on Hugging Face's website. We have tested the Stable Diffusion v1.5 checkpoint, which has pre-built ONNX models. This can be downloaded via the following git commands (skip the first one if you have already configured git-lfs): git lfs install git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 -b onnx The Stable Diffusion v1.5 checkpoint is available under the OpenRAIL-M license. For other models there is a one or two stage process to generate the ONNX format models. If the model is already in Hugging Face Diffusers format then you can run the convert_stable_diffusion_checkpoint_to_onnx.py file from the diffusers project as follows: python scripts/convert_stable_diffusion_checkpoint_to_onnx.py --model_path --output_path If the model is an original stable diffusion checkpoint then you first need to run: python scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path --scheduler_type lms --dump_path Both scripts require a suitable Python 3 virtual environment with diffusers and onnx installed. Setup ORT extensions You will also need to check out and compile onnxruntime-extensions for your platform. The repo is https://github.com/microsoft/ onnxruntime-extensions, and it can be compiled with ./build_lib.sh --config Release --update --build --parallel which generates the required library (libortextensions.[dylib,so] or ortextensions.dll) in the build//Release/lib/ folder. That library should be copied into the root of this directory. Running the GUI The GUI can be executed with mvn package exec:exec -DmodelPath= . It constructs a window where you can specify the parameters of the image you'd like to generate, and each image creates its own window where it can save the image as a png file. Use in other programs The com.oracle.labs.mlrg.sd4j.SD4J class provides a full image generation pipeline which can be used without the GUI directly from other code. Using a CUDA GPU To use the GPU you need to modify the pom file to depend on onnxruntime_gpu and swap CPU for CUDA in the exec-maven-plugin block. Implementation details This code provides a thin Tensor wrapper object which is a tuple of a direct ByteBuffer instance and a long shape array, which is used to provide easy access in and out of ORT's OnnxTensor objects. There's a Scheduler interface which the two available schedulers (LMS and Euler Ancestral) implement. The SD4J pipeline object is a suitable entry point for use without the GUI, and there is an example of such usage in the CLIApp class. Contributing This project welcomes contributions from the community. Before submitting a pull request, please review our contribution guide. Security Please consult the security guide for our responsible security vulnerability disclosure process License The code is available under the Universal Permissive License (UPL). It requires a Stable Diffusion model architecture checkpoint to work, and any Stable Diffusion models should be used under their licenses. There are 1000+ compatible models available on Hugging Face each of which are licensed separately, though many use a variant of the OpenRAIL-M license. The tokenizer onnx model is taken from the C# implementation, and is available under the MIT license. More details on the tokenizer are available in its README file. About Stable diffusion pipeline in Java using ONNX Runtime Topics java onnxruntime stable-diffusion Resources Readme License UPL-1.0 license Security policy Security policy Activity Stars 87 stars Watchers 2 watching Forks 4 forks Report repository Releases 1 tags Contributors 2 * @Craigacp Craigacp Adam Pocock * @spavlusieva spavlusieva Oleksandra Pavlusieva Languages * Java 100.0% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.