https://nvlabs.github.io/Sana/

Logo

Sana

Efficient High-Resolution Image Synthesis
with Linear Diffusion Transformer

Exploring the Frontiers of Efficient Generative Foundation Models

Enze Xie^1*, Junsong Chen^1*, Junyu Chen^2,3, Han Cai^1, Haotian Tang
^2,
Yujun Lin^2, Zhekai Zhang^2, Muyang Li^2, Ligeng Zhu^1, Yao Lu^1,
Song Han^1,2

^1NVIDIA, ^2MIT, ^3Tsinghua University
*Project co-lead

NVIDIA Logo MIT Logo THU Logo
Paper Code (Coming soon)
Image 1
Image 2
Image 3
Image 4
Image 5
Image 10
Image 7
Image 8
Image 9
Image 10
Image 11
Image 12
Image 13
Image 14
Image 15
Image 16
Image 17
Image 18
[]

About Sana

We introduce Sana, a text-to-image framework that can efficiently
generate images up to 4096 x 4096 resolution. Sana can synthesize
high-resolution, high-quality images with strong text-image alignment
at a remarkably fast speed, deployable on laptop GPU. Core designs
include: Deep compression autoencoder: unlike traditional AEs, which
compress images only 8x, we trained an AE that can compress images
32x, effectively reducing the number of latent tokens. Linear DiT: we
replace all vanilla attention in DiT with linear attention, which is
more efficient at high resolutions without sacrificing quality.
Decoder-only text encoder: we replaced T5 with modern decoder-only
small LLM as the text encoder and designed complex human instruction
with in-context learning to enhance the image-text alignment.
Efficient training and sampling: we propose Flow-DPM-Solver to reduce
sampling steps, with efficient caption labeling and selection to
accelerate convergence.
As a result, Sana-0.6B is very competitive with modern giant
diffusion model (e.g. Flux-12B), being 20 times smaller and 100+
times faster in measured throughput. Moreover, Sana-0.6B can be
deployed on a 16GB laptop GPU, taking less than 1 second to generate
a 1024 x 1024 resolution image. Sana enables content creation at low
cost.

latency comparison with SOTA methods

Several Core Design Details for Efficiency

    *   Deep Compression Autoencoder: We introduce a new Autoencoder
(AE) that aggressively increases the scaling factor to 32. Compared
with AE-F8, our AE-F32 outputs 16x fewer latent tokens, which is
crucial for efficient training and generating ultra-high-resolution
images, such as 4K resolution.
    *   Efficient Linear DiT: We introduce a new linear DiT,
replacing vanilla quadratic attention and reducing complexity from O
(N^2) to O(N) Mix-FFN, with 3x3 depth-wise convolution in MLP,
enhances the local information of tokens. Linear attention achieves
comparable results to vanilla, improving 4K generation by 1.7x in
latency. Mix-FFN also removes the need for positional encoding (NoPE)
without quality loss, marking the first DiT without positional
embedding.
    *   Decoder-only Small LLM as Text Encoder: We use Gemma, a
decoder-only LLM, as the text encoder to enhance understanding and
reasoning in prompts. Unlike CLIP or T5, Gemma offers superior text
comprehension and instruction-following. We address training
instability and design complex human instructions (CHI) to leverage
Gemma's in-context learning, improving image-text alignment.

pipeline for Sana

    *   Efficient Training and Inference Strategy: We propose
automatic labeling and training strategies to improve text-image
consistency. Multiple VLMs generate diverse re-captions, and a
CLIPScore-based strategy selects high-CLIPScore captions to enhance
convergence and alignment. Additionally, our Flow-DPM-Solver reduces
inference steps from 28-50 to 14-20 compared to the
Flow-Euler-Solver, with better performance.

flow-dpms vs flow-euler

Overall Performance

We compare Sana with the most advanced text-to-image diffusion models
in Table 1. For 512 x 512 resolution, Sana-0.6 demonstrates a
throughput that is 5x faster than PixArt-S, which has a similar model
size, and significantly outperforms it in FID, Clip Score, GenEval,
and DPG-Bench. For 1024 x 1024 resolution, Sana is considerably
stronger than most models with <3B parameters and excels in inference
latency. Our models achieve competitive performance even when
compared to the most advanced large model FLUX-dev. For instance,
while the accuracy on DPG-Bench is equivalent and slightly lower on
GenEval, Sana-0.6B's throughput is 39x faster, and Sana-1.6B is 23x
faster.

Sana performance

Sana-0.6B is Deployable on Laptop GPU

Your browser does not support the video tag.

Our Mission

Our mission is to develop efficient, lightweight, and accelerated AI
technologies that address practical challenges and deliver fast,
open-source solutions.

BibTeX

@misc{xie2024sana,
      title={Sana: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer},
      author={Enze Xie and Junsong Chen and Junyu Chen and Han Cai and Haotian Tang and Yujun Lin and Zhekai Zhang and Muyang Li and Ligeng Zhu and Yao Lu and Song Han},
      year={2024},
      eprint={2410.10629},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2410.10629},
    }

This website is licensed under a Creative Commons
Attribution-ShareAlike 4.0 International License.

Total clicks: