https://token-verse.github.io/

TokenVerse: Versatile Multi-concept Personalization in Token
Modulation Space


arXiv Code (coming soon)
Your browser does not support the <video> tag.

Abstract

We present TokenVerse -- a method for multi-concept personalization,
leveraging a pre-trained text-to-image diffusion model. Our framework
can disentangle complex visual elements and attributes from as little
as a single image, while enabling seamless plug-and-play generation
of combinations of concepts extracted from multiple images. As
opposed to existing works, which are restricted either in the type or
breadth of concepts they can handle, TokenVerse can handle multiple
images with multiple concepts each, and supports a wide-range of
concepts, including objects, accessories, materials, pose, and
lighting. Our work exploits a DiT-based text-to-image model, in which
the input text affects the generation through both attention and
modulation (shift and scale). We observe that the modulation space is
semantic and enables localized control over complex concepts.
Building on this insight, we devise an optimization-based framework
that takes as input an image and a text description, and finds for
each word a distinct direction in modulation space. These directions
can then be used to generate new images that combine the learned
concepts in a desired configuration. We demonstrate the effectiveness
of TokenVerse in challenging personalization settings, and showcase
its advantages over existing methods.

Results

[13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26]
[27] [28] [01] [02] [03] [04] [05] [06] [07] [08] [09] [10] [11] [12]

Try Me

Click on the highlighted words in the prompt below to select a
concept.


Doll

[concept-6]

Shirt

[concept-2]

Glasses

[concept-41]

Necklace

[concept-12]


A doll wearing a shirt, glasses and necklace in the beach.


Method

[method]


Our method leverages the exisitng modulation mechanism in DiT models
to extract concept specific information from user provided images.

(a) A pre-trained text-to-image DiT model processes both image and
text tokens via a series of DiT blocks. Each block consists of
modulation, attention and feed-forward modules. We focus on the
modulation block, in which the tokens are modulated via a vector y,
which is derived from a pooled text embedding.

(b) In our TokenVerse method, given a concept image and its
corresponding caption, we learn a personalized modulation vector
adjustment delta for each text embedding. These adjustments represent
personalized directions in the modulation space and are learned using
a simple reconstruction objective.

(c) At inference, the pre-learned direction vectors are added to the
text embeddings, enabling injection of personalized concepts into the
generated images.

Abstract Concepts

Lighting

[light-1] [light-2] [light-3]



Posing

[pose-1] [pose-2] [pose-3]



Texture

[tex-1] [tex-2] [tex-3]

Combining Abstract and Objects Concepts



[6] [1] [2] [3] [4] [5] [7] [8]

StoryTelling


An immediate application of TokenVerse is storytelling, where one can
generate a narrative consisting of images featuring the same objects
and scenes.
[storytelli]

This website is licensed under a Creative Commons
Attribution-ShareAlike 4.0 International License.

Website source code based on the Nerfies project page. If you want to
reuse their source code, please credit them appropriately.