https://brandonhan.uk/HeadSculpt/ HeadSculpt: Crafting 3D Head Avatars with Text Xiao Han^1,4*, Yukang Cao^2*, Kai Han^2, Xiatian Zhu^1,5, Jiankang Deng^3, Yi-Zhe Song^1,4, Tao Xiang^1,4+, Kwan-Yee K. Wong^2+ ^*Equal contributions ^+Corresponding authors ^1University of Surrey ^2The University of Hong Kong ^3 Imperial College London ^4iFlyTek-Surrey Joint Research Centre on AI ^5 Surrey Institute for People-Centred AI Paper ArXiv Supplementary Material Code (coming soon) --------------------------------------------------------------------- Abstract Recently, text-guided 3D generative methods have made remarkable advancements in producing high-quality textures and geometry, capitalizing on the proliferation of large vision-language and image diffusion models. However, existing methods still struggle to create high-fidelity 3D head avatars in two aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model whilst missing the necessary 3D awareness and head priors. This makes them prone to inconsistency and geometric distortions in the generated avatars. (2) They fall short in fine-grained editing. This is primarily due to the inherited limitations from the pre-trained 2D image diffusion models, which become more pronounced when it comes to 3D head avatars. In this work, we address these challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt for crafting (i.e., generating and editing) 3D head avatars from textual prompts. Specifically, we first equip the diffusion model with 3D awareness by leveraging landmark-based control and a learned textual embedding representing the back view appearance of heads, enabling 3D-consistent head avatar generations. We further propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with a high-resolution differentiable rendering technique. This enables identity preservation while following the editing instruction. We showcase HeadSculpt's superior fidelity and editing capabilities through comprehensive experiments and comparisons with existing methods. --------------------------------------------------------------------- 3D Head Avatar Generation HeadSculpt can create an assortment of head avatars, including humans (both celebrities and ordinary individuals) as well as non-human characters like superheroes, comic/game characters, paintings, and more. [panther] [panther_no] [hulk] [hulk_norma] [dali] [dali_norma] a DSLR portrait of a DSLR portrait of a DSLR portrait of Black Panther in Hulk Salvador Dali Marvel [naruto] [naruto_nor] [vincent] [vincent_no] [messi] [messi_norm] a head of Naruto a portrait of Vincent a DSLR portrait of Uzumaki van Gogh Lionel Messi [tolstoy] [tolstoy_no] [kratos] [kratos_nor] [soldier] [soldier_no] a DSLR portrait of Leo a DSLR portrait of a DSLR portrait of a Tolstoy Kratos in God of War female soldier, wearing a helmet [simpson] [simpson_no] [army] [army_norma] [boy] [boy_normal] a head of Simpson in a head of Terracotta a DSLR portrait of a the Simpsons Army boy with facial painting --------------------------------------------------------------------- 3D Head Avatar Editing In addition to head avatar creations, our method enables fine-grained editing, including local changes, shape/texture modifications, and style transfers. [saul] [saul_norma] [saul_clown] [saul_clown] [saul_old] [saul_old_n] a DSLR portrait of turn him into a clown make him older Saul Goodman [obama] [obama_norm] [obama_pixa] [obama_pixa] [obama_skul] [obama_skul] a DSLR portrait of turn his face into a Obama wearing a turn him into Pixar style skull baseball cap [geralt] [geralt_nor] [geralt_lau] [geralt_lau] [geralt_min] [geralt_min] a DSLR portrait of Geralt in The Witcher make him simling turn him into Minecraft 3 [hepburn] [hepburn_no] [hepburn_cl] [hepburn_cl] [hepburn_co] [hepburn_co] a DSLR portrait of make her a claymation make her color-restored Audrey Hepburn [caesar] [caesar_nor] [caesar_woo] [caesar_woo] [caesar_gog] [caesar_gog] a head of Caesar in make it carved out of as a swimmer with a Rise of the Planet of wood goggle the Apes --------------------------------------------------------------------- Comparisons with Others Stable-DreamFusion Latent-NeRF 3DFuse Fantasia3D.unofficial HeadSculpt (Ours) [batman_df] [batman_ln] [batman_3f] [batman_f3] [batman] a DSLR portrait of Batman [strange_df] [strange_ln] [strange_3f] [strange_f3] [strange] a DSLR portrait of Doctor Strange [two-face_df] [two-face_ln] [two-face_3f] [two-face_f3] [two-face] a DSLR portrait of Two-face in DC [napoleon_df] [napoleon_ln] [napoleon_3f] [napoleon_f3] [napoleon] a DSLR portrait of Napoleon Bonaparte --------------------------------------------------------------------- Method We craft high-resolution 3D head avatars in a coarse-to-fine manner. (a) We optimize neural field representations for the coarse model. (b) We refine or edit the model using the extracted 3D mesh and apply identity-aware editing score distillation if editing is the target. (c) The core of our pipeline is the prior-driven score distillation, which incorporates landmark control, enhanced view-dependent prompts, and an InstructPix2Pix branch. [pipeline] --------------------------------------------------------------------- BibTeX @article{han2023headsculpt, title={HeadSculpt: Crafting 3D Head Avatars with Text}, author={Xiao Han and Yukang Cao and Kai Han and Xiatian Zhu and Jiankang Deng and Yi-Zhe Song and Tao Xiang and Kwan-Yee K. Wong}, journal={arXiv preprint arXiv:2306.03038}, year={2023} } This website is constructed using the source code provided by Nerfies , and we are grateful for the template they provided. Allow us to express our appreciation for their contribution.