https://github.com/facebookresearch/audio2photoreal Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} facebookresearch / audio2photoreal Public * Notifications * Fork 7 * Star 220 Code and dataset for photorealistic Codec Avatars driven from audio License View license 220 stars 7 forks Activity Star Notifications * Code * Issues 1 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights facebookresearch/audio2photoreal This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 3 branches 1 tag Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/f] Use Git or checkout with SVN using the web URL. [gh repo clone facebo] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @alexanderrichard alexanderrichard Update README.md ... 9203585 Jan 4, 2024 Update README.md 9203585 Git stats * 2 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time assets Initial commit January 2, 2024 17:47 data_loaders Initial commit January 2, 2024 17:47 dataset Initial commit January 2, 2024 17:47 demo Initial commit January 2, 2024 17:47 diffusion Initial commit January 2, 2024 17:47 model Initial commit January 2, 2024 17:47 sample Initial commit January 2, 2024 17:47 scripts Initial commit January 2, 2024 17:47 train Initial commit January 2, 2024 17:47 utils Initial commit January 2, 2024 17:47 visualize Initial commit January 2, 2024 17:47 CODE_OF_CONDUCT.md Initial commit January 2, 2024 17:47 CONTRIBUTING.md Initial commit January 2, 2024 17:47 LICENSE Initial commit January 2, 2024 17:47 README.md Update README.md January 3, 2024 20:54 View code [ ] From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations Repository Contents Quickstart Installation Download data and models Dataset Visualize ground truth Pretrained models Running the pretrained models Face generation Body generation Visualization Training from scratch 1) Face diffusion model 2) Body diffusion model 3) Body VQ VAE 4) Body guide transformer License README.md From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations This repositiory contains a pytorch implementation of "From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations" audio2photoreal.mp4 This codebase provides: * train code * test code * pretrained motion models * access to dataset If you use the dataset or code, please cite our Paper @article{ng2024audio2photoreal, title={From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations}, author={Ng, Evonne and Romero, Javier and Bagautdinov, Timur and Bai, Shaojie and Darrell, Trevor and Kanazawa, Angjoo and Richard, Alexander}, journal={arXiv preprint arXiv:2401.01885}, year={2024} } Repository Contents * Quickstart: easy gradio demo that lets you record audio and render a video * Installation: environment setup and installation (for more details on the rendering pipeline, please refer to Codec Avatar Body) * Download data and models: download annotations and pre-trained models + Dataset desc.: description of dataset annotations + Visualize Dataset: script for visualizing ground truth annotations + model desc.: description of pretrained models * Running the pretrained models: how to generate results files and visualize the results using the rendering pipeline. + Face generation: commands to generate the results file for the faces + Body generation: commands to generate the results file for the bodies + Visualization: how to call into the rendering api. For full details, please refer to this repo. * Training from scratch (3 models): scripts to get the training pipeline running from scratch for face, guide poses, and body models. + Face diffusion model + Body diffusion + Body vq vae + Body guide transformer We annotate code that you can directly copy and paste into your terminal using the icon. Quickstart With this demo, you can record an audio clip and select the number of samples you want to generate. Make sure you have CUDA 11.7 and gcc/++ 9.0 for pytorch3d compatibility Install necessary components. This will do the environment configuration and install the corresponding rendering assets, prerequisite models, and pretrained models: conda create --name a2p_env python=3.9 conda activate a2p_env sh demo/install.sh Run the demo. You can record your audio and then render corresponding results! python -m demo.demo First, record your audio [demo1] [?] Hold tight because the rendering can take a while! You can change the number of samples (1-10) you want to generate, and download your favorite video by clicking on the download button on the top right of each video. [demo2] Installation The code has been tested with CUDA 11.7 and python 3.9, gcc/++ 9.0 If you haven't done so already via the demo setup, configure the environnments and download prerequisite models: conda create --name a2p_env python=3.9 conda activate a2p_env pip install -r scripts/requirements.txt sh scripts/download_prereq.sh To get the rendering working, please also make sure you install pytorch3d. pip install "git+https://github.com/facebookresearch/pytorch3d.git" Please see CA Bodies repo for more details on the renderer. Download data and models To download any of the datasets, you can find them at https:// github.com/facebookresearch/audio2photoreal/releases/download/v1.0/ .zip, where you can replace with any of PXB184, RLW104, TXB805, or GQS883. wget https://github.com/facebookresearch/audio2photoreal/releases/download/v1.0/.zip unzip .zip -d dataset/ rm .zip To download all of the datasets, you can simply run the following which will download and unpack all the models. sh scripts/download_alldatasets.sh Similarly, to download any of the models, you can find them at http:/ /audio2photoreal_models.berkeleyvision.org/_models.tar. # download the motion generation wget http://audio2photoreal_models.berkeleyvision.org/_models.tar tar xvf _models.tar rm _models.tar # download the body decoder/rendering assets and place them in the right place mkdir -p checkpoints/ca_body/data/ wget https://github.com/facebookresearch/ca_body/releases/download/v0.0.1-alpha/.tar.gz tar xvf .tar.gz --directory checkpoints/ca_body/data/ rm .tar.gz You can also download all of the models with this script: sh scripts/download_allmodels.sh The above model script will download both the models for motion generation and the body decoder/rendering models. Please view the script for more details. Dataset Once the dataset is downloaded and unzipped (via scripts/ download_datasets.sh), it should unfold into the following directory structure: |-- dataset/ |-- PXB184/ |-- data_stats.pth |-- scene01_audio.wav |-- scene01_body_pose.npy |-- scene01_face_expression.npy |-- scene01_missing_face_frames.npy |-- ... |-- scene30_audio.wav |-- scene30_body_pose.npy |-- scene30_face_expression.npy |-- scene30_missing_face_frames.npy |-- RLW104/ |-- TXB805/ |-- GQS883/ Each of the four participants (PXB184, RLW104, TXB805, GQS883) should have independent "scenes" (1 to 26 or so). For each scene, there are 3 types of data annotations that we save. *audio.wav: wavefile contining the raw audio (two channels, 1600*T samples) at 48kHz; channel 0 is the audio associated with the current person, channel 1 is the audio associated with their conversational partner. *body_pose.npy: (T x 104) array of joint angles in a kinematic skeleton. Not all of the joints are represented with 3DoF. Each 104-d vector can be used to reconstruct a full-body skeleton. *face_expression.npy: (T x 256) array of facial codes, where each 256-d vector reconstructs a face mesh. *missing_face_frames.npy: List of indices (t) where the facial code is missing or corrupted. data_stats.pth: carries the mean and std for each modality of each person. For the train/val/test split the indices are defined in data_loaders/ data.py as: train_idx = list(range(0, len(data_dict["data"]) - 6)) val_idx = list(range(len(data_dict["data"]) - 6, len(data_dict["data"]) - 4)) test_idx = list(range(len(data_dict["data"]) - 4, len(data_dict["data"]))) for any of the four dataset participants we train on. Visualize ground truth If you've properly installed the rendering requirements, you can then visualize the full dataset with the following command: python -m visualize.render_anno --save_dir --data_root --max_seq_length The videos will be chunked lengths according to specificed --max_seq_length arg, which you can specify (the default is 600). For example, to visualize ground truth annotations for PXB184, you can run the following. python -m visualize.render_anno --save_dir vis_anno_test --data_root dataset/PXB184 --max_seq_length 600 Pretrained models We train person-specific models, so each person should have an associated directory. For instance, for PXB184, their complete models should unzip into the following structure. |-- checkpoints/ |-- diffusion/ |-- c1_face/ |-- args.json |-- model:09d.pt |-- c1_pose/ |-- args.json |-- model:09d.pt |-- guide/ |-- c1_pose/ |-- args.json |-- checkpoints/ |-- iter-:07d.pt |-- vq/ |-- c1_pose/ |-- args.json |-- net_iter:06d.pth There are 4 models for each person and each model has an associated args.json. 1. a face diffusion model that outputs 256 facial codes conditioned on audio 2. a pose diffusion model that outputs 104 joint rotations conditioned on audio and guide poses 3. a guide vq pose model that outputs vq tokens conditioned on audio at 1 fps 4. a vq encoder-decoder model that vector quantizes the continuous 104-d pose space. Running the pretrained models To run the actual models, you will need to run the pretrained models and generate the associated results files before visualizing. Face generation To generate the results file for the face, python -m sample.generate --model_path --num_samples --num_repetitions --timestep_respacing ddim500 --guidance_param 10.0 The should be the path to the diffusion model that is associated with generating the face. E.g. for participant PXB184, the path might be ./checkpoints/diffusion/c1_face/model000155000.pt The other parameters are: --num_samples: number of samples to generate. To sample the full dataset, use 56 (except for TXB805, whcih is 58). --num_repetitions: number of times to repeat the sampling, such that total number of sequences generated is (num_samples * num_repetitions). --timestep_respacing: how many diffusion steps to take. Format will always be ddim. --guidance_param: how influential the conditioning is on the results. I usually use range 2.0-10.0, and tend towards higher for the face. A full example of running the face model for PXB184 with the provided pretrained models would then be: python -m sample.generate --model_path checkpoints/diffusion/c1_face/model000155000.pt --num_samples 10 --num_repetitions 5 --timestep_respacing ddim500 --guidance_param 10.0 This generates 10 samaples from the dataset 1 time. The output results file will be saved to: ./checkpoints/diffusion/c1_face/ samples_c1_face_000155000_seed10_/results.npy Body generation To generate the corresponding body, it will be very similar to generating the face, except now we have to feed in the model for generating the guide poses as well. python -m sample.generate --model_path --resume_trans --num_samples --num_repetitions --timestep_respacing ddim500 --guidance_param 10.0 Here, should point to the guide transformer. The full command would be: python -m sample.generate --model_path checkpoints/diffusion/c1_pose/model000340000.pt --resume_trans checkpoints/guide/c1_pose/checkpoints/iter-0100000.pt --num_samples 10 --num_repetitions 5 --timestep_respacing ddim500 --guidance_param 10.0 Similarly, the output will be saved to: ./checkpoints/diffusion/ c1_pose/samples_c1_pose_000340000_seed10_guide_iter-0100000.pt/ results.npy Visualization On the body generation side of things, you can also optionally pass in the --plot flag in order to render out the photorealistic avatar. You will also need to pass in the corresponding generated face codes with the --face_codes flag. Optionally, if you already have the poses precomputed, you an also pass in the generated body with the --pose_codes flag. This will save videos in the same directory as where the body's results.npy is stored. An example of the full command with the three new flags added is: python -m sample.generate --model_path checkpoints/diffusion/c1_pose/model000340000.pt --resume_trans checkpoints/guide/c1_pose/checkpoints/iter-0100000.pt --num_samples 10 --num_repetitions 5 --timestep_respacing ddim500 --guidance_param 10.0 --face_codes ./checkpoints/diffusion/c1_face/samples_c1_face_000155000_seed10_/results.npy --pose_codes ./checkpoints/diffusion/c1_pose/samples_c1_pose_000340000_seed10_guide_iter-0100000.pt/results.npy --plot The remaining flags can be the same as before. For the actual rendering api, please see Codec Avatar Body for installation etc. Important: in order to visualize the full photorealistic avatar, you will need to run the face codes first, then pass them into the body generation code. It will not work if you try to call generate with --plot for the face codes. Training from scratch There are four possible models you will need to train: 1) the face diffusion model, 2) the body diffusion model, 3) the body vq vae, 4) the body guide transformer. The only dependency is that 3) is needed for 4). All other models can be trained in parallel. 1) Face diffusion model To train the face model, you will need to run the following script: python -m train.train_diffusion --save_dir --data_root --batch_size --dataset social --data_format face --layers 8 --heads 8 --timestep_respacing '' --max_seq_length 600 Importantly, a few of the flags are as follows: --save_dir: path to directory where all outputs are stored --data_root: path to the directory of where to load the data from --dataset: name of dataset to load; right now we only support the 'social' dataset --data_format: set to 'face' for the face, as opposed to pose --timestep_respacing: set to '' which does the default spacing of 1k diffusion steps --max_seq_length: the maximum number of frames for a given sequence to train on A full example for training on person PXB184 is: python -m train.train_diffusion --save_dir checkpoints/diffusion/c1_face_test --data_root ./dataset/PXB184/ --batch_size 4 --dataset social --data_format face --layers 8 --heads 8 --timestep_respacing '' --max_seq_length 600 2) Body diffusion model Training the body model is similar to the face model, but with the following additional parameters python -m train.train_diffusion --save_dir --data_root --lambda_vel --batch_size --dataset social --add_frame_cond 1 --data_format pose --layers 6 --heads 8 --timestep_respacing '' --max_seq_length 600 The flags that differ from the face training are as follows: --lambda_vel: additional auxilary loss for training with velocity --add_frame_cond: set to '1' for 1 fps. if not specified, it will default to 30 fps. --data_format: set to 'pose' for the body, as opposed to face A full example for training on person PXB184 is: python -m train.train_diffusion --save_dir checkpoints/diffusion/c1_pose_test --data_root ./dataset/PXB184/ --lambda_vel 2.0 --batch_size 4 --dataset social --add_frame_cond 1 --data_format pose --layers 6 --heads 8 --timestep_respacing '' --max_seq_length 600 3) Body VQ VAE To train a vq encoder-decoder, you will need to run the following script: python -m train.train_vq --out_dir --data_root --batch_size --lr 1e-3 --code_dim 1024 --output_emb_width 64 --depth 4 --dataname social --loss_vel 0.0 --add_frame_cond 1 --data_format pose --max_seq_length 600 For person PXB184, it would be: python -m train.train_vq --out_dir checkpoints/vq/c1_vq_test --data_root ./dataset/PXB184/ --lr 1e-3 --code_dim 1024 --output_emb_width 64 --depth 4 --dataname social --loss_vel 0.0 --data_format pose --batch_size 4 --add_frame_cond 1 --max_seq_length 600 4) Body guide transformer Once you have the vq trained from 3) you can then pass it in to train the body guide pose transformer: python -m train.train_guide --out_dir --data_root --batch_size --resume_pth --add_frame_cond 1 --layers 6 --lr 2e-4 --gn --dim 64 For person PXB184, it would be: python -m train.train_guide --out_dir checkpoints/guide/c1_trans_test --data_root ./dataset/PXB184/ --batch_size 4 --resume_pth checkpoints/vq/c1_vq_test/net_iter300000.pth --add_frame_cond 1 --layers 6 --lr 2e-4 --gn --dim 64 After training these 4 models, you can now follow the "Running the pretrained models" section to generate samples and visualize results. You can also visualize the corresponding ground truth sequences by passing in the --render_gt flag. License The code and dataset are released under CC-NC 4.0 International license. About Code and dataset for photorealistic Codec Avatars driven from audio Resources Readme License View license Code of conduct Code of conduct Security policy Security policy Activity Stars 220 stars Watchers 8 watching Forks 7 forks Report repository Releases 1 v1.0 Latest Jan 2, 2024 Packages 0 No packages published Languages * Python 59.4% * Jupyter Notebook 40.4% * Shell 0.2% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.