https://magic-edit.github.io/

MagicEdit:

High-Fidelity Temporally Coherent Video Editing

Jun Hao Liew*,
Hanshu Yan*,
Jianfeng Zhang,
Zhongcong Xu,
Jiashi Feng
ByteDance Inc.   ^*Equal Contribution
arXiv Code

MagicEdit explicitly disentangles the learning of appearance and
motion to achieve high-fidelity and temporally coherent video
editing. It supports various editing applications, including video
stylization, local editing, video-MagicMix and video outpainting.

Video stylization

Video stylization enables one to (1) transform the source video into
a new video with a style-of- interest (e.g., realistic, cartoon), or
(2) creating a new scene with different subject (e.g., dog - cat) and
different background (e.g., living room - beach).

---------------------------------------------------------------------
                                  
                              > Animals

---------------------------------------------------------------------



                       "a monkey, playing on   "a teddy bear, eating
Source video           the coast, sea"         apple, green grassland
                                               with flowers"



Source video           "a cute cat, sitting    "a labrador dog,
                       on the ground"          sitting on the table"



Source video           "a husky dog, lying on  "a sea lion, lying on
                       the sands"              the ice, winter, snow"

---------------------------------------------------------------------
                                  
                              > Objects

---------------------------------------------------------------------



                       "a red car, moving on   "a red car, moving on
Source video           the road, autumn,       the road, mountain,
                       maple leaves"           green grass and trees"



                       "black and white        "bricks, falling down
Source video           biscuits, falling down  sequentially"
                       sequentially"



Source video           "stools, moving on the  "white cupcakes,
                       railway track"          moving on the table"

---------------------------------------------------------------------
                                  
                              > Scenes

---------------------------------------------------------------------



                       "boats and buildings,   "boats, floating on
Source video           floating in the space,  the sea, villas on the
                       stars"                  coastal"



                       "a submarine            "motorbike travelling
Source video           travelling in the sea,  in the tunnel, with
                       fish swimming"          graffiti on the wall"



                       "red train travelling,  "train travelling,
Source video           snow, winter"           japanese, Cherry
                                               blossoms"

---------------------------------------------------------------------
                                  
                               > Human

---------------------------------------------------------------------



                                    "a pretty               "a pretty
            "a pretty               girl,                   girl,
            girl,                   white                   white
Source      pink        Source      dress,      Source      singlet,
video       dress,      video       dark        video       dark
            living                  hair,                   pants, on
            room"                   flowers"                the
                                                            stage"



            "a pretty                                       "a pretty
            girl,                   "a pretty               girl,
Source      yellow      Source      girl, red   Source      white
video       dress,      video       dress,      video       dress,
            beach,                  ballroom"               dark
            coconut                                         hair"
            trees"

---------------------------------------------------------------------

Local editing

MagicEdit also allows users to make local modification to the video
while leaving other regions untouched (e.g., make the young lady wear
glasses).

---------------------------------------------------------------------



Source     "a          Source     "wearing     Source     "a
video      handsome    video      sunglasses"  video      handsome
           man"                                           man"



Source      "a senior   Source      "a young    Source      "a pretty
video       lady"       video       lady"       video       girl"

---------------------------------------------------------------------

Video-MagicMix

Similar to MagicMix, we can also mix two different concepts (e.g.,
rabbit and tiger) in the video domain to create a novel concept
(e.g., a rabbit-alike tiger).

---------------------------------------------------------------------



              Source video  "+ tiger"     "+ piglet"



              Source video  "+ tiger"     "+ cat"

---------------------------------------------------------------------

Video Outpainting

MagicEdit also supports video outpainting task without any
re-training.

---------------------------------------------------------------------
                                  
Source      Outpainted  Source      Outpainted  Source      Outpainted
video                   video                   video



"a handsome man,        "a cute dog, garden,    "a pretty girl,
jogging, on the road,   flowers"                sunset"
sunny"



"a pretty girl, grey    "an eagle, sea"        "a brown bear, forest"
t-shirt"

---------------------------------------------------------------------
                                  
         Video outpainting with different ratios and prompts

---------------------------------------------------------------------





Prompt = "a handsome man, jogging, on the      Prompt + "long pants"
road, trees, sunny"

BibTeX

@inproceedings{liew2023magicedit,
    author = {Liew, Jun Hao and Yan, Hanshu and Zhang, Jianfeng and Xu, Zhongcong and Feng, Jiashi},
    title = {MagicEdit: High-Fidelity and Temporally Coherent Video Editing},
    booktitle={arXiv},
    year = {2023}
}