[HN Gopher] MagicEdit: High-fidelity temporally coherent video e...
___________________________________________________________________
MagicEdit: High-fidelity temporally coherent video editing
Author : lnyan
Score : 178 points
Date : 2023-08-29 15:57 UTC (7 hours ago)
(HTM) web link (magic-edit.github.io)
(TXT) w3m dump (magic-edit.github.io)
| [deleted]
| jdprgm wrote:
| I have been reading a lot of adjacent papers to this recently.
| Here is a useful collection for anyone interested:
| https://github.com/zengyh1900/Awesome-Image-Inpainting
|
| I have noticed an overwhelming trend that the vast majority of
| authors tend to be Chinese sounding names even when associated
| with an US based university. Obviously some of those could be
| Americans as well but it stood out to such a degree I was curious
| if anyone had any insight.
|
| Also if there is code (this projects links to github but it's
| empty), it tends to be basically abandonware once these papers
| are published with no effort towards commercializing or turning
| into healthy open source projects for some reason.
| thatguy27 wrote:
| No code...
| [deleted]
| chankstein38 wrote:
| Right? I clicked on the "Code" link expecting code and found
| code for the project presentation page basically. The paper
| seems cool but also looks like something that'd take me a while
| to implement and I don't really have time right now.
| satvikpendem wrote:
| Imagine in 5-10 years where, just like people make video games
| all by themselves or with a small group of people, people can
| make their own movies that rival Hollywood productions, for a
| fraction of the cost as there's no need to hire anyone. When the
| output is just pixels on a screen and you can manipulate the
| placement of every pixel, it's really no different to Hollywood
| making them with real actors and crew members or someone drawing
| them, as is the case with current animation methods. Now, we can
| do the same process as drawing each frame but then make it look
| photorealistic.
| rebuilder wrote:
| I think we'll find, among other things, that making tons of
| minute decisions is exhausting. Iterating your way to an end
| result is a way of getting _something_ , but so far, the rule
| of thumb seems to be that the more control you have over a
| creative process, the harder it is to not end up with a stiff
| end product.
| flir wrote:
| We should call this the Chinese Democracy Problem.
| esperent wrote:
| Why?
| flir wrote:
| https://en.wikipedia.org/wiki/Chinese_Democracy
|
| A decade of perfectionism resulting in a mediocre
| product.
|
| That reference going over peoples' heads is just more
| evidence that I'm really, really old.
| StevePerkins wrote:
| Things will certainly change. However, I don't think
| professional studio productions will go away.
|
| In the age of MP3's and streaming services, "anyone" can put
| their music out there. And while many do, the most popular and
| successful music still comes from major labels.
|
| With Amazon's distribution platform and tooling for self-
| published authors, "anyone" can write a book. And while many do
| put out some great stuff, the overwhelming majority of books
| that don't come from major publishers are shovelware.
|
| As shitty of a curator as Hollywood might be, for most people
| there is value-add in curation.
| extr wrote:
| It's identical to the current state of the publishing world.
| They've already gone through this transition: the marginal
| cost to "publish" a book is zero. However to "publish a book
| that will people will read" is a different question
| altogether and is still very much gatekept by the major
| publishing houses.
| rcme wrote:
| Doesn't TikTok solve the distribution problem in a way
| that's not currently possible with published literature?
| echelon wrote:
| > It's identical to the current state of the publishing
| world. They've already gone through this transition: the
| marginal cost to "publish" a book is zero. However to
| "publish a book that will people will read" is a different
| question altogether and is still very much gatekept by the
| major publishing houses.
|
| Not at all identical.
|
| Writing and selling a book only requires a text editor (or
| typewriter), maybe a copy editor, and maybe a publicist.
| It's been that way for the better part of a century.
|
| Present-day films require enormous capital, equipment,
| logistics, personnel, pre-production, post-production, and
| marketing.
|
| Film is about to be wholly transformed by several orders of
| magnitude of cost, talent, personnel, and logistics
| reductions. All at the same time.
|
| Kids at home will have access to this tech and grow up on
| it.
| CyberDildonics wrote:
| That already happened. Cameras, lights and a computer
| that can edit are hundreds of time cheaper than they were
| a few decades ago. Kids can make videos, but they aren't
| making movies anyone wants to watch.
| echelon wrote:
| You always take the contrarian opinion to my comments,
| CyberDildonics.
|
| > That already happened. Cameras, lights and a computer
| that can edit are hundreds of time cheaper than they were
| a few decades ago.
|
| We're not talking about the same thing. None of these
| trends towards fast and dirty filmmaking incorporated Gen
| AI. You couldn't get Hollywood-quality out of an Android
| phone and a boom microphone. That will change.
|
| > Kids can make videos, but they aren't making movies
| anyone wants to watch.
|
| They're certainly watching each other's content. YouTube
| is filled with lots of young creators with enormous
| audiences. That trend will continue to grow.
| r00fus wrote:
| I'm finding that fanfic seems quite good (though the fan-
| service is a bit glaring) these days and it's entirely
| free.
| organsnyder wrote:
| There's still a lot of value in the publishing process.
| Yes, the gatekeeping is inequitable and suboptimal, but
| "publishing" is more than simply formatting a manuscript
| and making it available in retail channels. And a competent
| editor is doing much more than simply proofreading.
| cpill wrote:
| like what?
| johncalvinyoung wrote:
| And aside from Kindle, even the 'formatting the
| manuscript' part is very much underestimated by most
| self-publishers. Professional-grade typesetting and cover
| design makes a great deal of difference to reading
| comfort, shelf appeal, and overall perception of value.
| novok wrote:
| I think it will create a new middle class artist category,
| much like youtube, commodity video editing software and good
| 4k cameras have done today. Certain kinds of niche content
| will become more viable.
| TylerE wrote:
| The majors aren't so major anymore. Many big artists actually
| have their own labels now. A lot of 'em have distro deals...
| but it's a very different model. No advances, so the artist
| pays all recording costs, but they also end up owning the
| masters and having total creative control.
| 01100011 wrote:
| > In the age of MP3's and streaming services, "anyone" can
| put their music out there.
|
| Not the same thing. Putting music out still requires a lot
| of.. artistry. The tools don't do the work for you.
|
| "AI" is heading in a direction where you only need to supply
| the most rudimentary of ideas and it fills in the rest. At
| some point you won't even need to supply the ideas as the
| system will have some concept of what pleases you.
| echelon wrote:
| > Things will certainly change. However, I don't think
| professional studio productions will go away.
|
| You'll still have auteurs such as Wes Anderson shooting film,
| but the era of $100 million dollar blockbusters is coming to
| an end.
| tomjen3 wrote:
| Its an interesting concept, which will further fragment the
| amount of attention that each individual piece can get. If the
| actors are AI generated, then we are also past the point there
| sex and nudity can't be part of the experience. It will likely
| be more attractive than anything else for that reason alone.
| satvikpendem wrote:
| It'll be like YouTube, anyone can put out anything they want
| there, the vast majority of which will be bad, but there are
| still good content creators making high quality videos.
| tomjen3 wrote:
| I don't think so. There is a qualitative difference between
| youtube and what can be made for streaming, but there are
| also entire groups of things that are not on youtube: there
| are lots of video essays, but basically no TV shows over
| multiple episodes, in any genre.
|
| This is currently a limitation, but it doesn't have to be
| and with the new system it won't be.
| satvikpendem wrote:
| Huh? Many content creators on YouTube have made TV shows
| over many episodes on YouTube, Wong Fu for one.
| duped wrote:
| imho the thing that makes film the pinnacle of art is that it
| is the amalgamation of many artists' vision and interpretation.
| And what makes great film _great_ is when those differences
| form a cohesive story.
|
| And at the end of the day, art is a reflection of the human
| condition. Removing humans from the process is not a feature,
| it's a bug because it reduces the scope at which the art can
| address.
| throwuwu wrote:
| Since when is film the pinnacle of art? If collaboration
| makes it so then both music and architecture also have that
| trait. Theatre as well.
| duped wrote:
| imho: in my humble opinion.
|
| And you're right, but you also need to make all those art
| forms to make great film.
| satvikpendem wrote:
| I don't agree that film is the "pinnacle of art," but even
| discounting that, there are many works made by individuals
| throughout history that are _great,_ books and paintings for
| example. I don 't see anyone saying that these should have
| more than one person making such works but for movies and
| shows, it seems that we are used to the status quo and cannot
| see how in the future that great works can similarly arise
| from individuals via the advances of technology. I'm sure in
| 2100 when this tech is commonplace that people will think
| about how archaic it was to have thousands of people work to
| produce a film.
| dylan604 wrote:
| Even in animation, there are teams of people involved. Each
| team specializing in different aspects. This single person
| hollywood replacement dream is something to be really afraid
| of, at least as far as the quality of the content. I couldn't
| imagine watching something that only one person has ever worked
| on with no input from other people with
| suggestions/tweaks/edits to improve the product. We've already
| seen things like True Detective Season 2 that was produced and
| it _had_ people involved that did not push back.
| satvikpendem wrote:
| I played Stardew Valley, a game made entirely by one person,
| and by all accounts it is one of the highest rated games of
| all time. Can anyone make shit? Sure, that is true in any
| medium. But individuals or even small groups of people can do
| amazing things, if they have the tools. It is not "something
| to be really afraid of," which I find to be an extremely
| hyperbolic view.
| dylan604 wrote:
| Have you seen The Room? Anytime I see the same person's
| name in the credits for all the roles, I immediately start
| to get nervous.
|
| Also, small teams is not a single person
| satvikpendem wrote:
| > * Have you seen The Room? Anytime I see the same
| person's name in the credits for all the roles, I
| immediately start to get nervous.*
|
| Sounds like that's more of a personal problem than one
| about this sort of video generation in general.
|
| > _Also, small teams is not a single person_
|
| Yes, which is why I likened this to what was in my
| original comment: "just like people make video games all
| by themselves or _with a small group of people_ "
| cultofmetatron wrote:
| theres also movies like Primer
| (https://en.wikipedia.org/wiki/Primer_(film)
|
| which shows what can be create with minimal budget and
| crew. I heard a lot of the cast also doubled as backstage
| workers during the shoot.
| dylan604 wrote:
| >Yes, which is why I likened this to what was in my
| original comment: "just like people make video games all
| by themselves or with a small group of people"
|
| You're reading that out of context. I intentionally
| separated that comment. The original post I replied
| discussed all of the work being done by a single person.
| That what this is about. You introduced a small team. A
| small team is not one person. A small team still has the
| potential of having discussion on edits, creative, etc vs
| just the ideas of one person.
| satvikpendem wrote:
| You replied to my original post, did you not [0]? The
| post where I, as stated above, also included the words,
| "all by themselves or with a small group of people," no?
| The post where I explicitly did not "[discuss] all of the
| work being done by a single person" that denies work also
| being done by groups. You were the one who omitted the
| "with a small group of people" part and started talking
| only about a "single person hollywood replacement dream."
|
| I think you're just arguing semantics at this point, as
| to me, it doesn't really matter if it's a single person
| or a small team making stuff, my greater point was that
| it will be a lot cheaper than a full-blown Hollywood
| production and will usher in a new sort of industry like
| YouTubers today, as ubiquitous phone cameras have, only
| now these creators will also make photorealistic
| productions rather than just filming themselves doing
| things.
|
| [0] https://news.ycombinator.com/item?id=37310759
| chankstein38 wrote:
| It's also a kind of cult classic and there's strong
| evidence to suggest he knew what he was doing by making a
| horrible movie.
| NBJack wrote:
| Stardew Valley is great. But note that the author took a
| classic approach to solo assets and devlopment: pixel art
| on a 2d canvas. This is a great game in a well established
| medium, and the concept itself is part homage to Harvest
| Moon, originally a 2d title released in 1996.
|
| Contrast this with the fact that Steam is now averaging
| about 1000 new games per month. [1] There are undoubtedly
| some excellent games in there that haven't survived the
| onslaught of choices. Sadly, finding them without either
| significant marketing by the dev or the right conditions
| makes it nearly impossible to sift out the gems from the
| asset flips.
|
| [1] https://steamdb.info/stats/releases/
| brucethemoose2 wrote:
| And yet Steam is a shining example of curation compared
| to, say, the App Store or Google Play.
|
| Excellent curation will be critical to gen AI. The window
| for such curation to be established feels extremely
| small, otherwise "app stores" will take hold and we will
| end up with a sea of unnavigable spam.
| throwaway287391 wrote:
| Even assuming it's impossible for a single person to produce
| something as good as a large team, why would this be
| "something to be really afraid of"? There will still be
| demand for high-quality films -- why wouldn't that demand
| continue to be met? It's been the status quo for at least a
| decade that one "normal person" can make and publish a
| "movie" (e.g. filming something on their phone and posting it
| on YouTube), and yet Hollywood somehow hasn't been upended.
| rossjudson wrote:
| I think you'll see that high production quality becomes more
| accessible, while content quality remains as elusive as it has
| ever been...
|
| Unless you can successfully train the population to equate
| content quality with production values.
| [deleted]
| cdchn wrote:
| All of the source videos that have people's faces seem to be
| blurred. Are they blurred before transforming or do they blur it
| just to post the samples on their web page?
| AbrahamParangi wrote:
| The latter, presumably they didn't ask.
| [deleted]
| capableweb wrote:
| Interesting enough ByteDance (TikTok/Douyin) seems to be behind
| it.
| ipsum2 wrote:
| ByteDance does a lot of ML research on music/video.
| danielbln wrote:
| Their video editor CapCut is really good.
| simlevesque wrote:
| Their multi-object tracker ByteTrack is also one of the
| best available.
| xnx wrote:
| Coming soon to TikTok filters (if it doesn't already play a
| role in some of them)
| aantix wrote:
| [flagged]
| Timon3 wrote:
| It's crazy how consistent the Shutterstock logo is in the
| outpainted examples.
| [deleted]
| jjcm wrote:
| Seems like it's reversed along the Y axis as well? I'm curious
| what led to that. The nefarious side of my brain say it was a
| very basic attempt at making the source training data less
| immediately recognizable in any generated output, but I do
| wonder if there's a more innocent explanation.
| gs17 wrote:
| A "more innocent explanation" could simply be data
| augmentation. It seems pretty clear they don't care that it's
| obviously using watermarked Shutterstock videos.
| brink wrote:
| Kendrick Lamar: "I'm so sick and tired of the Photoshop"
|
| Who actually wants to look at this? It's a neat trick, but I
| greatly prefer to look at what's real, and I imagine most people
| outside of this AI hype bubble do too. It's implied that the
| stable diffusion here is making the video better, but by most
| definitions of the word "better", it's not.
| htrp wrote:
| Looks like its building on the same concepts as stable video.
|
| https://github.com/rese1f/StableVideo
| dylan604 wrote:
| what would happen if they didn't say "pretty girl"?
| pridkett wrote:
| I'm disappointed that all of their examples of humans are "a
| pretty girl". Yeah, I get that people use generative models for
| that, but there's a lot more you can do.
| phren0logy wrote:
| They seem to have thrown a few "handsome man" examples in
| there, too.
| dylan604 wrote:
| That's not the point though. To me, it reeks of a bunch of
| dude bros. I'm guessing they can't say "a hot chick"
| otherwise the LLM would show a baby chicken on fire or at
| least sweating in front of a fan??? Does "hot chick" even
| translate to Chinese well?
|
| It's the fact that they felt the need to use "pretty"
| instead of just "girl" or young woman.
| [deleted]
| xnx wrote:
| This looks like a nice improvement on current video to video
| techniques: https://stable-diffusion-art.com/video-to-video/
___________________________________________________________________
(page generated 2023-08-29 23:00 UTC)