[HN Gopher] Neural Actor: Neural Free-View Synthesis of Human Ac...
___________________________________________________________________
Neural Actor: Neural Free-View Synthesis of Human Actors with Pose
Control
Author : Hard_Space
Score : 134 points
Date : 2021-06-04 12:48 UTC (10 hours ago)
(HTM) web link (gvv.mpi-inf.mpg.de)
(TXT) w3m dump (gvv.mpi-inf.mpg.de)
| Reason077 wrote:
| "Stantard deviation"? Typo?
| TimPC wrote:
| I'm very glad they used a hyphen in the title as it completely
| changes the expected meaning.
| drcode wrote:
| I assume we'll see some optimized version of this for NPCs in
| video games soon.
| spywaregorilla wrote:
| Interesting. My first observation is that the rendered people all
| have really tight clothing on. The model guesses at some creases,
| but it probably wouldn't guesstimate cloth physics very well.
|
| I would tend to think you would have much better results
| synthesizing a mesh with a NN and getting a game engine or
| something to do this part with the animation.
| ridaj wrote:
| That's especially visible when comparing with the "driving
| person", who is floating in a loose open trench-coat that flits
| about as they dance
| kossTKR wrote:
| Love the dancing ha! Also is this ready to be used on a small
| project, knowing very little og NN's?
|
| I just need s bit of motion tracking to be used on various
| figures on a webgl website, have been looking at the iphone depth
| sensor, the kinect and Mixamo.
| m3kw9 wrote:
| How is this different from putting a skin on the real actor?
| montroser wrote:
| This is incredible. Interesting how combining the neural net with
| 3d mesh and texture map provides the best results of all the
| approaches they tried.
|
| I am both excited and terrified for what this stuff may yield in
| our future realities.
| iandanforth wrote:
| This page causes multiple mp4 download prompts to open in Firefox
| (v89.0, OSX) for me. Anyone else seeing this?
| philote wrote:
| Not for me (v88.0.1, OSX)
| Hard_Space wrote:
| No, not for me. I think you'll need to review file association
| actions in Firefox prefs. I have to do this regularly for PDF.
| jkingsman wrote:
| I'm on Chrome 90.0.4430.212 on Catalina and I had multiple
| videos with narration all start on page-open. Had to mute the
| page because I couldn't figure out which one was talking.
| Abishek_Muthian wrote:
| Soon those who have motor control issues or other problems would
| be able to just feed the script and get their virtual self on
| video with all the movements which isn't possible in reality.
|
| Which might be necessary, Considering even a LinkedIn profile
| requires some TikToking now in the name of 'Introduce yourself in
| a 30 second video'.
| grawprog wrote:
| The results remind me of high quality rotoscoped characters. Like
| if the original Mortal Kombat got an HD remaster or something.
| They look good but not real.
|
| If they do start replacing actors with these, everything's gonna
| look like a Ralph Bakshi movie.
| TchoBeer wrote:
| My first thought is that this will make for some pretty good
| memes
| fareesh wrote:
| TikTok filter - now you don't need to dance, just use the filter
| indiantinker wrote:
| Anybody can dance now! yay!
| hliyan wrote:
| Another 10 years of progress, better facial expression control
| and voice synthesis, I suspect highly paid actors will be a thing
| of the past.
| qayxc wrote:
| This has been prophesised before (last in the early 2000s) and
| didn't come to pass.
|
| I wouldn't expect that to change given this technology either.
| There's a market that would benefit greatly from this, though:
| marketing and training videos as well as stock-footage.
|
| Some companies (Samsung comes to mind) already use synthetic
| avatars to represent their products. Artificial influencers and
| the like will also take the place of real humans.
|
| Film actors, though, not so much. There's more to a successful
| actor than just appearing in film (who'd sit at meet & greets,
| autograph signings or attend shows?) so there's little reason
| to replace _living_ actors. Deceased ones, though...
| makapuf wrote:
| The tendency is even reverse with highly paid actors being
| hired for only the voice acting featuring in computer-
| animated movies (where great voice-only performers would be
| cheaper)
| refactor_master wrote:
| Humans basically want to belong to a tribe, which is why
| there's always a certain equilibrium with relatively few
| "famous" people on one side. If AI and the Internet made
| everyone exceptional, nobody would be exceptional, and we'd
| just find random idols to gather around anyway (sort of
| what we already do).
|
| Sure, you _could_ use AI to act, _or_ you could follow your
| most basic gut instinct and slap a famous-for-being-famous
| person on it and call it a day. Because it's going to work.
| CyberDildonics wrote:
| This is nonsense.
|
| First, actors are not famous for being famous, they are
| famous because people recognize them and eventually learn
| their names.
|
| Second, the name recognition is why they get cast as
| voices. Their name can then be used to promote the movie
| and the actors themselves can go out and promote the
| movie.
| refactor_master wrote:
| At first you get famous for being above average. Then you
| become more famous for being famous. Why are they paid in
| the several millions when most jobs are not? Are they
| _that_ much better at their job, than you are at yours?
| Probably not. But now the ball is rolling.
| CyberDildonics wrote:
| I'm sure that is what it might look like with a vague
| awareness of entertainment, but it is not only not
| reality it doesn't even make sense. It also has nothing
| to do with the thread.
| listic wrote:
| I dunno, when I first saw a Vocaloid on stage, my jaw dropped
| as I realized that in 10 years highly paid pop stars will be a
| thing of the past. Yet, it doesn't seem like music industry is
| switching to synthetic avatars.
| acjohnson55 wrote:
| When people talk about replacing artists with virtual artists,
| I think an important missing piece is that people are into
| performance _because_ a human with a story is doing the
| performing, not just the aesthetics of the end result. For the
| most part, great art isn 't hermetic. It's situated within the
| context of the artist's personal story and career arc, an the
| relationship of those to the audience / fanbase.
|
| We probably will see greater uptake of virtual art and artists,
| but it hasn't been and won't be a revolution that sweeps away
| human performance.
| jfengel wrote:
| If anything, it's the low-paid actors that would go instead.
|
| Highly-paid actors are celebrities. They are engaging in a way
| that is incredibly hard to describe but nonetheless seems to be
| widely recognized. They aren't necessarily "good actors" (a
| term that is itself hard to define), but people enjoy looking
| at them. People go to a film just because some well-known actor
| is in it, and that's what makes them so valuable.
|
| Every film set is chock full of talented people. Plenty of
| people could have done great work as, say, Tony Stark.
| Hollywood is full of beautiful and talented women who just
| never manage to get that "it factor". To displace highly-paid
| actors they'll have to replicate that, and that'll open up a
| whole new set of questions.
|
| The ones who go first will be the low-paid actors who surround
| them. They are the ones doing sitcom pilots that get shelved,
| or commercials for dog food and allergy medicine, or being
| saved from having a bus dropped on them. Such actors are
| actually reasonably well paid on the days the work (over $1,000
| a day for big-budget films or TV; $9,064 for a week as a guest
| star), though most of them work only a tiny fraction of the
| time. There are more people making a living at that than being
| a star, though the odds still suck.
| asiachick wrote:
| You're probably right but we already have the example of
| Hatsune Miku. Sure she's an exception but the software isn't
| there yet. At someone point someone will make a truly hit
| movie or TV show with 1 or more main synthetic characters.
|
| I'm sure when Tron came out lots people said "practical
| effects will never disappear". And yet with few exceptions
| they mostly have.
| jfengel wrote:
| We will definitely have digitally-created stars as a
| separate medium. We have for a long time had animated
| characters who don't speak. They still involve intensive
| human direction to create them, but they don't have human
| form or human voice. They're still plenty entertaining.
|
| They get to do things that humans can't do. They can
| entertain in entirely novel ways. Using them as a
| substitute for human actors is kinda the faster-horses-not-
| cars of movies.
|
| We use CGI characters for inhuman effects, but I think it's
| often not very effective. I don't care for most superhero
| movies because I don't really care about watching CGIs
| punch CGIs harder than human actors could do. There aren't
| any stakes; there's no dramatic risk. I'd rather watch a
| martial arts movie, which is limited by what humans are
| capable of.
|
| That's still something we're learning. Part of what made
| the Lord of the Rings films more interesting than the
| Hobbit follow-ons was that the latter used more practical
| effects. New Zealand was a big star in LotR; Hobbit used
| more CGI backgrounds. Audiences felt the sense of place.
|
| We'll improve the tech so that it seems even more
| realistic. The new technology used in The Mandalorian is
| really exciting (though I'd be curious to see how it played
| on a big screen). But it was at its best when it made up
| very real-world seeming worlds, while animation uses
| digital worlds in totally unrealistic and even more
| exciting ways.
|
| So I'm expecting that we'll continue to have practical
| effects in the best movies starring humans, and continue to
| have human stars. But we're still waiting for the first
| mega-hit CGI star who can do whatever magic it is Robert
| Downey Jr. does. When they can catch that, RDJ may well be
| out of a job -- not replaced, but the entire enterprise
| gone.
| knicholes wrote:
| My mind went towards stuntmen. Now, studios can record natural
| actions for a large database of actions, and pick and choose.
| Instead of the Wilhelm Scream, you could see "Flip #564" and
| "Crash #827". If it doesn't even matter what shape the stuntman
| is, it seems to me like the industry might be looking to fill
| that occupation less and less.
| CyberDildonics wrote:
| There have already been CG stunts for multiple decades. You
| still have to match the lighting of the photography to
| integrate the CG even if you have something like this
| captured with flat texture maps.
|
| The fidelity is also a big issue, flat textures and rough
| normals are not going to hold up to much scrutiny.
| eMGm4D0zgUAVXc7 wrote:
| The move to virtual characters is already happening on Twitch /
| YouTube, check out CodeMiko:
|
| https://www.youtube.com/watch?v=CsQjxEd-gsw
| aigen001 wrote:
| It's also makes business sense to use virtual characters.
|
| If you are a streamer whose sole income is based on sites
| like twitch/youtube, getting banned could mean the end of
| your career and social life. Many streamers live together and
| collaborate with one another and sites like twitch have rules
| where you can't stream with banned streamers. So getting
| banned doesn't just mean getting banned from the site, it
| also means getting banned from your friend group because
| interactions with you could get them banned too.
|
| With virtual avatars you could evade a ban by changing your
| model and also use a voice changer. I haven't personally
| heard of any cases where streamers use virtual characters to
| evade bans but I'm sure its at least happened.
|
| It also becomes easier to become a marketing tool. Since Code
| Miko isn't a physical entity, she could easily be used in
| video ads just by sending ad agencies her model. No more
| flights to video shoots, all you need is a 3d animator.
| TimPC wrote:
| I think this technology is in its infancy and we might be
| looking at far more than 10 years to even get the functionality
| to make this possible regardless of whether it could actually
| replace highly paid actors. Occlusion and multi-agent are
| typically hard problems in image spaces, so once you add
| objects and other agents to a scene it becomes a great deal
| more challenging to make the motions life-like and realistic.
| Admittedly if you're mapping onto an actor in the scene with
| other objects and just changing their appearance it might be
| doable but you might run into substantial challenges if your
| target is 5ft8 and your source is 6ft2 (or Vice Versa). If
| you're not using a source-target mapping at all you have the
| challenge of generating realistic movements given all the
| obstacles and interactions, which is much harder than the
| current one-agent zero-object space.
| whitten wrote:
| So this method seems to create an inner core "skeleton" that is
| then used with various shells.
|
| How is this method compared to a commercial product such as Poser
| ?
| qayxc wrote:
| Does Poser work with photos and videos as inputs?
|
| If not, there's your answer.
| bsenftner wrote:
| No.
| gedy wrote:
| You don't need to create or capture a mesh. This is
| synthesizing one from a few 2d images, with impressively
| accurate results.
| wokwokwok wrote:
| Poser requires an existing rigged, textured mesh.
|
| Poser requires that you manually animate the skeleton.
|
| This approach is more like a mixture of mocap/photometry; given
| an input video (a) and (b), synthesise a fully textured model
| from (a) and apply the motion from (b) on it.
|
| Of course, the output of this is horribly bad compared to state
| of the art 3d models; the point is that the inputs are trivial
| and it shows that NeRF are increasingly plausible as a solution
| for real use cases.
|
| Of course... this is all a bit 'proof of concept stuff'.
|
| > To further evaluate our method on a wider variety of body
| poses and more challenging textured clothing, we captured a new
| multi-view human performance corpus with 79 - 86 cameras at a
| resolution of 1285 x 940. It contains four sequences, N1-N4,
| and each has 12,000-16,000 frames for training and around 8,
| 000 frames for testing
|
| These captured images are also, notably, green screen, although
| not with the traditional mocap markers.
|
| So, no, you're not going to get a 3d animation from a video off
| your phone anytime even remotely soon.
|
| ...but, it's still pretty darn awesome, and they're going to
| release the datasets for others to play with.
| l1n wrote:
| I really really wish that there was a norm of uploading code at
| the same time as the paper. The number of "code coming soon!"
| buttons on these sorts of sites (oftentimes several years in the
| past) is far too high. If you can't play around with the model,
| how can you tell the level of cherry-picking that the method has?
___________________________________________________________________
(page generated 2021-06-04 23:01 UTC)