[HN Gopher] High-Speed Face-Tracking for Dynamic Facial Projecti...
___________________________________________________________________
High-Speed Face-Tracking for Dynamic Facial Projection Mapping
Author : bukka
Score : 104 points
Date : 2025-01-31 16:37 UTC (3 days ago)
(HTM) web link (www.vision.ict.e.titech.ac.jp)
(TXT) w3m dump (www.vision.ict.e.titech.ac.jp)
| bhouston wrote:
| Neat stuff. I wonder if one could use a neural network to try to
| do prediction as well.
|
| Human movements are super stereotyped in the face and you could
| likely predict with decent accuracy the next frame given the
| previous frames.
| tsumnia wrote:
| Check out a lot of DisneyResearchHub's videos on YouTube. They
| are regularly posting research on object/face tracking and
| projections.
|
| That said as you mentioned faces are pretty stereotyped. We
| "solved" face mapping two decades ago with Tim Cootes and Paul
| Ekman's work. We're able to quickly map rough estimates using
| traditional haar cascade classifiers and Viola-Jones with
| AdaBoost. Neural Networks may help, but we have other solutions
| that also handle the problem with "relative" ease (ignoring
| lighting, occlusion, etc.)
| _flux wrote:
| Pretty cool. I've had this idea of converting face data into MIDI
| CC data to use with DAWs or synths, this could be a useful
| algorithm for it.
|
| Is the source code available?
| sporkl wrote:
| Not this project, I've seen this something like this done with
| Google's MediaPipe models to get face tracking data into
| Max/MSP.
|
| https://cycling74.com/forums/n4m-facemesh-handpose-google-me...
| croes wrote:
| > to create immersive experiences that have been used in the
| makeup and entertainment industries.
|
| Or for fake propaganda videos. Dangerous time for these kinds of
| progress.
| Hugsun wrote:
| What sort of propaganda do you forsee this enabling?
| croes wrote:
| ,,Leaked" videos of political enemies saying abysmal things
|
| The Protocols of the Elders of Zion on steroids.
| grayhatter wrote:
| Uh... this is a white paper about projecting an image onto
| a moving face. Not super imposing, but projecting... with a
| projector... using light... so what exactly are you worried
| about with this tech? Or is it just that you're afraid of
| face tracking in general? And it's not really about this
| topic specifically?
| croes wrote:
| It's about high speed face tracking.
|
| Can you imagine other use cases than projection?
| grayhatter wrote:
| > Can you imagine other use cases than projection?
|
| yes
|
| vtubers
|
| human interface device for quadriplegics
|
| automated camera tracking/zoom for video calls
|
| emotion recognition support software for people with
| difficulty reading or understanding emotions (ASD)
|
| truly smart, smart home that can track and respond to
| gaze, e.g. "can you turn off that light" and it will turn
| off the light they're looking at
|
| sleep detection for smart locks
|
| liveness detection for identify verification systems
|
| vr tracking for interactions that both escape the uncanny
| valley, and feel more real time
|
| need more?
| croes wrote:
| Yes, list the malicious ones too.
| grayhatter wrote:
| impersonation?
|
| are there others?
| HanayamaTriplet wrote:
| You don't need high speed face tracking to make video
| edits, those are not going to be done in real time.
| xfeeefeee wrote:
| The artistic possibilities of this are really fascinating.
| Looking forward to what people come up with! Projection mapping
| interactive art is always mesmerizing
| bj-rn wrote:
| This (older) collab is pretty nice:
| https://www.vision.ict.e.titech.ac.jp/projects/WOW_TOKYO_AYA...
| addandsubtract wrote:
| What is the SotA OSS for face tracking these days? I've never
| been able to make face tracking work without having alignment
| issues during movement.
| brcmthrowaway wrote:
| You mean lag?
|
| try mediapipe w/ gpu support
| zipy124 wrote:
| I don't know if it's SoTa but previously in a joint
| academic+industrial project I've had very good success with
| faceboxes for face tracking and then adaptive Wing loss for
| facial alignment. With c++ and an ONNX runtime, I achieved very
| fast and accurate results, being robust to lighting and harsh
| angles.
|
| https://github.com/protossw512/AdaptiveWingLoss
|
| https://github.com/sfzhang15/FaceBoxes
| vessenes wrote:
| Ooooh, this is so cool. I didn't realize I was seeing projected
| 'makeup' on faces until they showed some lower latency prior
| work. Once I saw it, I could see occasional misalignment. But,
| this is just an awesome and fun thing. I can imagine it being
| just amazing for live theater costuming and story telling.
|
| That said, it looks like the tech currently works at 1-2m range.
| I'd guess this is because that's an easy range for a projector to
| work at and provide high res, in which case there's no reason you
| couldn't get a zoom setup going.
|
| Perhaps some enterprising director of the next Disney theater
| production will put the money in to get this working reliably on
| stage.
| suyash wrote:
| Would love to know how to build such a system but with a normal
| camera and projector if possible for creative art+tech projects!
| brcmthrowaway wrote:
| Can't mediapipe detect faces at 60fps?
| m3kw9 wrote:
| Is good but when you need that extra lens shift camera, it goes
| from mass market to super niche.
| echelon wrote:
| It seems like they've done two things here - the physical
| optical apparatus setup and the algorithm.
|
| I'm wondering why the physical alignment is so important. Are
| camera distortion models and mapping, view projection, etc.
| just too slow or low quality to run?
|
| I suppose I'll have to take a look at their paper later.
___________________________________________________________________
(page generated 2025-02-03 23:01 UTC)