[HN Gopher] Handsfree.js - integrate face, hand, and/or pose tra...
___________________________________________________________________
Handsfree.js - integrate face, hand, and/or pose tracking to front
end projects
Author : detaro
Score : 138 points
Date : 2021-06-05 12:04 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| ranguna wrote:
| Wow this is amazing! I remember a few years ago in my thesis I
| was trying to use something similar in ROS to detect hand
| gestures and make a robot drive toward you, it's incredible to
| see this in the browser!
| rgovostes wrote:
| On Safari for Mac, you must enable Develop - Experimental
| Features - WebGL 2.0.
| esens wrote:
| I have tried the underlying library, MediaPipe/TensorFlowJS, and
| it is only okay. Once you try to use it for visual augmentation,
| I found it doesn't track that well and it has problems with
| scale.
|
| So, while it may look good with the skeletons, notice they do not
| have useful demos, mostly just proof of concept demos that do
| almost nothing.
| blensor wrote:
| Have you tried the newest MoveNet from google? That thing works
| really well. I only started experimenting with the Lighting
| model (not Thunder) for our VR game but even that is pretty
| robust.
|
| If you want to see how stable it is, have a look at that:
| https://youtu.be/zz9S5hgrWpM?t=147
|
| It's tracking me with the Oculus Quest on the head, so it does
| not even have facial features to go on
| jonplackett wrote:
| Thanks for the link - Does MoveNet give you 3D positions /
| rotations for each joint or just a 2D position on the image?
| blensor wrote:
| Just the 2D positions plus a confidence level. In the video
| we are using only one view so it's no true full body
| tracking yet. We are currently trying to use two views to
| get a rough 3D tracking. The good thing is that the
| positions of the detected points are relatively stable so
| with a big enough baseline that should work out for a rough
| 3D pose estimation
| gregsadetsky wrote:
| I'll use this post / opportunity to ask the HN community
|
| (I understand that this is a technically complicated, and
| potentially sensitive subject)
|
| I've been approached by a few people who are doing what they can
| to lower the cost of prescription glasses. It's a mix of
| (licensed) ophthalmologists, people interested in offering
| community health services, etc.
|
| What is the current state of the art for pupillary distance (PD)
| measurement based on face detection / pupil tracking using laptop
| webcams or mobile (front) cameras?
|
| I imagine (obviously?) that there must be scholar research on the
| acceptable error margins for a PD measurement (depending on the
| type of vision condition, i.e. farsightedness, etc.)?
|
| Would using something like Handsfree.js or
| https://github.com/esimov/pigo (which has pupil detection) be a
| good start, or would these be an ~order of magnitude off in terms
| of the necessary margins?
|
| Thanks a lot.
| notum wrote:
| It comes down to physics. Cameras and their lenses come in all
| shapes and sizes, most of webcams don't expose their geometry
| in a programmatically accessible way, even if they did, most of
| them don's have means of determining distance to the subject.
|
| We can detect faces, eyes and pupils, but with no reference
| points we have no idea how big or far apart they are.
| gregsadetsky wrote:
| Thanks!
|
| What about sites (such as Warby Parker) that make you hold up
| a credit card (which has a standard size) and use that to
| determine (via what I assume is a rule of three) the PD? [0]
|
| I can see of course problems with the card not being
| "perpendicular" and other 3d distortions.
|
| [0] - https://www.warbyparker.com/pd/instructions
| mncharity wrote:
| Fwiw, there's https://www.eyeque.com/pdcheck/ .
|
| And Google's MediaPipe Iris used iris diameter for distance
| estimation.[1]
|
| EyeQue is perhaps a cautionary tale. "Change the world" "few
| dollars of optics + phone = D+CYL" become $45 one-person-per-
| subscription "vision tracker".
|
| [1] https://ai.googleblog.com/2020/08/mediapipe-iris-real-
| time-i...
| twobitshifter wrote:
| What are some of the applications for this?
| umvi wrote:
| You could integrate this with adtech to help make
| advertisements more interactive and to analyze how the viewer
| is receiving the ad by their pose and facial expressions. Based
| on that you could have an algorithm serve only ads that improve
| facial expressions and poses to positive ones
| Solocomplex wrote:
| Another excellent reason to disable JavaScript in your
| browser
| jcdietrich wrote:
| Could play punch the monkey!
| lolinder wrote:
| I hope we haven't yet gotten to the point where people will
| mindlessly click "Yes!" when a random website asks for webcam
| access.
| Gys wrote:
| https://github.com/midiblocks/handsfree#examples
| blensor wrote:
| I can't speak about this specific project but pose tracking in
| the browser can actually be really useful to get it on mobile
| without the need for dedicated apps.
|
| We are currently experimenting with using phones (or tablets or
| basically anything with a browser and a camera) as a tracking
| source for full body tracking on the Oculus Quest ([1]) and
| being able to just use a Tensorflow model in the browser has
| cut down the time of development on the mobile device side to
| almost nothing.
|
| There are of course a whole lot of other issues one buys into
| with that approach (browser security policies, the whole thing
| not working in certain countries due to problems with loading
| the model) but that's manageable
|
| [1] https://www.youtube.com/watch?v=zz9S5hgrWpM (relevant part
| starts at 0:29)
| danielskogly wrote:
| I found this demo of him playing "Into the Breach" with only
| Face Gestures very impressive!
| https://twitter.com/GoingHandsfree/status/140022171083874713...
| notjustanymike wrote:
| WebXR comes to mind.
| albybisy wrote:
| the library and the demos are not loading for me.
___________________________________________________________________
(page generated 2021-06-05 23:00 UTC)