https://github.com/midiblocks/handsfree Skip to content Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Project management - + Integrations - + GitHub Sponsors - + Customer stories- * Team * Enterprise * Explore + Explore GitHub - Learn and contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} MIDIBlocks / handsfree * Notifications * Star 841 * Fork 42 Quickly integrate face, hand, and/or pose tracking to your frontend projects in a snap handsfree.js.org Apache-2.0 License 841 stars 42 forks Star Notifications * Code * Issues 29 * Pull requests 1 * Actions * Projects 3 * Wiki * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 4 branches 38 tags Code Clone HTTPS GitHub CLI [https://github.com/M] Use Git or checkout with SVN using the web URL. [gh repo clone MIDIBl] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @MIDIBlocks MIDIBlocks Merge branch 'dev' ... ddc4682 Mar 13, 2021 Merge branch 'dev' ddc4682 Git stats * 926 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time boilerplate #110 Ups build and updates changelog Mar 13, 2021 build/lib #110 Ups build and updates changelog Mar 13, 2021 docs #110 Ups build and updates changelog Mar 13, 2021 extension #0 Remove unused events Mar 10, 2021 src #110 Ups build and updates changelog Mar 13, 2021 .gitignore #78 Removes requires for CDN to work, ups version to 8.0.2 Dec 19, 2020 .npmignore #102 Removes build/extension from npm Jan 23, 2021 LICENSE #1 Added license and attribution links Nov 6, 2019 README.md #110 Ups build and updates changelog Mar 13, 2021 deploy.sh #63 Defines defaults.feedback.$target in constructor (solves build bug) Nov 23, 2020 manifest.json #0 Starts handsfree-ifying Super Mario 64, new branding Mar 10, 2021 package-lock.json #110 Ups build and updates changelog Mar 13, 2021 package.json #110 Ups build and updates changelog Mar 13, 2021 rollup.config.js #110 Adds babel to rollup to transpile optional chaining Mar 13, 2021 View code Explore the interactive docs at: Handsfree.js.org Contents Quickstart Installing from CDN Installing from NPM Hosting the models yourself Example Workflow Quickstart Workflow Examples Face Tracking Examples Hand Tracking Examples Pose Estimation Examples Local Development Command line scripts Dev Notes The Handsfree Browser Extension How it works How to install Google Chrome Handsfree Browsing Explore the interactive docs at: Handsfree.js.org License & Attributions License: Apache 2.0 Attributions Special Thanks README.md handsfree.js.org Quickly integrate face, hand, and/or pose tracking to your frontend projects in a snap [6874747073] [6874747073] [6874747073] [6874747073] [6874747073] Powered by: [6874747073] [6874747073] [68747470733a2] --------------------------------------------------------------------- Explore the interactive docs at: Handsfree.js.org Or try it right away with the serverless boilerplates in /boilerplate /! --------------------------------------------------------------------- Contents This repo is broken into 3 main parts: The actual library itself found in /src/, the documentation for it in /docs/, and the Handsfree Browser Extension in /extension/. * Handsfree.js + Quickstart with CDN (fastest way to get started) + Quickstart with NPM (requires server or bundler) * Development guide * Handsfree Browser Extension --------------------------------------------------------------------- Quickstart Installing from CDN Note: models loaded from the CDN may load slower on the initial page load, but should load much faster once cached by the browser. This option is great if you don't have or need a server, or if you're prototyping on a site like CodePen. You can also just download this repo and work with one of the /boilerplate/. Installing from NPM # From your projects root npm i handsfree // Inside your app import Handsfree from 'handsfree' // Let's use handtracking and enable the plugins tagged with "browser" const handsfree = new Handsfree({showDebug: true, hands: true}) handsfree.enablePlugins('browser') handsfree.start() Hosting the models yourself The above will load models, some over 10Mb, from the Unpkg CDN. If you'd rather host these yourself (for example, to use offline) then you can eject the models from the npm package into your project's public folder: # Move the models into your project's public directory # - change PUBLIC below to where you keep your project's assets # ON WINDOWS xcopy /e node_modules\handsfree\build\lib PUBLIC # EVERYWHERE ELSE cp -r node_modules/handsfree/build/lib/* PUBLIC import Handsfree from 'handsfree' const handsfree = new Handsfree({ hands: true, // Set this to your where you moved the models into assetsPath: '/PUBLIC/assets', }) handsfree.enablePlugins('browser') handsfree.start() --------------------------------------------------------------------- Example Workflow The following aims to give you a quick overview of how things work. The key takeaway is that everything is centered around hooks/plugins, which are basically named callbacks which are run on every frame and can be toggled on and off. Quickstart Workflow The following workflow demonstrates how to use all features of Handsfree.js. Check out the Guides and References to dive deeper, and feel free to post on the Google Groups or Discord if you get stuck! // Let's enable face tracking with the default Face Pointer const handsfree = new Handsfree({weboji: true}) handsfree.enablePlugins('browser') // Now let's start things up handsfree.start() // Let's create a plugin called "logger" // - Plugins run on every frame and is how you "plug in" to the main loop // - "this" context is the plugin itself. In this case, handsfree.plugin.logger handsfree.use('logger', data => { console.log(data.weboji.morphs, data.weboji.rotation, data.weboji.pointer, data, this) }) // Let's switch to hand tracking now. To demonstrate that you can do this live, // let's create a plugin that switches to hand tracking when both eyebrows go up handsfree.use('handTrackingSwitcher', {weboji} => { if (weboji.state.browsUp) { // Disable this plugin // Same as handsfree.plugin.handTrackingSwitcher.disable() this.disable() // Turn off face tracking and enable hand tracking handsfree.update({ weboji: false, hands: true }) } }) // You can enable and disable any combination of models and plugins handsfree.update({ // Disable weboji which is currently running weboji: false, // Start the pose model pose: true, // This is also how you configure (or pre-configure) a bunch of plugins at once plugin: { fingerPointer: {enabled: false}, faceScroll: { vertScroll: { scrollSpeed: 0.01 } } } }) // Disable all plugins handsfree.disablePlugins() // Enable only the plugins for making music (not actually implemented yet) handsfree.enablePlugins('music') // Overwrite our logger to display the original model APIs handsfree.plugin.logger.onFrame = (data) => { console.log(handsfree.model.pose?.api, handsfree.model.weboji?.api, handsfree.model.pose?.api) } --------------------------------------------------------------------- Examples Face Tracking Examples Face Pointers Motion Parallax Display [6874747073] [6874747073] Puppeteering Industrial Robots Playing desktop games with face clicks [6874747073] [6874747073] --------------------------------------------------------------------- Hand Tracking Examples Hand Pointers Use with Three.js [6874747073] [6874747073] Playing desktop games with pinch Laser pointers but with your clicks finger [6874747073] [6874747073] --------------------------------------------------------------------- Pose Estimation Examples Flappy Pose - Flappy Bird but where you have to flap your arms [6874747073] --------------------------------------------------------------------- Local Development If you'd like to contribute to the library or documentation then the following will get you going: * Install NodeJS and git * Clone this repository: git clone https://github.com/handsfreejs/ handsfree * Install dependencies by running npm i in a terminal from the project's root * Start development on localhost:8080 by running npm start * Hit CTRL+C from the terminal to close the server Once you've run the above, you can just use npm start. If you pull the latest code, remember to run npm i to get any new dependencies (this shouldn't happen often). Command line scripts # Start local development on localhost:8080 npm start # Builds the library, documentation, and extension npm run build # Build only the library /dist/lib/ npm run build:lib # Build only the documentation at /dist/docs/ npm run build:docs # Build only the extension at /dist/extension npm run build:extension # Publish library to NPM npm login npm publish # Deploy documentation to handsfree.js.org deploy.sh Dev Notes * See vuepress-component-font-awesome for adding new icons to the documentation. Remember to run npm run fa:build when adding new font icons so that they are copied over into the docs/.vuepress/ components/FA folder * You may occasionally need to restart server when adding new files to the /docs, this is true when changing /docs /.vuepress.config.js as well --------------------------------------------------------------------- The Handsfree Browser Extension The Browser Extension is a designed to help you browse the web handsfree through face and/or hand gestures. The goal is to develop a "Userscript Manager" like Tampermonkey, but for handsfree-ifying web pages, games, apps, WebXR and really any other type of content found the web. How it works [6874747073] * When you first install the extension, /src/background/ handsfree.js checks if you've approved the webcam. If not, then it'll open the options page in src/options/stream-capture.html * The popup panel has a "Start/Stop Webcam" button that communicates with the background script to start the webcam: /src /popup/index.html * The background page is where the models are stored and run. This keeps everything isolated and only asks for webcam permission once (vs on every domain): /src/background/handsfree.js * The background page also uses the "Picture in Picture" API to "pop the webcam" out of the browser. It renders the webcam feed and debug canvases into a single canvas, and uses that as the srcObject to a separate video element which is the PiP'ed How to install Google Chrome Install as an unpacked chrome extension. 1. Visit chrome://extensions 2. Enable Developer Mode on the top right 3. Then click Load unpacked and select this project's root folder [6874747073] Handsfree Browsing By default, each page will get a "Face Pointer" or a set of "Palm Pointers" for you to browse pages handsfree. A person controlling a virtual mouse pointer by tilting their head around A person scrolling a page by pinching their index and thumb together and raising or lowering their pinched hand However, in addition to the pointers you can add custom handsfree interactions. For example, here's a demonstration of handsfree-ifying different things: Creating generative art with hand gestures A person pinching the air to "pinch" a blob. Moving a pinched blob causes it to sing in a different pitch --------------------------------------------------------------------- Explore the interactive docs at: Handsfree.js.org Or try it right away with the serverless boilerplates in /boilerplate /! --------------------------------------------------------------------- License & Attributions License: Apache 2.0 The Handsfree.js core is available for free and commercial use under Apache 2.0. Each of the models are also available for free and commercial use under Apache 2.0: * Jeeliz Weboji (Apache 2.0) * MediaPipe Models (Apache 2.0) * TensorFlow.js Models (Apache 2.0) Attributions I'd like to also thank the following people and projects: * 98.css by @jdan (MIT) - Used as boilerplate for documentation theme * handpose-facemesh-demos by [@LingDong-] - Used as a boilerplate for the Handpose Three.js setup --------------------------------------------------------------------- Special Thanks * @Golan and the The STUDIO for Creative Inquiry for hosting me for a residency during 2019 and for helping me approach projects in a more expressive way. Also for inviting me back for a multi-month residency in Spring 2021! * @AnilDash for supporting the project during Winter 2018 out of the blue and the opportunities to share my project on Glitch.com * The School of AI for the 2018 Fellowship in support of this project * @jessscon and Google PAIR for the very early support that made starting this project possible * Everyone who's previously supported the project through GitHub Sponsors, Patreon, GoFundMe, and through Twitter and everywhere else over the years About Quickly integrate face, hand, and/or pose tracking to your frontend projects in a snap handsfree.js.org Topics computer-vision weboji face-pointer Resources Readme License Apache-2.0 License Releases 38 Handsfree.js 8 - A complete rewrite around MediaPipe w/ better documentation Latest Dec 21, 2020 + 37 releases Packages 0 No packages published Used by 31 * @handsfreejs * @nathansbradshaw * @newbie2019-hub * @KoltonG * @MIDIBlocks * @Makeathon3077 * @UPstartDeveloper * @jlaframboise + 23 Contributors 5 * @MIDIBlocks * @labofoz * @theaspiringdev * @jjtolton * @dependabot Languages * JavaScript 92.7% * HTML 4.4% * CSS 2.8% * Shell 0.1% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.