https://github.com/Jcparkyn/dpoint Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} Jcparkyn / dpoint Public * Notifications * Fork 1 * Star 63 Open-source digital stylus using camera tracking and inertial measurements License MIT license 63 stars 1 fork Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights Jcparkyn/dpoint This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 2 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/J] Use Git or checkout with SVN using the web URL. [gh repo clone Jcpark] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @Jcparkyn Jcparkyn Fix typo in filename ... 9ae7f9e Nov 11, 2023 Fix typo in filename 9ae7f9e Git stats * 148 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time assets Update readme October 9, 2023 21:26 electronics Add KiCad files October 9, 2023 14:14 markers Add PDF version of stylus-markers November 11, 2023 11:16 microcontroller/dpoint-arduino Improve tap interrupt parameters October 11, 2023 16:35 print Fix typo in filename November 11, 2023 12:26 python Update board details for calibrate_camera November 11, 2023 11:16 .gitattributes Remove ipynb noise from linguist stats October 9, 2023 21:37 .gitignore Add python project May 13, 2023 14:28 LICENSE Add license October 9, 2023 19:21 README.md Add setup guide November 11, 2023 11:44 setup-guide.md Add images to setup guide November 11, 2023 12:01 View code D-POINT: Digital Pen with Optical-Inertial Tracking Design Hardware Visual pose estimation (VPE) Inertial fusion README.md D-POINT: Digital Pen with Optical-Inertial Tracking D-POINT is an open-source digital stylus that uses camera tracking and inertial measurements to achieve 6DoF (six degrees of freedom) inputs, with low latency, pressure sensitivity, and sub-millimetre accuracy. The stylus can be used on any flat surface, and works with consumer-grade webcams. writing-demo.mp4 This project was part of my undergraduate thesis for electrical engineering. I've open-sourced the code and design files in the hopes that they might be useful to somebody, but it's not intended to be a "plug and play" DIY project. If you want to try building it anyway, follow the setup guide. Design This is a very brief overview of how the system works. For all the details, plus literature review and lots of evaluation, read the full thesis (note: I haven't published this online yet). Block diagram showing how the system works Hardware The main body of the stylus was 3D printed as two halves, shown below. The stylus contains a force sensor, a Li-ion battery which charges over USB-C, and an Arduino-based development board for logic and Bluetooth. Eight printed ArUco markers are glued to the back of the stylus, for visual pose estimation. CAD drawing showing the hardware design of the stylus Visual pose estimation (VPE) The VPE process involves the four main steps: 1. Marker detection: First, we use OpenCV to detect the corners of each visible ArUco marker on the stylus. 2. Rolling shutter correction: We use a simple 2D motion model to estimate and correct for the effects of rolling shutter on the observed corner locations. 3. Perspective-n-Point (PnP): From these corner positions, we use a PnP algorithm to estimate the pose of the stylus relative to the camera. When possible, we use the pose from the previous frame as a starting point to refine with virtual visual servoing (VVS), otherwise we fall back to SQPnP. 4. Coordinate conversion: Using the calibrated pose of the stylus and the drawing surface relative to the camera, we calculate the position and orientation of the stylus tip relative to the drawing surface. Inertial fusion We use an Extended Kalman Filter (EKF) to fuse the VPE estimates with the inertial data from the accelerometer and gyroscope, and refine the estimates in real-time using the Rauch-Tung-Striebel (RTS) algorithm. To account for time delay from the camera frames, we use a negative-time measurement update algorithm. The EKF is implemented using NumPy and Numba. Using inertial measurements allows us to dramatically reduce latency compared to a camera-only implementation, while also improving accuracy and report rate for fast movements. About Open-source digital stylus using camera tracking and inertial measurements Topics stylus aruco ekf 6dof Resources Readme License MIT license Activity Stars 63 stars Watchers 1 watching Forks 1 fork Report repository Languages * Python 89.7% * C++ 10.3% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.