https://www.wired.com/story/apple-vision-pro-persona-eye-tracking-spy-typing/ Skip to main content Open Navigation Menu To revisit this article, visit My Profile, then View saved stories. Close Alert WIRED Apple Vision Pro's Eye Tracking Exposed What People Type * Security * Politics * Gear * The Big Story * Business * Science * Culture * Ideas * Merch More Search * Security * Politics * Gear * The Big Story * Business * Science * Culture * Ideas * Merch * Podcasts * Video * Newsletters * Magazine * Travel * Steven Levy's Plaintext Column * WIRED Classics from the Archive * Events * WIRED Insider * WIRED Consulting * Jobs * Coupons Matt Burgess Security Sep 12, 2024 6:00 AM Apple Vision Pro's Eye Tracking Exposed What People Type The Vision Pro uses 3D avatars on calls and for streaming. These researchers used eye tracking to work out the passwords and PINs people typed with their avatars. A man wearing a VR headset Photo-Illustration: Wired Staff; Getty Save Save You can tell a lot about someone from their eyes. They can indicate how tired you are, the type of mood you're in, and potentially provide clues about health problems. But your eyes could also leak more secretive information: your passwords, PINs, and messages you type. Today, a group of six computer scientists are revealing a new attack against Apple's Vision Pro mixed reality headset where exposed eye-tracking data allowed them to decipher what people entered on the device's virtual keyboard. The attack, dubbed GAZEploit and shared exclusively with WIRED, allowed the researchers to successfully reconstruct passwords, PINs, and messages people typed with their eyes. "Based on the direction of the eye movement, the hacker can determine which key the victim is now typing," says Hanqiu Wang, one of the leading researchers involved in the work. They identified the correct letters people typed in passwords 77 percent of the time within five guesses and 92 percent of the time in messages. To be clear, the researchers did not gain access to Apple's headset to see what they were viewing. Instead, they worked out what people were typing by remotely analyzing the eye movements of a virtual avatar created by the Vision Pro. This avatar can be used in Zoom calls, Teams, Slack, Reddit, Tinder, Twitter, Skype, and FaceTime. The researchers alerted Apple to the vulnerability in April, and the company issued a patch to stop the potential for data to leak at the end of July. It is the first attack to exploit people's "gaze" data in this way, the researchers say. The findings underline how people's biometric data--information and measurements about your body--can expose sensitive information and be used as part of the burgeoning surveillance industry. Eye Spy Your eyes are your mouse when using the Vision Pro. When typing, you look at a virtual keyboard that hovers around, and can be moved and resized. When you're looking at the right letter, tapping two fingers together works as a click. What you do stays within the headset, but if you want to jump on a quick Zoom, FaceTime some friends, or livestream, you'll likely end up using a Persona--the sort of ghostly 3D avatar the Vision Pro creates by scanning your face. "These technologies ... can inadvertently expose critical facial biometrics, including eye-tracking data, through video calls where the user's virtual avatar mirrors their eye movements," the researchers write in a preprint paper detailing their findings. Wang says the work relies on two biometrics that can be extracted from recordings of a Persona: the eye aspect ratio (EAR) and eye gaze estimation. (As well as Wang, the research was completed by Siqi Dai, Max Panoff, and Shuo Wang from the University of Florida, Haoqi Shan from blockchain security company CertiK, and Zihao Zhan from Texas Tech University.) Most Popular * The Best Hearing Aids We've Personally Tested and Vetted With an Expert Gear The Best Hearing Aids We've Personally Tested and Vetted With an Expert By Christopher Null * The Best Wireless Earbuds for Everyone Gear The Best Wireless Earbuds for Everyone By Parker Hall * How Do You Solve a Problem Like Polestar? Gear How Do You Solve a Problem Like Polestar? By Carlton Reid * The 21 Best Movies on Amazon Prime Right Now Culture The 21 Best Movies on Amazon Prime Right Now By Matt Kamen * The GAZEploit attack consists of two parts, says Zhan, one of the lead researchers. First, the researchers created a way to identify when someone wearing the Vision Pro is typing by analyzing the 3D avatar they are sharing. For this, they trained a recurrent neural network, a type of deep learning model, with recordings of 30 people's avatars while they completed a variety of typing tasks. When someone is typing using the Vision Pro, their gaze fixates on the key they are likely to press, the researchers say, before quickly moving to the next key. "When we are typing our gaze will show some regular patterns," Zhan says. Wang says these patterns are more common during typing than if someone is browsing a website or watching a video while wearing the headset. "During tasks like gaze typing, the frequency of your eye blinking decreases because you are more focused," Wang says. In short: Looking at a QWERTY keyboard and moving between the letters is a pretty distinct behavior. The second part of the research, Zhan explains, uses geometric calculations to work out where someone has positioned the keyboard and the size they've made it. "The only requirement is that as long as we get enough gaze information that can accurately recover the keyboard, then all following keystrokes can be detected." Combining these two elements, they were able to predict the keys someone was likely to be typing. In a series of lab tests, they didn't have any knowledge of the victim's typing habits, speed, or know where the keyboard was placed. However, the researchers could predict the correct letters typed, in a maximum of five guesses, with 92.1 percent accuracy in messages, 77 percent of the time for passwords, 73 percent of the time for PINs, and 86.1 percent of occasions for emails, URLs, and webpages. (On the first guess, the letters would be right between 35 and 59 percent of the time, depending on what kind of information they were trying to work out.) Duplicate letters and typos add extra challenges. "It's very powerful to know where someone is looking," says Alexandra Papoutsaki, an associate professor of computer science at Pomona College who has studied eye tracking for years and reviewed the GAZEploit research for WIRED. Papoutsaki says the work stands out as it only relies on the video feed of someone's Persona, making it a more "realistic" space for an attack to happen when compared to a hacker getting hands-on with someone's headset and trying to access eye tracking data. "The fact that now someone, just by streaming their Persona, could expose potentially what they're doing is where the vulnerability becomes a lot more critical," Papoutsaki says. While the attack was created in lab settings and hasn't been used against anyone using Personas in the real world, the researchers say there are ways hackers could have abused the data leakage. They say, theoretically at least, a criminal could share a file with a victim during a Zoom call, resulting in them logging into, say, a Google or Microsoft account. The attacker could then record the Persona while their target logs in and use the attack method to recover their password and access their account. Quick Fixes The GAZEploit researchers reported their findings to Apple in April and subsequently sent the company their proof-of-concept code so the attack could be replicated. Apple fixed the flaw in a Vision Pro software update at the end of July, which stops the sharing of a Persona if someone is using the virtual keyboard. An Apple spokesperson confirmed the company fixed the vulnerability, saying it was addressed in VisionOS 1.3. The company's software update notes do not mention the fix, but it is detailed in the company's security-specific note. The researchers say Apple assigned CVE-2024-40865 for the vulnerability and recommend people download the latest software updates. Most Popular * The Best Hearing Aids We've Personally Tested and Vetted With an Expert Gear The Best Hearing Aids We've Personally Tested and Vetted With an Expert By Christopher Null * The Best Wireless Earbuds for Everyone Gear The Best Wireless Earbuds for Everyone By Parker Hall * How Do You Solve a Problem Like Polestar? Gear How Do You Solve a Problem Like Polestar? By Carlton Reid * The 21 Best Movies on Amazon Prime Right Now Culture The 21 Best Movies on Amazon Prime Right Now By Matt Kamen * The research highlights how people's personal data can be inadvertently leaked or exposed. In recent years, police have extracted fingerprints from photographs posted online and identified people by the way they walk in CCTV footage. Law enforcement have also started testing Vision Pros as part of their surveillance efforts. These privacy and surveillance concerns are likely to become more pressing as wearable technology becomes smaller, cheaper, and able to capture more information about people. "As wearables like glasses, XR, and smartwatches become more integrated into everyday life, users often overlook how much information these devices can collect about their activities and intentions, and the associated privacy risks," says Cheng Zhang, an assistant professor at Cornell University who also reviewed the Vision Pro research at WIRED's request. (Zhang's work has involved creating wearables to help interpret human behaviors.) "This paper clearly demonstrates one specific risk with gaze typing, but it's just the tip of the iceberg," Zhang says. "While these technologies are developed for positive purposes and applications, we also need to be aware of the privacy implications and start taking measures to mitigate potential risks for the future generation of everyday wearables." Update 2:30 pm ET, September 12, 2024: Following publication, Apple directed WIRED to a security note where the Vision Pro fix is mentioned. We've updated the story to include this note. You Might Also Like ... * In your inbox: Our biggest stories, handpicked for you each day * How one bad CrowdStrike update crashed the world's computers * The Big Story: How soon might the Atlantic Ocean break? * Welcome to the internet's hyper-consumption era [undefined] Matt Burgess is a senior writer at WIRED focused on information security, privacy, and data regulation in Europe. He graduated from the University of Sheffield with a degree in journalism and now lives in London. Send tips to Matt_Burgess@wired.com. Senior writer * TopicsapplesecurityprivacysurveillancevulnerabilitiesVision Pro Read More Hackers Threaten to Leak Planned Parenthood Data Hackers Threaten to Leak Planned Parenthood Data Plus: Kaspersky's US business sold, Nigerian sextortion scammers jailed, and Europe's controversial encryption plans return. Matt Burgess A Single Iranian Hacker Group Targeted Both Presidential Campaigns, Google Says A Single Iranian Hacker Group Targeted Both Presidential Campaigns, Google Says APT42, which is believed to work for Iran's Revolutionary Guard Corps, targeted about a dozen people associated with both Trump's and Biden's campaigns this spring, according to Google's Threat Analysis Group. Andy Greenberg The Slow-Burn Nightmare of the National Public Data Breach The Slow-Burn Nightmare of the National Public Data Breach Social Security numbers, physical addresses, and more--all available online. After months of confusion, leaked information from a background-check firm underscores the long-term risks of data breaches. Lily Hay Newman Powerful Spyware Exploits Enable a New String of 'Watering Hole' Attacks Powerful Spyware Exploits Enable a New String of 'Watering Hole' Attacks Suspected Russian hackers have compromised a series of websites to utilize sophisticated spyware exploits that are eerily similar to those created by NSO Group and Intellexa. Lily Hay Newman The US Government Wants You&-Yes, You&-to Hunt Down Generative AI Flaws The US Government Wants You--Yes, You--to Hunt Down Generative AI Flaws The AI ethics nonprofit Humane Intelligence and the US National Institute of Standards and Technology are launching a series of contests to get more people probing for problems in generative AI systems. Lily Hay Newman Taylor Swift Concert Terror Plot Was Thwarted by Key CIA Tip Taylor Swift Concert Terror Plot Was Thwarted by Key CIA Tip Plus: China-linked hackers infiltrate US internet providers, authorities crack down on a major piracy operation, and a ransomware gang claims attacks during the Paris Olympics. Lily Hay Newman The US Navy Is Going All In on Starlink The US Navy Is Going All In on Starlink The Navy is testing out the Elon Musk-owned satellite constellation to provide high-speed internet access to sailors at sea. It's part of a bigger project that's about more than just getting online. Jared Keller Why It's So Hard to Fully Block X in Brazil Why It's So Hard to Fully Block X in Brazil With 20,000 internet providers across the country, the technical challenges of blocking X in Brazil mean some connections are slipping through the cracks. Lily Hay Newman WIRED WIRED is where tomorrow is realized. It is the essential source of information and ideas that make sense of a world in constant transformation. The WIRED conversation illuminates how technology is changing every aspect of our lives--from culture to business, science to design. The breakthroughs and innovations that we uncover lead to new ways of thinking, new connections, and new industries. More From WIRED * Subscribe * Newsletters * FAQ * WIRED Staff * Editorial Standards * Archive * RSS * Accessibility Help Reviews and Guides * Reviews * Buying Guides * Mattresses * Electric Bikes * Soundbars * Streaming Guides * Coupons * Submit an Offer * Become a Partner * Coupons Contact * Code Guarantee * Advertise * Contact Us * Customer Care * Jobs * Press Center * Conde Nast Store * User Agreement * Privacy Policy * Your California Privacy Rights (c) 2024 Conde Nast. All rights reserved. WIRED may earn a portion of sales from products that are purchased through our site as part of our Affiliate Partnerships with retailers. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of Conde Nast. Ad Choices Select international site United States * Italia * Japon * Czech Republic & Slovakia * * * * * *