https://arstechnica.com/gadgets/2022/05/the-full-saga-of-apples-troubled-mixed-reality-headset-has-been-revealed/ Skip to main content * Biz & IT * Tech * Science * Policy * Cars * Gaming & Culture * Store * Forums Subscribe [ ] Close Navigate * Store * Subscribe * Videos * Features * Reviews * RSS Feeds * Mobile Site * About Ars * Staff Directory * Contact Us * Advertise with Ars * Reprints Filter by topic * Biz & IT * Tech * Science * Policy * Cars * Gaming & Culture * Store * Forums Settings Front page layout Grid List Site theme Black on white White on black Sign in Comment activity Sign up or login to join the discussions! [ ] [ ] [Submit] [ ] Stay logged in | Having trouble? Sign up to comment and more Sign up Mixed reality -- The full saga of Apple's troubled mixed reality headset has been revealed It's a complicated tale of processors, base stations, politics, and Jony Ive. Samuel Axon - May 23, 2022 8:55 pm UTC A man in a tee-shirt sits onstage. Enlarge / Jony Ive speaks onstage during the 2017 New Yorker TechFest in New York City. Brian Ach/Getty Images reader comments 191 with 130 posters participating Share this story * Share on Facebook * Share on Twitter * Share on Reddit A series of reports in The Information paint a detailed picture of progression, politics, and problems facing Apple's plan to develop a virtual, augmented, or mixed reality headset since the initiative picked up steam back in 2015. Citing several people familiar with the product, including some who worked on it directly, the reports describe a contest of wills over the direction of the device. The standoff was between Apple's mixed reality product team (called the "Technology Development Group") and famed Apple designer Jony Ive and his industrial design team. The report sheds light on Apple's direction for the device, which Bloomberg recently reported is nearing launch. Further Reading Prominent Apple staffers write letters to management, resign over office return They also claim that Apple CEO Tim Cook has been relatively hands-off from the product compared to others like the iPhone, and that the Technology Development Group's location in a separate office from the main Apple headquarters has been a source of problems and frustration. The Information's sources say that Apple's mixed reality efforts began almost accidentally when the company purchased a German AR startup called Metaio to use some of its technology on Project Titan, its self-driving car project. Another key moment was when Apple hired the AR/VR project team leader, Mike Rockwell, away from Dolby Laboratories. Starting in 2015, Rockwell built a team that included Metaio co-founder Peter Meier and Apple Watch manager, Fletcher Rothkopf. In 2016, several AR demos were shown to members of Apple's board. In one, a tiny triceratops grew to lifesize before the board members' eyes. In another, a room was transformed into an immersive, leafy environment. But the board was not Rockwell and the company's most significant barrier. According to The Information, it was Ive, who oversaw both the industrial design and human interface teams at Apple. Advertisement Ive and his crew argued against a VR headset because they believe VR separates users from the people and world around them, and that VR headsets look unfashionable. But the Technology Development Group gained the industrial design team's support by presenting a concept: an outward-facing screen on the front of the headset that showed images of the wearer's facial expressions and eyes to people around them. The wearer could see the people around them through an external camera feed. Rockwell and his colleagues developed and released ARKit in 2017, an application development suite that allowed developers to make AR apps for iPhone and iPad using technologies and techniques that could be adapted to a headset later. Further Reading Amazon is hiring to build an "advanced" and "magical" AR/VR product At first, Rockwell and the rest of the mixed reality team wanted the headset to be tethered to a base station to provide maximally impressive graphics and performance, and some on the team envisioned it as primarily a tool for professionals and creatives to use at their desks. But Ive didn't like either of those ideas and wanted it to be a mass-market lifestyle product that consumers could take on the go. Apple's most senior leadership backed Ive's plan, and Ive still has an active role in the headset's development, even though he now works with Apple as a consultant. The decision to make the headset a standalone device reportedly caused significant headaches. For example, some felt the most optimal way to make it work well on its own would have been to put more functions on a single chip. But since the silicon work was already done, they had to find ways to battle the latency inherent in having multiple chips in the device communicate with each other. They had also been making software assuming that the base station plan would go ahead. Advertisement Nonetheless, the device is moving into the final phases of development. Bloomberg claimed last week that an advanced version of the product was recently shown to Apple's board, and that Apple has "ramped up" development of the headset's software, which is an iOS offshoot called rOS. (The R stands for "reality.") The Information's reporting reveals many details about the upcoming headset. It would feature a resolution of at least 4K for each eye, which the team believed was the bare minimum for users not to perceive the image as pixelated, in contrast to most current consumer VR headsets. Its built-in processor would be closely related to the M2 processor expected to reach Macs and iPads in the coming months. Further Reading Apple's AR/VR headset isn't just a prototype anymore, sources say The headset would also have 14 cameras, some facing outward and some facing inward. It would allow users to see the outside world and enable nearby people to see a video representation of the user's eyes. It would live track the wearer's face and body movements to map to a 3D avatar (likely similar to the iPhone's Memoji) that could be used to have remote meetings and social gatherings with other headset wearers who are far away. Because of the limited processing capabilities of the M2 chip in a headset without a tethered base station (the canceled base station was said to have ultra-high-performance M1 Ultra), the avatars would be cartoony. The Information's sources also say that more photo-realistic avatars were attempted when the base station was a part of the plan, but the uncanny valley was an issue. Apple initially planned to launch the headset in 2019, but it now looks like it could instead be announced either later this year or in 2023. Additionally, Apple plans to introduce more natural-looking AR glasses as a follow-up product, but that device may still be years away from shipping. reader comments 191 with 130 posters participating Share this story * Share on Facebook * Share on Twitter * Share on Reddit Samuel Axon Based in Chicago, Samuel is the Senior Reviews Editor at Ars Technica, where he leads the site's gadget and reviews section. He covers Apple and display tech, as well as hardware and software for developers and creative professionals. He has been writing about technology for 15 years, and is an AR and game developer for iOS and other platforms. Email samuel.axon@arstechnica.com Advertisement You must login or create an account to comment. Channel Ars Technica - Previous story Next story - Related Stories Today on Ars * Store * Subscribe * About Us * RSS Feeds * View Mobile Site * Contact Us * Staff * Advertise with us * Reprints Newsletter Signup Join the Ars Orbital Transmission mailing list to get weekly updates delivered to your inbox. Sign me up - CNMN Collection WIRED Media Group (c) 2022 Conde Nast. All rights reserved. Use of and/or registration on any portion of this site constitutes acceptance of our User Agreement (updated 1/1/20) and Privacy Policy and Cookie Statement (updated 1/1 /20) and Ars Technica Addendum (effective 8/21/2018). Ars may earn compensation on sales from links on this site. Read our affiliate link policy. Your California Privacy Rights | Do Not Sell My Personal Information The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of Conde Nast. Ad Choices