https://spectrum.ieee.org/history-of-augmented-reality [ ] IEEE.orgIEEE Xplore Digital LibraryIEEE StandardsMore Sites Sign InJoin IEEE The August 2022 issue of IEEE Spectrum is here! Download PDF | Close bar How a Parachute Accident Helped Jump-start Augmented Reality Share FOR THE TECHNOLOGY INSIDER Search: [ ] Explore by topic AerospaceArtificial IntelligenceBiomedicalComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensors TelecommunicationsTransportation IEEE Spectrum FOR THE TECHNOLOGY INSIDER Topics AerospaceArtificial IntelligenceBiomedicalComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensors TelecommunicationsTransportation Sections FeaturesNewsOpinionCareersDIYThe Big PictureEngineering Resources More Special ReportsCollectionsExplainersPodcastsVideosNewslettersTop Programming LanguagesRobots Guide For IEEE Members Current IssueMagazine ArchiveThe InstituteTI Archive For IEEE Members Current IssueMagazine ArchiveThe InstituteTI Archive IEEE Spectrum About UsContact UsReprints & PermissionsAdvertising Follow IEEE Spectrum Support IEEE Spectrum IEEE Spectrum is the flagship publication of the IEEE -- the world's largest professional organization devoted to engineering and applied sciences. Our articles, podcasts, and infographics inform our readers about developments in technology, engineering, and science. Join IEEE Subscribe About IEEEContact & SupportAccessibilityNondiscrimination PolicyTerms IEEE Privacy Policy (c) Copyright 2022 IEEE -- All rights reserved. A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. IEEE websites place cookies on your device to give you the best user experience. By using our websites, you agree to the placement of these cookies. To learn more, read our Privacy Policy. view privacy policy accept & close Enjoy more free content and benefits by creating an account Saving articles to read later requires an IEEE Spectrum account The Institute content is only available for members Downloading full PDF issues is exclusive for IEEE Members Access to Spectrum's Digital Edition is exclusive for IEEE Members Following topics is a feature exclusive for IEEE Members Adding your response to an article requires an IEEE Spectrum account Create an account to access more content and features on IEEE Spectrum, including the ability to save articles to read later, download Spectrum Collections, and participate in conversations with readers and editors. For more exclusive content and features, consider Joining IEEE. Join the world's largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum's articles, archives, PDF downloads, and other benefits. Learn more - CREATE AN ACCOUNTSIGN IN JOIN IEEESIGN IN Close Access Thousands of Articles -- Completely Free Create an account and get exclusive content and features: Save articles, download collections, and talk to tech insiders -- all free! For full access and benefits, join IEEE as a paying member. CREATE AN ACCOUNTSIGN IN TopicMagazineHistory of TechnologyTypeFeature How a Parachute Accident Helped Jump-start Augmented Reality In 1992, hardware for the first interactive AR system literally fell from the skies Louis Rosenberg 07 Apr 2022 11 min read Man wearing goggles and earphones, holding wired controllers in each hand. Louis Rosenberg tests Virtual Fixtures, the first interactive augmented-reality system that he developed at Wright-Patterson Air Force Base, in 1992. Louis Rosenberg I climb into an upper-body exoskeleton that's covered in sensors, motors, gears, and bearings, and then lean forward, tilting my head up to press my face against the eyepieces of a vision system hanging from the ceiling. In front of me, I see a large wooden board, painted black and punctuated by a grid of metal holes. The board is real. So is the peg in my hand that I'm trying to move from one hole to another, as fast as I can. When I begin to move the peg, a virtual cone appears over the target hole, along with a virtual surface easing toward it. I can feel the surface as I slide the peg along it toward the cone and into the hole. This was the Virtual Fixtures platform, which was developed in the early 1990s to test the potential of "perceptual overlays" to improve human performance in manual tasks that require dexterity. And it worked. These days, virtual-reality experts look back on the platform as the first interactive augmented-reality system that enabled users to engage simultaneously with real and virtual objects in a single immersive reality. The project began in 1991, when I pitched the effort as part of my doctoral research at Stanford University. By the time I finished--three years and multiple prototypes later--the system I had assembled filled half a room and used nearly a million dollars' worth of hardware. And I had collected enough data from human testing to definitively show that augmenting a real workspace with virtual objects could significantly enhance user performance in precision tasks. Given the short time frame, it might sound like all went smoothly, but the project came close to getting derailed many times, thanks to a tight budget and substantial equipment needs. In fact, the effort might have crashed early on, had a parachute--a real one, not a virtual one--not failed to open in the clear blue skies over Dayton, Ohio, during the summer of 1992. Before I explain how a parachute accident helped drive the development of augmented reality, I'll lay out a little of the historical context. Thirty years ago, the field of virtual reality was in its infancy, the phrase itself having only been coined in 1987 by Jaron Lanier, who was commercializing some of the first headsets and gloves. His work built on earlier research by Ivan Sutherland, who pioneered head-mounted display technology and head-tracking, two critical elements that sparked the VR field. Augmented reality (AR)--that is, combining the real world and the virtual world into a single immersive and interactive reality--did not yet exist in a meaningful way. Back then, I was a graduate student at Stanford University and a part-time researcher at NASA's Ames Research Center, interested in the creation of virtual worlds. At Stanford, I worked in the Center for Design Research, a group focused on the intersection of humans and technology that created some of the very early VR gloves, immersive vision systems, and 3D audio systems. At NASA, I worked in the Advanced Displays and Spatial Perception Laboratory of the Ames Research Center, where researchers were exploring the fundamental parameters required to enable realistic and immersive simulated worlds. Of course, knowing how to create a quality VR experience and being able to produce it are not the same thing. The best PCs on the market back then used Intel 486 processors running at 33 megahertz. Adjusted for inflation, they cost about US $8,000 and weren't even a thousandth as fast as a cheap gaming computer today. The other option was to invest $60,000 in a Silicon Graphics workstation--still less than a hundredth as fast as a mediocre PC today. So, though researchers working in VR during the late 80s and early 90s were doing groundbreaking work, the crude graphics, bulky headsets, and lag so bad it made people dizzy or nauseous plagued the resulting virtual experiences. Nine diagrams, in a grid, each showing a simple pegboard with four holes, with a variety of transparent representations of planes and other guide surfaces in front of it. These early drawings of a real pegboard combined with virtual overlays generated by a computer--an early version of augmented reality--were created by Louis Rosenberg as part of his Virtual Fixtures project.Louis Rosenberg I was conducting a research project at NASA to optimize depth perception in early 3D-vision systems, and I was one of those people getting dizzy from the lag. And I found that the images created back then were definitely virtual but far from reality. Still, I wasn't discouraged by the dizziness or the low fidelity, because I was sure the hardware would steadily improve. Instead, I was concerned about how enclosed and isolated the VR experience made me feel. I wished I could expand the technology, taking the power of VR and unleashing it into the real world. I dreamed of creating a merged reality where virtual objects inhabited your physical surroundings in such an authentic manner that they seemed like genuine parts of the world around you, enabling you to reach out and interact as if they were actually there. I was aware of one very basic sort of merged reality--the head-up display-- in use by military pilots, enabling flight data to appear in their lines of sight so they didn't have to look down at cockpit gauges. I hadn't experienced such a display myself, but became familiar with them thanks to a few blockbuster 1980s hit movies, including Top Gun and Terminator. In Top Gun a glowing crosshair appeared on a glass panel in front of the pilot during dogfights; in Terminator, crosshairs joined text and numerical data as part of the fictional cyborg's view of the world around it. Neither of these merged realities were the slightest bit immersive, presenting images on a flat plane rather than connected to the real world in 3D space. But they hinted at interesting possibilities. I thought I could move far beyond simple crosshairs and text on a flat plane to create virtual objects that could be spatially registered to real objects in an ordinary environment. And I hoped to instill those virtual objects with realistic physical properties. A square board with eight numbered holes. Two mesh cones appear in front of holes 2 and 6The Fitts's Law peg-insertion task involves having test subjects quickly move metal pegs between holes. The board shown here was real, the cones that helped guide the user to the correct holes virtual.Louis Rosenberg I needed substantial resources--beyond what I had access to at Stanford and NASA--to pursue this vision. So I pitched the concept to the Human Sensory Feedback Group of the U.S. Air Force's Armstrong Laboratory, now part of the Air Force Research Laboratory. To explain the practical value of merging real and virtual worlds, I used the analogy of a simple metal ruler. If you want to draw a straight line in the real world, you can do it freehand, going slow and using significant mental effort, and it still won't be particularly straight. Or you can grab a ruler and do it much quicker with far less mental effort. Now imagine that instead of a real ruler, you could grab a virtual ruler and make it instantly appear in the real world, perfectly registered to your real surroundings. And imagine that this virtual ruler feels physically authentic--so much so that you can use it to guide your real pencil. Because it's virtual, it can be any shape and size, with interesting and useful properties that you could never achieve with a metal straightedge. Of course, the ruler was just an analogy. The applications I pitched to the Air Force ranged from augmented manufacturing to surgery. For example, consider a surgeon who needs to make a dangerous incision. She could use a bulky metal fixture to steady her hand and avoid vital organs. Or we could invent something new to augment the surgery--a virtual fixture to guide her real scalpel, not just visually but physically. Because it's virtual, such a fixture would pass right through the patient's body, sinking into tissue before a single cut had been made. That was the concept that got the military excited, and their interest wasn't just for in-person tasks like surgery but for distant tasks performed using remotely controlled robots. For example, a technician on Earth could repair a satellite by controlling a robot remotely, assisted by virtual fixtures added to video images of the real worksite. The Air Force agreed to provide enough funding to cover my expenses at Stanford along with a small budget for equipment. Perhaps more significantly, I also got access to computers and other equipment at Wright-Patterson Air Force Base near Dayton, Ohio. And what became known as the Virtual Fixtures Project came to life, working toward building a prototype that could be rigorously tested with human subjects. And I became a roving researcher, developing core ideas at Stanford, fleshing out some of the underlying technologies at NASA Ames, and assembling the full system at Wright-Patterson. A sketch of a person wearing earphones and googles gazing at two green cones and a grey pegboard with four holes.In this sketch of his augmented-reality system, Louis Rosenberg shows a user of the Virtual Fixtures platform wearing a partial exoskeleton and peering at a real pegboard augmented with cone-shaped virtual fixtures.Louis Rosenberg Now about those parachutes. As a young researcher in my early twenties, I was eager to learn about the many projects going on around me at these various laboratories. One effort I followed closely at Wright-Patterson was a project designing new parachutes. As you might expect, when the research team came up with a new design, they didn't just strap a person in and test it. Instead, they attached the parachutes to dummy rigs fitted with sensors and instrumentation. Two engineers would go up in an airplane with the hardware, dropping rigs and jumping alongside so they could observe how the chutes unfolded. Stick with my story and you'll see how this became key to the development of that early AR system. Back at the Virtual Fixtures effort, I aimed to prove the basic concept--that a real workspace could be augmented with virtual objects that feel so real, they could assist users as they performed dexterous manual tasks. To test the idea, I wasn't going to have users perform surgery or repair satellites. Instead, I needed a simple repeatable task to quantify manual performance. The Air Force already had a standardized task it had used for years to test human dexterity under a variety of mental and physical stresses. It's called the Fitts's Law peg-insertion task, and it involves having test subjects quickly move metal pegs between holes on a large pegboard. So I began assembling a system that would enable virtual fixtures to be merged with a real pegboard, creating a mixed-reality experience perfectly registered in 3D space. I aimed to make these virtual objects feel so real that bumping the real peg into a virtual fixture would feel as authentic as bumping into the actual board. I wrote software to simulate a wide range of virtual fixtures, from simple surfaces that prevented your hand from overshooting a target hole, to carefully shaped cones that could help a user guide the real peg into the real hole. I created virtual overlays that simulated textures and had corresponding sounds, even overlays that simulated pushing through a thick liquid as it it were virtual honey. A drawing of a person lying flat on a white surface, with green and white panels crisscrossing the body.One imagined use for augmented reality at the time of its creation was in surgery. Today, augmented reality is used for surgical training, and surgeons are beginning to use it in the operating room.Louis Rosenberg For more realism, I modeled the physics of each virtual element, registering its location accurately in three dimensions so it lined up with the user's perception of the real wooden board. Then, when the user moved a hand into an area corresponding to a virtual surface, motors in the exoskeleton would physically push back, an interface technology now commonly called "haptics." It indeed felt so authentic that you could slide along the edge of a virtual surface the way you might move a pencil against a real ruler. To accurately align these virtual elements with the real pegboard, I needed high-quality video cameras. Video cameras at the time were far more expensive than they are today, and I had no money left in my budget to buy them. This was a frustrating barrier: The Air Force had given me access to a wide range of amazing hardware, but when it came to simple cameras, they couldn't help. It seemed like every research project needed them, most of far higher priority than mine. Which brings me back to the skydiving engineers testing experimental parachutes. These engineers came into the lab one day to chat; they mentioned that their chute had failed to open, their dummy rig plummeting to the ground and destroying all the sensors and cameras aboard. This seemed like it would be a setback for my project as well, because I knew if there were any extra cameras in the building, the engineers would get them. But then I asked if I could take a look at the wreckage from their failed test. It was a mangled mess of bent metal, dangling circuits, and smashed cameras. Still, though the cameras looked awful with cracked cases and damaged lenses, I wondered if I could get any of them to work well enough for my needs. By some miracle, I was able to piece together two working units from the six that had plummeted to the ground. And so, the first human testing of an interactive augmented-reality system was made possible by cameras that had literally fallen out of the sky and smashed into the earth. To appreciate how important these cameras were to the system, think of a simple AR application today, like Pokemon Go. If you didn't have a camera on the back of your phone to capture and display the real world in real time, it wouldn't be an augmented-reality experience; it would just be a standard video game. The same was true for the Virtual Fixtures system. But thanks to the cameras from that failed parachute rig, I was able to create a mixed reality with accurate spatial registration, providing an immersive experience in which you could reach out and interact with the real and virtual environments simultaneously. As for the experimental part of the project, I conducted a series of human studies in which users experienced a variety of virtual fixtures overlaid onto their perception of the real task board. The most useful fixtures turned out to be cones and surfaces that could guide the user's hand as they aimed the peg toward a hole. The most effective involved physical experiences that couldn't be easily manufactured in the real world but were readily achievable virtually. For example, I coded virtual surfaces that were "magnetically attractive" to the peg. For the users, it felt as if the peg had snapped to the surface. Then they could glide along it until they chose to yank free with another snap. Such fixtures increased speed and dexterity in the trials by more than 100 percent. Of the various applications for Virtual Fixtures that we considered at the time, the most commercially viable back then involved manually controlling robots in remote or dangerous environments--for example, during hazardous waste clean-up. If the communications distance introduced a time delay in the telerobotic control, virtual fixtures became even more valuable for enhancing human dexterity. Today, researchers are still exploring the use of virtual fixtures for telerobotic applications with great success, including for use in satellite repair and robot-assisted surgery. An employee badge with a photo of a man and logo of NASA. Text on badge reads Ames Research Center Louis Rosenberg spent some of his time working in the Advanced Displays and Spatial Perception Laboratory of the Ames Research Center as part of his research in augmented reality.Louis Rosenberg I went in a different direction, pushing for more mainstream applications for augmented reality. That's because the part of the Virtual Fixtures project that had the greatest impact on me personally wasn't the improved performance in the peg-insertion task. Instead, it was the big smiles that lit up the faces of the human subjects when they climbed out of the system and effused about what a remarkable experience they had had. Many told me, without prompting, that this type of technology would one day be everywhere. And indeed, I agreed with them. I was convinced we'd see this type of immersive technology go mainstream by the end of the 1990s. In fact, I was so inspired by the enthusiastic reactions people had when they tried those early prototypes, I founded a company in 1993--Immersion --with the goal of pursuing mainstream consumer applications. Of course, it hasn't happened nearly that fast. At the risk of being wrong again, I sincerely believe that virtual and augmented reality, now commonly referred to as the metaverse, will become an important part of most people's lives by the end of the 2020s. In fact, based on the recent surge of investment by major corporations into improving the technology, I predict that by the early 2030s augmented reality will replace the mobile phone as our primary interface to digital content. And no, none of the test subjects who experienced that early glimpse of augmented reality 30 years ago knew they were using hardware that had fallen out of an airplane. But they did know that they were among the first to reach out and touch our augmented future. From Your Site Articles * Augmented Reality's Moats Will Be Made of Money - IEEE Spectrum > * This Is the Year for Apple's AR Glasses--Maybe - IEEE Spectrum > * Looking Through Mojo Vision's Newest AR Contact Lens - IEEE ... > * Apple's Metaverse Snub Highlights AR/VR Tech Woes - IEEE Spectrum > Related Articles Around the Web * Virtual fixtures: Perceptual tools for telerobotic manipulation | IEEE ... > * Virtual Human Interaction Lab: VHIL > * Augmented reality - Wikipedia > augmented realityvirtual realitytech history Louis Rosenberg Louis Rosenberg is CEO and chief scientist of Unanimous AI, a company developing AI algorithms modeled on biological swarms. After creating the first interactive augmented-reality system for the U.S. Air Force in the early 1990s, Rosenberg founded the early virtual-reality company Immersion, the 3D-digitizer company MicroScribe, and the early augmented-reality company Outland Research. Rosenberg has been awarded over 300 patents worldwide for his work in VR, AR, AI, and human-computer interaction. He lives with his family on a 40-acre animal sanctuary along with hundreds of chickens, ducks, geese, turkeys, cows, pigs, goats, and sheep (real sheep, not virtual). The Conversation (2) [defa] Fridolin Wild12 Jul, 2022 M Question: usually, coining "Augmented Reality" is attributed to Caudell and Mizell - round about that time (the quoted publication is from 1992, but they had been using the phrase already earlier). Any references for Lanier? 1 Reply Hide replies Show More Replies a silver car driving down the road with a mountain of switchbacks behind it TopicTypeNewsTransportation New EV Prototype Leaves Range Anxiety in the Dust 6h 5 min read Conceptual illustration shows a brain shape made of circuits on a multilayered chip structure. Artificial IntelligenceTopicTypeComputingNews Artificial Synapses 10,000x Faster Than Real Thing 06 Aug 2022 3 min read A photo of an iRobot Roomba with an Amazon logo digitally added to it RoboticsNewsTypeTopic Amazon to Acquire iRobot F or $1.7 Billion 05 Aug 2022 4 min read RoboticsNewsTypeTopic Video Friday: Build a Chair Your weekly selection of awesome robot videos Evan Ackerman Evan Ackerman is a senior editor at IEEE Spectrum. Since 2007, he has written over 6,000 articles on robotics and technology. He has a degree in Martian geology and is excellent at playing bagpipes. 05 Aug 2022 2 min read A humanoid robot assembles an Ikea chair next to a human controlling that robot using a hardware system that duplicates limb motions video fridayrobotics Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion. IEEE CASE 2022: 20-24 August 2022, MEXICO CITY CLAWAR 2022: 12-14 September 2022, AZORES, PORTUGAL ANA Avatar XPRIZE Finals: 4-5 November 2022, LOS ANGELES CoRL 2022: 14-18 December 2022, AUCKLAND, NEW ZEALAND Enjoy today's videos! This probably counts as hard mode for Ikea chair assembly. [ Naver Lab ] As anyone working with robotics knows, it's mandatory to spend at least 10 percent of your time just mucking about with them because it's fun, as GITAI illustrates with its new 10-meter robotic arm. [ GITAI ] Well, this is probably the weirdest example of domain randomization in simulation for quadrupeds that I've ever seen. [ RSL ] The RoboCup 2022 was held in Bangkok, Thailand. The final match was between B-Human from Bremen (jerseys in black) and HTWK Robots from Leipzig (jerseys in blue). The video starts with one of our defending robots starting a duel with the opponent. After a short time a pass is made to another robot, which tries to score a goal, but the opponent goalie is able to catch the ball. Afterwards another attacker robot is already waiting at the center circle, to take its chance to score a goal, through all four opponent robots. [ Team B-Human ] The mission to return Martian samples back to Earth will see a European 2.5-meter-long robotic arm pick up tubes filled with precious soil from Mars and transfer them to a rocket for a historic interplanetary delivery. [ ESA ] I still cannot believe that this is an approach to robotic fruit-picking that actually works. [ Tevel Aerobotics ] This video shows the basic performance of the humanoid robot Torobo, which is used as a research platform for JST's Moonshot R &D program. [ Tokyo Robotics ] Volocopter illustrates why I always carry two violins with me everywhere. You know, just in case. [ Volocopter ] We address the problem of enabling quadrupedal robots to perform precise shooting skills in the real world using reinforcement learning. Developing algorithms to enable a legged robot to shoot a soccer ball to a given target is a challenging problem that combines robot motion control and planning into one task. [ Hybrid Robotics ] I will always love watching Cassie try very, very hard to not fall over, and then fall over. <3 [ Michigan Robotics ] I don't think this paper is about teaching bipeds to walk with attitude, but it should be. [ DLG ] Modboats are capable of collective swimming in arbitrary configurations! In this video you can see three different configurations of the Modboats swim across our test space and demonstrate their capabilities. [ ModLab ] How have we built our autonomous driving technology to navigate the world safely? It comes down to three easy steps: Sense, Solve, and Go. Using a combination of lidar, camera, radar, and compute, the Waymo Driver can visualize the world, calculate what others may do, and proceed smoothly and safely, day and night. [ Waymo ] Alan Alda discusses evolutionary robotics with Hod Lipson and Jordan Pollack on Scientific American Frontiers in 1999. [ Creative Machines Lab ] Brady Watkins gives us insight into how a big company like Softbank Robotics looks into the robotics market. [ Robohub ] Keep Reading |Show less TransportationTopicTypeComputingNews Who Actually Owns Tesla's Data? The company, says the company--but other interpretations persist Mark Harris Mark Harris is an investigative science and technology reporter based in Seattle, with a particular interest in robotics, transportation, green technologies, and medical devices. He's on Twitter at @meharris and email at mark(at)meharris(dot)com. Email or DM for Signal number for sensitive/encrypted messaging. 05 Aug 2022 4 min read Nighttime photograph of a man in a car at an outdoor Tesla charging lot. A Tesla user charges his Model S in Burbank, Calif. Philip Cheung/The New York Times Tesla MotorsAutopilotBig Data On 29 September 2020, a masked man entered a branch of the Wells Fargo bank in Washington, D.C., and handed the teller a note: "This is a robbery. Act calm give me all hundreds." The teller complied. The man then fled the bank and jumped into a gray Tesla Model S. This was one of three bank robberies the man attempted the same day. When FBI agents began investigating, they reviewed Washington, D.C.'s District Department of Transportation camera footage, and spotted a Tesla matching the getaway vehicle's description. The license plate on that car showed that it was registered to Exelorate Enterprises LLC, the parent company of Steer EV--a D.C.-based monthly vehicle-subscription service. Agents served a subpoena on Steer EV for the renter's billing and contact details. Steer EV provided those--and also voluntarily supplied historical GPS data for the vehicle. The data showed the car driving between, and parking at, each bank at the time of the heists. The renter was arrested and, in September, sentenced to four years in prison. "If an entity is collecting, retaining, [and] sharing historical location data on an individualized level, it's extraordinarily difficult to de-identify that, verging on impossible." --John Verdi, Future of Privacy Forum In this case, the GPS data likely came from a device Steer EV itself installed in the vehicle (neither Steer nor Tesla responded to interview requests). However, according to researchers, Tesla is potentially in a position to provide similar GPS tracks for many of its 3 million customers. For Teslas built since mid-2017, "every time you drive, it records the whole track of where you drive, the GPS coordinates and certain other metrics for every mile driven," says Green, a Tesla owner who has reverse engineered the company's Autopilot data collection. "They say that they are anonymizing the trigger results," but, he says, "you could probably match everything to a single person if you wanted to." Each of these trip logs, and other data "snapshots" captured by the Autopilot system that include images and video, is stripped of its identifying VIN and given a temporary, random ID number when it is uploaded to Tesla, says Green. However, he notes, that temporary ID can persist for days or weeks, connecting all the uploads made during that time. Black and white photo of a man in a suit looking off into the distanceElon Musk, CEO of Tesla MotorsMark Mahaney/Redux Given that some trip logs will also likely record journeys between a driver's home, school, or place of work, guaranteeing complete anonymity is unrealistic, says John Verdi, senior vice president of policy at the Future of Privacy Forum: "If an entity is collecting, retaining, [and] sharing historical location data on an individualized level, it's extraordinarily difficult to de-identify that, verging on impossible." Tesla, like all other automakers, has a policy that spells out what it can and cannot do with the data it gets from customers' vehicles, including location information. This states that while the company does not sell customer and vehicle data, it can share that data with service providers, business partners, affiliates, some authorized third parties, and government entities according to the law. Owners can buy a special kit for US $1,400 that allows them to access data on their own car's event data recorder, but this represents just a tiny subset of the data the company collects, and is related only to crashes. Owners living in California and Europe benefit from legislation that means Tesla will provide access to more data generated by their vehicles, although not the Autopilot snapshots and trip logs that are supposedly anonymized. Once governments realize that a company possesses such a trove of information, it could be only a matter of time before they seek access to it. "If the data exists...and in particular exists in the domain of somebody who's not the subject of those data, it's much more likely that a government will eventually get access to them in some way," says Bryant Walker Smith, an associate professor in the schools of law and engineering at the University of South Carolina. "Individuals ought to think about their cars more like they think about their cellphones." --John Verdi, Future of Privacy Forum This is not necessarily a terrible thing, Walker says, who suggests that such rich data could unlock valuable insights into which roads or intersections are dangerous. The wealth of data could also surface subtle problems in the vehicles themselves. In many ways, the data genie is already out of the bottle, according to Verdi. "Individuals ought to think about their cars more like they think about their cellphones," he says. "The auto industry has a lot to learn from the ways that mobile-phone operating systems handle data permissions.... Both iOS and Android have made great strides in recent years in empowering consumers when it comes to data collection, data disclosure, and data use." Tesla permits owners to control some data sharing, including Autopilot and road segment analytics. If they want to opt out of data collection completely, they can ask Tesla to disable the vehicle's connectivity altogether. However, this would mean losing features such as remote services, Internet radio, voice commands, and Web browser functionality, and even safety-related over-the-air updates. Green says he is not aware of anyone who has successfully undergone this nuclear option. The only real way to know you've prevented data sharing, he says, is to "go to a repair place and ask them to remove the modem out of the car." Tesla almost certainly has the biggest empire of customer and vehicle data among automakers. It also appears to be the most aggressive in using that data to develop its automated driving systems, and to protect its reputation in the courts of law and public opinion, even to the detriment of some of its customers. But while the world's most valuable automaker dominates the discussion around connected cars, others are not far behind. Elon Musk's insight--to embrace the data-driven world that our other digital devices already inhabit--is rapidly becoming the industry standard. When our cars become as powerful and convenient as our phones, it is hardly surprising that they suffer the same challenges around surveillance, privacy, and accountability. From Your Site Articles * The Radical Scope of Tesla's Data Hoard - IEEE Spectrum > * Tesla's Autopilot Depends on a Deluge of Data - IEEE Spectrum > Related Articles Around the Web * Tesla is collecting insane amount of data from its Full Self-Driving ... > * Tesla's Autopilot Safety Data Touted by Elon Musk Is Misleading ... > Keep Reading |Show less {"imageShortcodeIds":[]} TransportationWhitepaper GPIOs: Critical IP for Functional Safety Applications Understand the safety mechanisms in an automotive-ready GPIO IP library suite to detect the faults in GPIO cells Synopsys 3h 1 min read GPIOs: Critical IP for Functional Safety Applications The prevalence and complexity of electronics and software in automotive applications are increasing with every new generation of cars. The critical functions within the system on a chip (SoC) involve hardware and software that perform automotive-related signal communication at high data rates to and from the components off-chip. Every SoC includes general purpose IOs (GPIOs) on its periphery. For automotive SoCs, GPIO IP is typically developed as Safety Element out of Context and delivered with a set of Assumptions of Use. It is important that the GPIO blocks are treated as a safety related logic. In this role, GPIOs need safety analysis to mitigate any faults occurring in them before the result of fault occurrence causes a system-wide failure. This white paper describes some of the commonly used safety mechanisms in an automotive-ready GPIO library suite. It will then describe how safety related deliverables are helpful to SoC integrators in their design of safe SoCs. Keep Reading |Show less Trending Stories The most-read stories on IEEE Spectrum right now TopicTypeNewsTransportation New EV Prototype Leaves Range Anxiety in the Dust 6h 5 min read a silver car driving down the road with a mountain of switchbacks behind it TransportationTopicTypeComputingNews Who Actually Owns Tesla's Data? 05 Aug 2022 4 min read Nighttime photograph of a man in a car at an outdoor Tesla charging lot. TransportationTopicNewsType The Radical Scope of Tesla's Data Hoard 03 Aug 2022 5 min read Aerial view of a Tesla charging lot Artificial IntelligenceTopicTypeComputingNews Artificial Synapses 10,000x Faster Than Real Thing 06 Aug 2022 3 min read Conceptual illustration shows a brain shape made of circuits on a multilayered chip structure. TransportationTopicTypeFeature Self-Driving Cars Work Better With Smart Roads 29 Jun 2022 9 min read A photograph shows a single car headed toward the viewer on the rightmost lane of a three-lane road that is bounded by grassy parkways, one side of which is planted with trees. In the foreground a black vertical pole is topped by a crossbeam bearing various instruments. TransportationTopicTypeComputingNews Tesla's Autopilot Depends on a Deluge of Data 04 Aug 2022 5 min read Close-up of the Autopilot screen in a Tesla EnergyAnalysisTypeTopic Rhode Island's Renewable Energy Goal Is a Beacon for Other States 04 Aug 2022 3 min read Wind turbines in an offshore installation RoboticsNewsTypeTopic Video Friday: Build a Chair 05 Aug 2022 2 min read A humanoid robot assembles an Ikea chair next to a human controlling that robot using a hardware system that duplicates limb motions