https://spectrum.ieee.org/nuclear-submarine [ ] IEEE.orgIEEE Xplore Digital LibraryIEEE StandardsMore Sites Sign InJoin IEEE Will AI Steal Submarines' Stealth? Share FOR THE TECHNOLOGY INSIDER Search: [ ] Explore by topic AerospaceArtificial IntelligenceBiomedicalComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensors TelecommunicationsTransportation IEEE Spectrum FOR THE TECHNOLOGY INSIDER Topics AerospaceArtificial IntelligenceBiomedicalComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensors TelecommunicationsTransportation Sections FeaturesNewsOpinionCareersDIYThe Big PictureEngineering Resources More Special ReportsCollectionsExplainersPodcastsVideosNewslettersTop Programming LanguagesRobots Guide For IEEE Members Current IssueMagazine ArchiveThe InstituteTI Archive For IEEE Members Current IssueMagazine ArchiveThe InstituteTI Archive IEEE Spectrum About UsContact UsReprints & PermissionsAdvertising Follow IEEE Spectrum Support IEEE Spectrum IEEE Spectrum is the flagship publication of the IEEE -- the world's largest professional organization devoted to engineering and applied sciences. Our articles, podcasts, and infographics inform our readers about developments in technology, engineering, and science. Join IEEE Subscribe About IEEEContact & SupportAccessibilityNondiscrimination PolicyTerms IEEE Privacy Policy (c) Copyright 2022 IEEE -- All rights reserved. A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. IEEE websites place cookies on your device to give you the best user experience. By using our websites, you agree to the placement of these cookies. To learn more, read our Privacy Policy. view privacy policy accept & close Enjoy more free content and benefits by creating an account Saving articles to read later requires an IEEE Spectrum account The Institute content is only available for members Downloading full PDF issues is exclusive for IEEE Members Access to Spectrum's Digital Edition is exclusive for IEEE Members Following topics is a feature exclusive for IEEE Members Adding your response to an article requires an IEEE Spectrum account Create an account to access more content and features on IEEE Spectrum, including the ability to save articles to read later, download Spectrum Collections, and participate in conversations with readers and editors. For more exclusive content and features, consider Joining IEEE. Join the world's largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum's articles, archives, PDF downloads, and other benefits. Learn more - CREATE AN ACCOUNTSIGN IN JOIN IEEESIGN IN Close Access Thousands of Articles -- Completely Free Create an account and get exclusive content and features: Save articles, download collections, and talk to tech insiders -- all free! For full access and benefits, join IEEE as a paying member. CREATE AN ACCOUNTSIGN IN Artificial IntelligenceTopicTypeFeature Will AI Steal Submarines' Stealth? Better detection will make the oceans transparent--and perhaps doom mutually assured destruction Natasha Bajema 7h 11 min read A photo of a submarine in the water under a partly cloudy sky. The Virginia-class fast attack submarine USS Virginia cruises through the Mediterranean in 2010. Back then, it could effectively disappear just by diving. U.S. Navy Submarines are valued primarily for their ability to hide. The assurance that submarines would likely survive the first missile strike in a nuclear war and thus be able to respond by launching missiles in a second strike is key to the strategy of deterrence known as mutually assured destruction. Any new technology that might render the oceans effectively transparent, making it trivial to spot lurking submarines, could thus undermine the peace of the world. For nearly a century, naval engineers have striven to develop ever-faster, ever-quieter submarines. But they have worked just as hard at advancing a wide array of radar, sonar, and other technologies designed to detect, target, and eliminate enemy submarines. The balance seemed to turn with the emergence of nuclear-powered submarines in the early 1960s. In a 2015 study for the Center for Strategic and Budgetary Assessment, Bryan Clark, a naval specialist now at the Hudson Institute, noted that the ability of these boats to remain submerged for long periods of time made them "nearly impossible to find with radar and active sonar." But even these stealthy submarines produce subtle, very-low-frequency noises that can be picked up from far away by networks of acoustic hydrophone arrays mounted to the seafloor. --------------------------------------------------------------------- And now the game of submarine hide-and-seek may be approaching the point at which submarines can no longer elude detection and simply disappear. It may come as early as 2050, according to a recent study by the National Security College of the Australian National University, in Canberra. This timing is particularly significant because the enormous costs required to design and build a submarine are meant to be spread out over at least 60 years. A submarine that goes into service today should still be in service in 2082. Nuclear-powered submarines, such as the Virginia-class fast-attack submarine, each cost roughly US $2.8 billion, according to the U.S. Congressional Budget Office. And that's just the purchase price; the total life cycle cost for the new Columbia-class ballistic-missile submarine is estimated to exceed $395 billion. The twin problems of detecting submarines of rival countries and protecting one's own submarines from detection are enormous, and the technical details are closely guarded secrets. Many naval experts are speculating about sensing technologies that could be used in concert with modern AI methodologies to neutralize a submarine's stealth. Rose Gottemoeller, former deputy secretary general of NATO, warns that "the stealth of submarines will be difficult to sustain, as sensing of all kinds, in multiple spectra, in and out of the water becomes more ubiquitous." And the ongoing contest between stealth and detection is becoming increasingly volatile as these new technologies threaten to overturn the balance. We have new ways to find submarines Today's sensing technologies for detecting submarines are moving beyond merely hearing submarines to pinpointing their position through a variety of non-acoustic techniques. Submarines can now be detected by the tiny amounts of radiation and chemicals they emit, by slight disturbances in the Earth's magnetic fields, and by reflected light from laser or LED pulses. All these methods seek to detect anomalies in the natural environment, as represented in sophisticated models of baseline conditions that have been developed within the last decade, thanks in part to Moore's Law advances in computing power. [svg]Airborne laser-based sensors can detect submarines lurking near the surface.IEEE Spectrum According to experts at the Center for Strategic and International Studies, in Washington, D.C., two methods offer particular promise. Lidar sensors transmit laser pulses through the water to produce highly accurate 3D scans of objects. Magnetic anomaly detection (MAD) instruments monitor the Earth's magnetic fields and can detect subtle disturbances caused by the metal hull of a submerged submarine. Both sensors have drawbacks. MAD works only at low altitudes or underwater. It is often not sensitive enough to pick out the disturbances caused by submarines from among the many other subtle shifts in electromagnetic fields under the ocean. Lidar has better range and resolution and can be installed on satellites, but it consumes a lot of power--a standard automotive unit with a range of several hundred meters can burn 25 watts. Lidar is also prohibitively expensive, especially when operated in space. In 2018, NASA launched a satellite with laser imaging technology to monitor changes in Earth's surface--notably changes in the patterns on the ocean's surface; the satellite cost more than $1 billion. Indeed, where you place the sensors is crucial. Underwater sensor arrays won't put an end to submarine stealth by themselves. Retired Rear Adm. John Gower, former submarine commander for the Royal Navy of the United Kingdom, notes that sensors "need to be placed somewhere free from being trolled or fished, free from seismic activity, and close to locations from which they can be monitored and to which they can transmit collected data. That severely limits the options available." One way to get around the need for precise placement is to make the sensors mobile. Underwater drone swarms can do just that, which is why some experts have proposed them as the ultimate antisubmarine capability. Clark, for instance, notes that such drones now have enhanced computing power and batteries that can last for two weeks between charges. The U.S. Navy is working on a drone that could run for 90 days. Drones are also now equipped with the chemical, optical, and geomagnetic sensors mentioned earlier. Networked underwater drones, perhaps working in conjunction with airborne drones, may be useful for not only detecting submarines but also destroying them, which is why several militaries are investing heavily in them. A photo of a plane on a runway.A U.S. Navy P-8 Poseidon aircraft, equipped to detect submarines, awaits refueling in Okinawa, Japan, in 2020. U.S.Navy For example, the Chinese Navy has invested in a fishlike undersea drone known as Robo-Shark, which was designed specifically for hunting submarines. Meanwhile, the U.S. Navy is developing the Low-Cost Unmanned Aerial Vehicle Swarming Technology, for conducting surveillance missions. Each Locust drone weighs about 6 kilograms, costs $15,000, and can be outfitted with MAD sensors; it can skim low over the ocean's surface to detect signals under the water. Militaries study the drone option because it might work. Then again, it very well might not. A photo of a robotic shark. Robo-Shark, a 2.2-meter-long submersible made by Boya Gongdao Robot Technology, of Beijing, is said to be capable of underwater surveillance and unspecified antisubmarine operations. The company says that the robot moves at up to 5 meters per second (10 knots) by using a three-joint structure to wave the caudal fin, making less noise than a standard propeller would. robosea.org Gower considers underwater drones to be "the least likely innovation to make a difference in the decline of submarine stealth." A navy would need a lot of drones, data rates are exceedingly slow, and a drone's transmission range is short. Drones are also noisy and extremely easy to detect. "Not to mention that controlling thousands of underwater drones far exceeds current technological capabilities," he adds. Gower says it could be possible "to use drones and sonar networks together in choke points to detect submarine patrols." Among the strategically important submarine patrol choke points are the exit routes on either side of Ireland, for U.K. submarines; those around the islands of Hainan and Taiwan, for Chinese submarines; in the Barents or Kuril Island chain, for Russian submarines; and the Straits of Juan de Fuca, for U.S. Pacific submarines. On the other hand, he notes, "They could be monitored and removed since they would be close to sovereign territories. As such, the challenges would likely outweigh the gains." Gower believes a more powerful means of submarine detection lies in the "persistent coverage of the Earth's surface by commercial satellites," which he says "represents the most substantial shift in our detection capabilities compared to the past." More than 2,800 of these satellites are already in orbit. Governments once dominated space because the cost of building and launching satellites was so great. These days, much cheaper satellite technology is available, and private companies are launching constellations of tens to thousands of satellites that can work together to image every bit of the Earth's surface. They are outfitted with a wide range of sensing technologies, including synthetic aperture radar (SAR), which scans a scene down below while moving over a great distance, providing results like those you'd get from an extremely long antenna. Since these satellite constellations view the same locations multiple times per day, they can capture small changes in activity. Experts have known for decades about the possibility of detecting submarines with SAR based on the wake patterns they form as they move through the ocean. To detect such patterns, known as Bernoulli humps and Kelvin wakes, the U.S. Navy has invested in the AN/APS-154 Advanced Airborne Sensor, developed by Raytheon. The aircraft-mounted radar is designed to operate at low altitudes and appears to be equipped with high-resolution SAR and lidar sensors. Commercial satellites equipped with SAR and other imaging instruments are now reaching resolutions that can compete with those of government satellites and offer access to customers at extremely affordable rates. In other words, there's lots of relevant, unclassified data available for tracking submarines, and the volume is growing exponentially. One day this trend will matter. But not just yet. Jeffrey Lewis, director of the East Asia Nonproliferation Program at the James Martin Center for Nonproliferation Studies, regularly uses satellite imagery in his work to track nuclear developments. But tracking submarines is a different matter. "Even though this is a commercially available technology, we still don't see submarines in real time today," Lewis says. The day when commercial satellite imagery reduces the stealth of submarines may well come, says Gower, but "we're not there yet. Even if you locate a submarine in real time, 10 minutes later, it's very hard to find again." Artificial intelligence coordinates other sub-detecting tech Though these new sensing methods have the potential to make submarines more visible, no one of them can do the job on its own. What might make them work together is the master technology of our time: artificial intelligence. "When we see today's potential of ubiquitous sensing capabilities combined with the power of big-data analysis," Gottemoeller says, "it's only natural to ask the question: Is it now finally possible?" She began her career in the 1970s, when the U.S. Navy was already worried about Soviet submarine-detection technology. Submarines can now be detected by the tiny amounts of radiation and chemicals they emit, by slight disturbances in the Earth's magnetic fields, and by reflected light from laser or LED pulses. Unlike traditional software, which must be programmed in advance, the machine-learning strategy used here, called deep learning, can find patterns in data without outside help. Just this past year, DeepMind's AlphaFold program achieved a breakthrough in predicting how amino acids fold into proteins, making it possible for scientists to identify the structure of 98.5 percent of human proteins. Earlier work in games, notably Go and chess, showed that deep learning could outdo the best of the old software techniques, even when running on hardware that was no faster. For AI to work in submarine detection, several technical challenges must be overcome. The first challenge is to train the algorithm, which involves acquiring massive volumes and varieties of sensor data from persistent satellite coverage of the ocean's surface as well as regular underwater collection in strategic locations. Using such data, the AI can establish a detailed model of baseline conditions, then feed new data into the model to find subtle anomalies. Such automated sleuthing is what's likeliest to detect the presence of a submarine anywhere in the ocean and predict locations based on past transit patterns. The second challenge is collecting, transmitting, and processing the masses of data in real time. That task would require a lot more computing power than we now have, both in fixed and on mobile collection platforms. But even today's technology can start to put the various pieces of the technical puzzle together. Nuclear deterrence depends on the ability of submarines to hide For some years to come, the vastness of the ocean will continue to protect the stealth of submarines. But the very prospect of greater ocean transparency has implications for global security. Concealed submarines bearing ballistic missiles provide the threat of retaliation against a first nuclear strike. What if that changes? "We take for granted the degree to which we rely upon having a significant portion of our forces exist in an essentially invulnerable position," Lewis says. Even if new developments did not reduce submarine stealth by much, the mere perception of such a reduction could undermine strategic stability. A gray unmanned helicopter, notably lacking a cockpit or any kind of window, is shown hovering against a clear, blue sky. It carries a downward-poinging sensor under its nose. A Northrop Grumman MQ-8C, an uncrewed helicopter, has recently been deployed by the U.S. Navy in the Indo-Pacific area for use in surveillance. In the future, it will also be used for antisubmarine operations. Northrop Grumman Gottemoeller warns that "any perception that nuclear-armed submarines have become more targetable will lead to questions about the survivability of second-strike forces. Consequently, countries are going to do everything they can to counter any such vulnerability." Experts disagree on the irreversibility of ocean transparency. Because any technological breakthroughs will not be implemented overnight, "nations should have ample time to develop countermeasures [that] cancel out any improved detection capabilities," says Matt Korda, senior research associate at the Federation of American Scientists, in Washington, D.C. However, Roger Bradbury and eight colleagues at the National Security College of the Australian National University disagree, claiming that any technical ability to counter detection technologies will start to decline by 2050. Korda also points out that ocean transparency, to the extent that it occurs, "will not affect countries equally. And that raises some interesting questions." For example, U.S. nuclear-powered submarines are "the quietest on the planet. They are virtually undetectable. Even if submarines become more visible in general, this may have zero meaningful effect on U.S. submarines' survivability." Sylvia Mishra, a new-tech nuclear officer at the European Leadership Network, a London-based think tank, says she is "more concerned about the overall problem of ambiguity under the sea." Until recently, she says, movement under the oceans was the purview of governments. Now, though, there's a growing industry presence under the sea. For example, companies are laying many underwater fiber-optic communication cables, Mishra says, "which may lead to greater congestion of underwater inspection vehicles, and the possibility for confusion." A large, cylindrical vehicle is shown just as it has been lowered below the surface of the water, suspended by two green cables.A Snakehead, a large underwater drone designed to be launched and recovered by U.S. Navy nuclear-powered submarines, is shown at its christening ceremony in Narragansett Bay in Newport, R.I.U.S. Navy Confusion might come from the fact that drones, unlike surface ships, do not bear a country flag, and therefore their ownership may be unclear. This uncertainty, coupled with the possibility that the drones could also carry lethal payloads, increases the risk that a naval force might view an innocuous commercial drone as hostile. "Any actions that hold the strategic assets of adversaries at risk may produce new touch points for conflict and exacerbate the risk of war," says Mishra. Given the strategic importance of submarine stealth, Gower asks, "Why would any country want to detect and track submarines? It's only something you'd do if you want to make a nuclear-armed power nervous." Even in the Cold War, when the United States and the U.K. routinely tracked Soviet ballistic-missile submarines, they did so only because they knew their activities would go undetected--that is, without risking escalation. Gower postulates that this was dangerously arrogant: "To actively track second-strike nuclear forces is about as escalatory as you might imagine." "All nuclear-armed states place a great value on their second-strike forces," Gottemoeller says. If greater ocean transparency produces new risks to their survivability, real or perceived, she says, countries may respond in two ways: build up their nuclear forces further and take new measures to protect and defend them, producing a new arms race; or else keep the number of nuclear weapons limited and find other ways to bolster their viability. Ultimately, such considerations have not dampened the enthusiasm of certain governments for acquiring submarines. In September 2021 the Australian government announced an enhanced trilateral partnership with the United States and the United Kingdom. The new deal, known as AUKUS, will provide Australia with up to eight nuclear-powered submarines with the most coveted propulsion technology in the world. However, it could be at least 20 years before the Royal Australian Navy can deploy the first of its new subs. [svg]The Boeing Orca, the largest underwater drone in the U.S. Navy's inventory, was christened in April, in Huntington Beach, Calif. The craft is designed, among other things, for use in antisubmarine warfare. The Boeing Company As part of its plans for nuclear modernization, the United States has started replacing its entire fleet of 14 Ohio-class ballistic-missile submarines with new Columbia-class boats. The replacement program is projected to cost more than $128 billion for acquisition and $267 billion over their full life cycles. U.S. government officials and experts justify the steep cost of these submarines with their critical role in bolstering nuclear deterrence through their perceived invulnerability. To protect the stealth of submarines, Mishra says, "There is a need for creative thinking. One possibility is exploring a code of conduct for the employment of emerging technologies for surveillance missions." There are precedents for such cooperation. During the Cold War, the United States and the Soviet Union set up a secure communications system--a hotline--to help prevent a misunderstanding from snowballing into a disaster. The two countries also developed a body of rules and procedures, such as never to launch a missile along a potentially threatening trajectory. Nuclear powers could agree to exercise similar restraint in the detection of submarines. The stealthy submarine isn't gone; it still has years of life left. That gives us ample time to find new ways to keep the peace. From Your Site Articles * World's Largest Swarm of Miniature Robot Submarines - IEEE ... > * Submarines - IEEE Spectrum > * Scientists Explore Underwater Quantum Links for Submarines ... > * DARPA's Self-Driving Submarine Hunter Steers Like a Human ... > Related Articles Around the Web * The Capacity of the Navy's Shipyards to Maintain Its Submarines ... > * Attack Submarines - SSN > United States Navy > Displayy-FactFiles > * Ballistic Missile Submarines | Commander, Submarine Force, U.S. ... > * Fleet Ballistic Missile Submarines - SSBN > United States Navy ... > sensorssubmarinestealth technologylidaracoustic sensorartificial intelligencenuclear deterrence {"imageShortcodeIds":["30133857"]} Natasha Bajema Dr. Natasha Bajema has held long-term assignments at the National Defense University, in the U.S. Office of the Secretary of Defense, and at the U.S. Department of Energy's National Nuclear Security Administration. She's currently Director of the Converging Risks Lab at the Council on Strategic Risks. The Conversation (0) A MetaHuman character open for edits in the MetaHuman software. Consumer ElectronicsTopicTypeComputingNews Does MetaHuman's Digital Clone Cross the Uncanny Valley? 8h 4 min read illustration of open mouth BiomedicalTopicTypeSensorsNews A Breath Test for Monitoring Glucose Levels 15 Jul 2022 2 min read A pair of very mechanical looking robot arms grasp a cable on a test stand RoboticsNewsTypeTopic Video Friday: Robot Arms for the ISS 15 Jul 2022 4 min read TelecommunicationsTopicNewsType SpaceX and Dish's Super-Shady War for the World Or, why three billionaires are girding for battle over spectrum supremacy--and why it matters Michael Dumiak Michael Dumiak is a Berlin-based writer and reporter covering science and culture and a longtime contributor to IEEE Spectrum. For Spectrum , he has covered digital models of ailing hearts in Belgrade, reported on technology from Minsk and shale energy from the Estonian-Russian border, explored cryonics in Saarland, and followed the controversial phaseout of incandescent lightbulbs in Berlin. He is author and editor of Woods and the Sea: Estonian Design and the Virtual Frontier. 15 Jul 2022 3 min read map of globe with blue orbs of light around iStockphoto satellitesspacexstarlinkdelldish networkElon Musk Billionaires, satellite links, political chicanery: a present-day, oligopolistic game of jockeying for prime placement in the 12-gigahertz spectrum has at least a few of the ingredients of a thriller. Or--given the outsize personalities involved (including Elon Musk and Michael Dell) and the epic, five-year duration of the dispute to date--maybe more like a space opera. At issue is a set of frequencies where Musk's SpaceX sets its Starlink Internet service, the company's well-publicized play for broadband beaming down from low-Earth orbit to satellite dishes in remote areas. Charlie Ergen's Dish Network Corp., which transmits TV on these frequencies and is one of the two big satellite viewing providers in the United States, has launched a 5G wireless service and wants to increase its signal volume in this wavelength. Musk's side says the move would make debilitating static for his satellites; Ergen's engineers say that's nonsense. As for Dell (you may recall Dell laptops) his private investment firm holds interest in some of the airwaves in play. At the moment, they're siding with Dish. The current field, more precisely 12.2 to 12.7 GHz in the Ku microwave band, is a lot of bandwidth lightly used--primarily today for assorted satellite broadcasts, live feeds, ISS tracking, and military recon drones. But the corporates fighting over it recently cranked up their clashing. The sides are lobbying a shorthanded Federal Communications Commission, with recent highlight swipes including Musk blasting his foes as "super shady and unethical" while taking return fire from Dish for "flimsy" and "far-fetched" objections to opening bandwidth. But what's it mean for those outside the immediate fray? For civilians going about their daily business? For people--possible satellite service subscribers, all--around the world? Only a handful of people who understand the nature of possible interference and related issues seem to be paying attention now. "Rights to use frequencies have not been sharply defined, and the overlapping permits generate controversy," says Thomas Hazlett, a Clemson University economist who writes about bandwidth battles (and once served as FCC chief economist). But the rulings--and market activities that result--stand to have real social impact wherever signals from satellite broadcasts or satellite Internet connections may one day fall. Which means pretty much everywhere. It's a stark contrast given the economic value placed on frequency rights. More than 100 bandwidth auctions over the last 30 years have netted about US $280 billion for the U.S. Treasury. And as with television, radio, and the railway before that, citizens aren't likely to tune in until more tangible developments happen. But others are paying attention, and the pressures are intense. "In the U.S., it is purely market driven," says Shahed Mazumder, global director of telecom solutions at Aerospike, a database firm. SpaceX has launched thousands of new satellites; Dish is trying to move into new services. Neither wants interference. "The political pressure, the business pressure, the monetary pressure: There are legitimately major things going on here," says Mike Dano, who's been following the dispute as editorial director at Light Reading, a news website covering international teleco. Billions of dollars of value, potentially, to be created or destroyed depending on how an FCC engineer finally decides on it." Meanwhile the politically appointed commissioners at the FCC are down one member, splitting it 2-2 along party lines. It makes controversial calls more difficult. More techie influences may also affect the spectrum spat. In 2018 the United States became the first to approve a spectrum-sharing setup in the Citizens Broadband Radio Service band (3.5 GHz). It's an advanced concept allowing different sets of users to share spectrum--making more room. Dano says the FCC is under pressure to allocate 12 GHz in a way set up for spectrum sharing. This notably did not happen with U.S 5G network rollouts, which turned into a snarling issue earlier this year over fears of interference in the C-band between high-speed cellular service towers and plane altimeters in low-visibility conditions on approach to airports. Meanwhile Starlink wants to open its satellite show in further-flung places, seeking to open gateways in the United Kingdom. While its 12-GHz fight with Dish is centered in the United States and Dish's U.S. services, satellite spectrum allocation is...special. Space has international dimensions, points out Plum Consulting's Selcuk Kirtay, who was writing about spectrum sharing in 2002. Slicing up the Ku band has history--it even left its mark at a global astronautical confab in then-Czechoslovakia in Star Wars-era 1977. Ofcom, the U.K. regulator, is monitoring developments with U.S. allocations in 12 GHz. The German network authority is preparing to discuss the 12.2-to-12.7-GHz frequency range at next year's ITU World Radiocommunication Conference in the United Arab Emirates. Stay tuned, say experts and satellite industry watchers. "How the conflicts are resolved in the USA will materially affect markets around the world," Hazlett says. From Your Site Articles * SpaceX, OneWeb, or Kepler Communications: Who Really ... > * SpaceX Claims to Have Redesigned Its Starlink Satellites to ... > Related Articles Around the Web * Starlink - Wikipedia > * Starlink > Keep Reading |Show less RoboticsTopicTypeGuest Article Wandering Robots in the Wild Sometimes it's okay to not know where you are, as long as you keep moving Nick Walker Amal Nanavati 14 Jul 2022 7 min read A small black and white wheeled robot with a round head with its back turned looks down a long and empty academic hallway Kuri wanders down a hallway as it explores a building. kurirobot navigationroboticssocial robots In order to better understand how people will interact with mobile robots in the wild, we need to take them out of the lab and deploy them in the real world. But this isn't easy to do. Roboticists tend to develop robots under the assumption that they'll know exactly where their robots are at any given time--clearly that's an important capability if the robot's job is to usefully move between specific locations. But that ability to localize generally requires the robot to have powerful sensors and a map of its environment. There are ways to wriggle out of some of these requirements: If you don't have a map, there are methods that build a map and localize at the same time, and if you don't have a good range sensor, visual navigation methods use just a regular RGB camera, which most robots would have anyway. Unfortunately, these alternatives to traditional localization-based navigation are either computationally expensive, not very robust, or both. We ran into this problem when we wanted to deploy our Kuri mobile social robot in the halls of our building for a user study. Kuri's lidar sensor can't see far enough to identify its location on a map, and its onboard computer is too weak for visual navigation. After some thought, we realized that for the purposes of our deployment, we didn't actually need Kuri to know exactly where it was most of the time. We did need Kuri to return to its charger when it got low on battery, but this would be infrequent enough that a person could help with that if necessary. We decided that perhaps we could achieve what we wanted by just letting Kuri abandon exact localization, and wander . Robotic Wandering If you've seen an older-model robot vacuum cleaner doing its thing, you're already familiar with what wandering looks like: The robot drives in one direction until it can't anymore, maybe because it senses a wall or because it bumps into an obstacle, and then it turns in a different direction and keeps going. If the robot does this for long enough, it's statistically very likely to cover the whole floor, probably multiple times. Newer and fancier robot vacuums can make a map and clean more systematically and efficiently, but these tend to be more expensive. You can think of a wandering behavior as consisting of three parts: 1. Moving in a straight line 2. Detecting event(s) that trigger the selection of a new direction 3. A method that's used to select a new direction Many possible wandering behaviors turn out not to work very well. For example, we found that having the robot move a few meters before selecting a new direction at random led it to get stuck moving back and forth in long corridors. The curve of the corridors meant that simply waiting for the robot to collide before selecting a new direction quickly devolved into the robot bouncing between the walls. We explored variations using odometry information to bias direction selection, but these didn't help because the robot's estimate of its own heading--which was poor to begin with--would degrade every time the robot turned. In the end, we found that a preference for moving in the same direction as long as possible--a strategy we call informed direction selection--was most effective at making Kuri roam the long, wide corridors of our building. Informed direction selection uses a local costmap--a small, continuously updating map of the area around the robot--to pick the direction that is easiest for the robot to travel in, breaking ties in preference for directions that are closer to the previously selected direction. The resulting behavior can look like a wave; the robot commits to a direction, but eventually an obstacle comes into view on the costmap and the local controller starts to turn the robot slightly to "get around it." If it were a small obstruction, like a person walking by, the robot would circumnavigate and continue in roughly the original direction, but in the case of large obstacles like walls, the local controller will eventually detect that it has drifted too far from the original linear plan and give up. Informed direction selection will kick in and trace lines through the costmap to find the most similar heading that goes through free space. Typically, this will be the line that moves along and slightly away from the wall. A simple graphic illustrating the path of a robot moving down a hallway where it cyclically gets closer and farther from walls Our wandering behavior is more complicated than something like always choosing to turn 90 degrees without considering any other context, but it's much simpler than any approach that involves localization, since the robot just needs to be able to perceive obstacles in its immediate vicinity and keep track of roughly which direction it's traveling in. Both of these capabilities are quite accessible, as there are implementations in core ROS packages that do the heavy lifting, even for basic range sensors and noisy inertial measurement units and wheel encoders. Like more intelligent autonomous-navigation approaches, wandering does sometimes go wrong. Kuri's lidar has a hard time seeing dark surfaces, so it would occasionally wedge itself against them. We use the same kinds of recovery behaviors that are common in other systems, detecting when the robot hasn't moved (or hasn't moved enough) for a certain duration, then attempting to rotate in place or move backward. We found it important to tune our recovery behaviors to unstick the robot from the hazards particular to our building. In our first rounds of testing, the robot would reliably get trapped with one tread dangling off a cliff that ran along a walkway. We were typically able to get the robot out via teleoperation, so we encoded a sequence of velocity commands that would rotate the robot back and forth to reengage the tread as a last-resort recovery. This type of domain-specific customization is likely necessary to fine-tune wandering behaviors for a new location. Other types of failures are harder to deal with. During testing, we occasionally ran the robot on a different floor, which had tables and chairs with thin, metallic legs. Kuri's lidar couldn't see these reliably and would sometimes "clothesline" itself with the seat of the chair, tilting back enough to lose traction. No combination of commands could recover the robot from this state, so adding a tilt-detection safety behavior based on the robot's cliff sensors would've been critical if we had wanted to deploy on this floor. Using Human Help Eventually, Kuri needs to get to a charger, and wandering isn't an effective way of making that happen. Fortunately, it's easy for a human to help. We built some chatbot software that the robot used to ping a remote helper when its battery was low. Kuri is small and light, so we opted to have the helper carry the robot back to its charger, but one could imagine giving a remote helper a teleoperation interface and letting them drive the robot back instead. A graphic of a floor plan of an academic building showing that the robot was able to make it to most of the space possibleKuri was able to navigate all 350 meters of hallway on this floor, which took it 32 hours in total. We deployed this system for four days in our building. Kuri was able to navigate all 350 meters of hallway on the floor, and ran for 32 hours total. Each of the 12 times Kuri needed to charge, the system notified its designated helper, and they found the robot and placed it on its charger. The robot's recovery behaviors kept it from getting stuck most of the time, but the helper needed to manually rescue it four times when it got wedged near a difficult-to-perceive banister. Wandering with human help enabled us to run an exploratory user study on remote interactions with a building photographer robot that wouldn't have been possible otherwise. The system required around half an hour of the helper's time over the course of its 32-hour deployment. A well-tuned autonomous navigation system could have done it with no human intervention at all, but we would have had to spend a far greater amount of engineering time to get such a system to work that well. The only other real alternative would have been to fully teleoperate the robot, a logistical impossibility for us. To Wander, or Not to Wander? It's important to think about the appropriate level of autonomy for whatever it is you want a robot to do. There's a wide spectrum between "autonomous" and "teleoperated," and a solution in the middle may help you get farther along another dimension that you care more about, like cost or generality. This can be an unfashionable suggestion to robotics researchers (for whom less-than-autonomous solutions can feel like defeat), but it's better to think of it as an invitation for creativity: What new angles could you explore if you started from an 80 percent autonomy solution rather than a fully autonomous solution? Would you be able to run a system for longer, or in a place you couldn't before? How could you sprinkle in human assistance to bridge the gap? We think that wandering with human help is a particularly effective approach in some scenarios that are especially interesting to human-robot interaction researchers, including: * Studying human perceptions of robots * Studying how robots should interact with and engage bystanders * Studying how robots can interact with remote users and operators You obviously wouldn't want to build a commercial mail-courier robot using wandering, but it's certainly possible to use wandering to start studying some of the problems these robots will face. And you'll even be able to do it with expressive and engaging platforms like Kuri (give our code a shot!), which wouldn't be up for the task otherwise. Even if wandering isn't a good fit for your specific use case, we hope you'll still carry the mind-set with you--that simple solutions can go a long way if you budget just a touch of human assistance into your system design. Nick Walker researches how humans and robots communicate with one another, with an eye toward future home and workplace robots. While he was a Ph.D. student at the University of Washington, he worked on both implicit communication--a robot's motion, for instance--and explicit communication, such as natural-language commands. Amal Nanavati does research in human-robot interaction and assistive technologies. His past projects have included developing a robotic arm to feed people with mobility impairments, developing a mobile robot to guide people who are blind, and cocreating speech-therapy games for and with a school for the deaf in India. Beyond his research at the University of Washington, Amal is an activist and executive board member of UAW 4121. Keep Reading |Show less Artificial IntelligenceTopicTypeTransportationSponsored Article AI Tool for COVID Monitoring Offers Solution for Urban Congestion Researchers at NYU have developed an AI solution that can leverage public video feeds to better inform decision makers Dexter Johnson Dexter Johnson is a contributing editor at IEEE Spectrum, with a focus on nanotechnology. 09 Jun 2022 7 min read C2SMART Center/New York University congestiontrafficsmart citiesNew York Cityc2smartCOVID-19nyu tandon This is a sponsored article brought to you by NYU's Tandon School of Engineering. In the midst of the COVID-19 pandemic, in 2020, many research groups sought an effective method to determine mobility patterns and crowd densities on the streets of major cities like New York City to give insight into the effectiveness of stay-at-home and social distancing strategies. But sending teams of researchers out into the streets to observe and tabulate these numbers would have involved putting those researchers at risk of exposure to the very infection the strategies were meant to curb. Researchers at New York University's (NYU) Connected Cities for Smart Mobility towards Accessible and Resilient Transportation (C2SMART) Center, a Tier 1 USDOT-funded University Transportation Center, developed a solution that not only eliminated the risk of infection to researchers, and which could easily be plugged into already existing public traffic camera feeds infrastructure, but also provided the most comprehensive data on crowd and traffic densities that had ever been compiled previously and cannot be easily detected by conventional traffic sensors. To accomplish this, C2SMART researchers leveraged publicly available New York City Department of Transportation (DOT) video feeds from the cover over 700 locations throughout New York City and applied a deep-learning, camera-based object detection method that enabled researchers to calculate pedestrian and traffic densities without ever needing to go out onto the streets. "Our idea was to take advantage of these DOT camera feeds and record them so we could better understand social distancing behavior of pedestrians," said Kaan Ozbay, Director of C2SMART and Professor at NYU. To do this, Ozbay and his team wrote a "crawler"--essentially a tool to index the video content automatically--to capture the low-quality images from the video feeds available on the internet. They then used an off-the-shelf deep-learning image-processing algorithm to process each frame of the video to learn what each frame contains: a bus, a car, a pedestrian, a bicycle, etc. The system also blurs out any identifying images such as faces, without impacting the effectiveness of the algorithm. The system developed by the NYU team can help inform decision-makers' understanding of a wide-range of questions ranging from crisis management responses such as social distancing behaviors to traffic congestion "This allows us to identify what is in the frame to determine the relationship between the objects in that frame," said Ozbay. "Then, based on a new method that obviates the need for actual in-situ referencing we devised, we're able to accurately measure the distance between people in the frame to see if they are too close to each other, or it's just too crowded." The easy thing would have been to just count how many people were within each frame. However, as Jingqin Gao, Senior Research Associate at NYU, explained, the reason they pursued an object detection method rather than mere enumeration is because the public feed is not continuous, with gaps lasting several seconds throughout the feed. "Instead of trying to very accurately count pedestrians crossing a line, we are trying to understand pedestrian density in urban environments, especially for those places that are typically crowded, like bus stops and crosswalks," said Gao. "We wanted to know whether they were changing their behavior amid the pandemic." Gao explained that the aim was to determine the pedestrian density and pedestrian social distancing patterns at scale and see how those patterns have changed since pre-COVID conditions, instead of tracking individual pedestrians. "For instance, we wanted to know if there was a change from pre-COVID when people were going out in the early morning for commuting purposes versus during the lockdown when they might be going out later in the afternoon," she added. "By exploring these different trends, we were trying to better understand if there are new patterns during and after the lockdown." Diagram showing camera data acquisition and pedestrian detection framework Camera data acquisition and pedestrian detection framework. C2SMART Center/New York University In general, these kinds of short count studies in traffic engineering only cover a few hours over several days, according to Ozbay. In those studies, people go out and collect data, and then they process it manually, even sometimes having to count cars by hand, for example. But this method would be impossible at the scale of C2SMART's work, Ozbay explained; in order to cover the hundreds of locations with 24-hour coverage over many months, the job has to be performed by an artificial intelligence (AI) algorithm instead of human or conventional traffic counters. There are complications that the AI has to overcome from each video feed: the locations are different, the camera angles and height are different, and they are subject to different lighting and positional factors. "It's not like the AI can learn just one intersection and automatically apply it to another one. It needs to learn each intersection individually," added Ozbay. To enable this AI solution, the C2SMART researchers started with an object detection model, namely You Only Look Once (YOLO), which is pre-trained using Microsoft's COCO data set. Gao explained that they also retrained and localized this object detection model with additional images and various customized post-processing filters to compensate for the low-resolution image produced by New York City DOT video feeds. \u200bScreenshot of the COVID-19 Data Dashboard Screenshot of the COVID-19 Data Dashboard created as part of the project. C2SMART Center/New York University While the off-the-shelf object detection model could work in this instance with some customization, when it came to measuring the distances between the objects, the NYU researchers had to develop a novel algorithm, which they refer to as a reference-free distance approximation algorithm. "If you're measuring something from an image, you may need some reference point," said Gao. "Historically, researchers might need to actually go to the site and measure the distance. But with our methodology, we can use the pixel size on the image of the person and the real height of that person to determine distance." While this project was inspired by the COVID-19 pandemic, the fast-moving nature of the disease precluded these findings from significantly impacting New York City's COVID policies. However, the project has produced a COVID-19 Data Dashboard and a video of how it was developed and operates is provided below. Ozbay explained that the project demonstrated to several city agencies that they were sitting on very valuable actionable data that could be used for many different purposes. "City agencies have approached us on several projects that are related to this one, but in a different context," said Ozbay. "Now we are working with New York's Department of Design and Construction (DDC) and the DOT to use the same kind of approach to analyze traffic around work zones and other key facilities such as intersections and on-street parking without them needing to actually go out to those locations." Ozbay notes that this initial project for COVID-19 has opened up possibilities for this kind of AI algorithm analysis of video feed data to be applied to a wide range of projects to provide critical understanding in a more efficient way. Diagram of object detection using cameras Potential applications of AI-powered object detection using city cameras. C2SMART Center/New York University Ozbay believes that much of the process NYU has developed can be handled internally by IT experts within their organization. For example, they should be able to handle the data acquisition and saving of it. But Ozbay believes that on the AI issues they will likely need to lean on experts within the academic or commercial realm to help them with this, since AI is always in a state of development, on a nearly monthly basis. "This solution will never become like Microsoft Word," said Ozbay. "It will always require some improvements and some changes and tweaking for the foreseeable future." Gao, who used to work for the DOT before taking on her current role, added that there's always a steady stream of commercial entities offering the DOT their product suites. "These commercial solutions frequently recommend buying and installing new cameras," she said. "What we have demonstrated here is that we can provide a solution based on current infrastructure." Based on his experience working with other cities and states throughout his career, Ozbay mentioned that most cities throughout the United States employ similar kinds of traffic camera systems used in New York City. "This method allows for cities throughout the country to provide a dual or triple usage of their existing infrastructure," said Ozbay. "There are a lot of opportunities to do this at a large scale for extended periods with little to no infrastructure cost." Eight images showing city scenes with cars and people that are identified by object recognition Example of other smart city use cases using this research framework: (a) detecting parking occupancy; (b) monitoring bus lane usage; (c) identifying illegal parking/double parking, (d) tracking and counting vehicles; and (e) using pedestrian density info at bus stops to assess transit demand. C2SMART Center/New York University Ozbay hopes the success of the technology will lead to other DOTs across the country learning of the technology and taking an interest in adopting it themselves. "If you can make it in New York, you can make it anywhere," he quipped. "We'll be happy to share with them our code and anything that may be of value to them from our experience." While the final product of this research may change the way traffic information has collected and used, it has also served as an important training tool for NYU students--not just postdoctoral researchers, but two undergraduate students at NYU's Tandon School of Engineering as well. "Our aim as an engineering school is not just to write papers, but to develop products that can be commercialized, and also to train the next generation of engineers on real projects where they can see how engineering contributes to and can help improve society," said Ozbay. Gao and Ozbay added that the two undergraduate students who worked on this project for two years are going on to graduate school to study along the lines of this project. "These students come to us without much knowledge, they become exposed to different research, and we let them pick what they are interested in. We train them very slowly," said Ozbay. "If they remain interested, they eventually become part of our research team." In future research, Ozbay envisions their work moving from just object recognition to building trajectories from these video feeds. If they are successful in this goal, Ozbay believes it has huge implications for applications like real-time traffic safety, an emerging area of research C2SMART is a major player. He added: "With trajectory building we can see the movement of vehicles in relation to each other as well as to pedestrians. This will not only help us identify risks in real-time but also establish and implement measures to mitigate those risks using advanced versions of methods we have already developed in the past." This research utilizes real-time public traffic camera information, which is publicly streamed by the New York City Department of Transportation (DOT) through a publicly open web site at https:// nyctmc.org/. Additional offline video data was provided by New York City DOT and New York City Department of Design and Construction (DDC) under the Memorandum of Understanding ("MOU") between the City of New York, acting by and through DDC and DOT, and C2SMART, a center within New York University. From Your Site Articles * Reimagining Public Buses - IEEE Spectrum > * NYU Researchers Paving New Path for Robotics - IEEE Spectrum > * NYU Researchers Pave the Way for Future Shared Mobility - IEEE ... > Related Articles Around the Web * AI - C2SMART Home > * C2SMART - C2SMART Home > * Connected Cities with Smart Transportation (C2SMART) | NYU ... > Keep Reading |Show less {"imageShortcodeIds":["29948834"]} Trending Stories The most-read stories on IEEE Spectrum right now ComputingTopicNewsTypeTelecommunications 150,000 Qubits Printed on a Chip 14 Jul 2022 4 min read multicolored data visualization TelecommunicationsTopicNewsType SpaceX and Dish's Super-Shady War for the World 15 Jul 2022 3 min read map of globe with blue orbs of light around Artificial IntelligenceTopicTypeFeature DALL-E 2's Failures Are the Most Interesting Thing About It 14 Jul 2022 10 min read Four photorealistic images of men. One is sitting at a desk in an office environment, one is shown drawing on a computer screen, and two are close-ups of faces. ComputingTopicNewsType RISC-V Guns for Raspberry Pi, Legacy Chips 13 Jul 2022 3 min read blue computer network with floating square white lines Consumer ElectronicsTopicTypeComputingNews Does MetaHuman's Digital Clone Cross the Uncanny Valley? 8h 4 min read A MetaHuman character open for edits in the MetaHuman software. BiomedicalTopicNewsType Nerve Coolers: Pain Relief Without Drugs 12 Jul 2022 3 min read Soft miniature implantable device used to help relieve pain BiomedicalTopicTypeSensorsNews A Breath Test for Monitoring Glucose Levels 15 Jul 2022 2 min read illustration of open mouth BiomedicalTopicTypeFeatureRobotics Today's Robotic Surgery Turns Surgical Trainees Into Spectators 05 Jul 2022 10 min read Photo of an operating room. On the left side of the image, two surgeons sit at consoles with their hands on controls. On the right side, a large white robot with four arms operates on a patient.