https://spectrum.ieee.org/large-language-models-math [ ] IEEE.orgIEEE Xplore Digital LibraryIEEE StandardsMore Sites Sign InJoin IEEE AI Language Models Are Struggling to "Get" Math Share FOR THE TECHNOLOGY INSIDER Search: [ ] Explore by topic AerospaceArtificial IntelligenceBiomedicalComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensors TelecommunicationsTransportation IEEE Spectrum FOR THE TECHNOLOGY INSIDER Topics AerospaceArtificial IntelligenceBiomedicalComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensors TelecommunicationsTransportation Sections FeaturesNewsOpinionCareersDIYThe Big PictureEngineering Resources More Special ReportsCollectionsExplainersPodcastsVideosNewslettersTop Programming LanguagesRobots Guide For IEEE Members Current IssueMagazine ArchiveThe InstituteTI Archive For IEEE Members Current IssueMagazine ArchiveThe InstituteTI Archive IEEE Spectrum About UsContact UsReprints & PermissionsAdvertising Follow IEEE Spectrum Support IEEE Spectrum IEEE Spectrum is the flagship publication of the IEEE -- the world's largest professional organization devoted to engineering and applied sciences. Our articles, podcasts, and infographics inform our readers about developments in technology, engineering, and science. Join IEEE Subscribe About IEEEContact & SupportAccessibilityNondiscrimination PolicyTerms IEEE Privacy Policy (c) Copyright 2022 IEEE -- All rights reserved. A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. IEEE websites place cookies on your device to give you the best user experience. By using our websites, you agree to the placement of these cookies. To learn more, read our Privacy Policy. view privacy policy accept & close Enjoy more free content and benefits by creating an account Saving articles to read later requires an IEEE Spectrum account The Institute content is only available for members Downloading full PDF issues is exclusive for IEEE Members Access to Spectrum's Digital Edition is exclusive for IEEE Members Following topics is a feature exclusive for IEEE Members Adding your response to an article requires an IEEE Spectrum account Create an account to access more content and features on IEEE Spectrum, including the ability to save articles to read later, download Spectrum Collections, and participate in conversations with readers and editors. For more exclusive content and features, consider Joining IEEE. Join the world's largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum's articles, archives, PDF downloads, and other benefits. Learn more - CREATE AN ACCOUNTSIGN IN JOIN IEEESIGN IN Close Access Thousands of Articles -- Completely Free Create an account and get exclusive content and features: Save articles, download collections, and talk to tech insiders -- all free! For full access and benefits, join IEEE as a paying member. CREATE AN ACCOUNTSIGN IN Artificial IntelligenceTopicTypeComputingNews AI Language Models Are Struggling to "Get" Math Should this be telling us something? Dan Garisto 7h 4 min read The words Alice has five more balls than Bob, who has two balls after he gives four to Charlie. How many balls does Alice have? in blue and distorted iStockphoto/IEEE Spectrum large language modelsmachine learningartificial intelligence mathematics If computers are good at anything, they are good at math. So it may come as a surprise that after much struggling, top machine-learning researchers have recently made breakthroughs in teaching computers math. Over the past year, researchers from the University of California, Berkeley, OpenAI, and Google have made leaps and bounds in teaching language models--algorithms similar to GPT-3 and DALL-E 2--basic math concepts. However, until very recently, language models regularly failed to solve even simple word problems, such as "Alice has five more balls than Bob, who has two balls after he gives four to Charlie. How many balls does Alice have?" "When we say computers are very good at math, they're very good at things that are quite specific," says Guy Gur-Ari, a machine-learning expert at Google. Computers are good at arithmetic--plugging numbers in and calculating is child's play. But outside of formal structures, computers struggle. "I think there's this notion that humans doing math have some rigid reasoning system--that there's a sharp distinction between knowing something and not knowing something." --Ethan Dyer, Google Solving word problems, or "quantitative reasoning," is deceptively tricky because it requires a robustness and rigor that many other problems don't. If any step during the process goes wrong, the answer will be wrong. While DALL-E's impressive images may leave out fingers or create strange eyes, mistakes are more glaring when it comes to math. "When multiplying really large numbers together...they'll forget to carry somewhere and be off by one," says Vineet Kosaraju, a machine-learning expert at OpenAI. Other mistakes made by language models are less human, such as misinterpreting 10 as a 1 and a 0, not 10. "We work on math because we find it independently very interesting," says Karl Cobbe, a machine-learning expert at OpenAI. But as Gur-Ari puts it, if it's good at math, "it's probably also good at solving many other useful problems." As machine-learning models are trained on larger samples of data, they tend to grow more robust and make fewer mistakes. But scaling up seems to go only so far with quantitative reasoning; researchers realized that the mistakes language models make seemed to require a more targeted approach. Last year, two different teams of researchers, at UC Berkeley and OpenAI, released two data sets, MATH and GSM8K, respectively, which contain thousands of math problems across geometry, algebra, precalculus, and more. "We basically wanted to see if it was a problem with data sets," says Steven Basart, a researcher at the Center for AI Safety who worked on MATH. Language models were known to be bad at word problems--but how bad were they, and could they be fixed by introducing better formatted, bigger data sets? The MATH group found just how challenging quantitative reasoning is for top-of-the-line language models, which scored less than 7 percent. (A human grad student scored 40 percent, while a math olympiad champ scored 90 percent.) Models attacking GSM8K problems, which had easier grade-school-level problems, reached about 20 percent accuracy. The OpenAI researchers used two main techniques: fine-tuning and verification. In fine-tuning, researchers take a pretrained language model that includes irrelevant information (Wikipedia articles on zambonis, the dictionary entry for "gusto," and the like) and then show the model, Clockwork Orange-style, only the relevant information (math problems). Verification, on the other hand, is more of a review session. "The model gets to see a lot of examples of its own mistakes, which is really valuable," Cobbe says. At the time, OpenAI predicted a model would need to be trained on 100 times more data to reach 80 percent accuracy on GSM8K. But in June, Google's Minerva announced 78 percent accuracy with minimal scaling upwards. "It's ahead of any of the trends that we were expecting," Cobbe says. Basart agrees. "That's shocking. I thought it would take longer," he says. Minerva uses Google's own language model, Pathways Language Model (PaLM), which is fine-tuned on scientific papers from the arXiv online preprint server and other sources with formatted math. Two other strategies helped Minerva. In "chain-of-thought prompting," Minerva was required to break down larger problems into more palatable chunks. The model also used majority voting--instead of being asked for one answer, it was asked to solve the problem 100 times. Of those answers, Minerva picked the most common answer. The gains from these new strategies were enormous. Minerva shot up to 50 percent accuracy on MATH and nearly 80 percent accuracy on GSM8K, as well as the MMLU, a more general set of STEM questions that included chemistry and biology. When Minerva was asked to redo a random sample of slightly tweaked questions, it performed just as well, suggesting that its capabilities were not from mere memorization. What Minerva knows--or doesn't know--about math is fuzzier. Unlike proof assistants, which come with built-in structure, Minerva and other language models have no formal structure. They can have strange, messy reasoning and still arrive at the right answer. As numbers grow larger, the language models' accuracy falters, something that would never happen on a TI-84. "Just how smart is it--or isn't it?" asks Cobbe. Though models like Minerva might arrive at the same answer as a human, the actual process they're following could be wildly different. On the other hand, chain-of-thought prompting is familiar to any human student who's been asked to "show your work." "I think there's this notion that humans doing math have some rigid reasoning system--that there's a sharp distinction between knowing something and not knowing something," says Ethan Dyer, a machine-learning expert at Google. But humans give inconsistent answers, make errors, and fail to apply core concepts, too. The borders, at this frontier of machine learning, are blurred. From Your Site Articles * Meta's Challenge to OpenAI--Give Away a Massive Language Model - > * Andrew Ng: Unbiggen AI - IEEE Spectrum > Related Articles Around the Web * The race to understand the thrilling, dangerous world of language AI ... > * How Large Language Models Will Transform Science, Society, and AI > large language modelsmachine learningartificial intelligence mathematics Dan Garisto Dan Garisto is a freelance science journalist who covers physics and other physical sciences. His work has appeared in Scientific American , Physics, Symmetry, Undark, and other outlets. The Conversation (0) portrait of a man in a black blazer and a blue button up shirt against a white background The InstituteTopicTypeCareersProfile A Serial Entrepreneur Shares Lessons Learned on His Road to Success 11 Oct 2022 4 min read battery icon with lightning bolts on sides Artificial IntelligenceTopicEnergyTypeRoboticsNews Robots and AI Could Optimize Lithium-Ion Batteries 11 Oct 2022 2 min read illustration of a mountain scene with different types of energy sources EnergyTopicNewsType Geothermal May Beat Batteries for Energy Storage 10 Oct 2022 3 min read Related Stories BiomedicalTopicTypeNews AI Can Offer Insight Into Who Responds to Antidepressants Artificial IntelligenceTopicTypeComputingNews Machine Learning Shaking Up Hard Sciences, Too ComputingTopicArtificial IntelligenceTypeNews AI's Grandmaster Status Overshadows Chess Scandal Artificial IntelligenceTopicMagazineTypeFeatureSeptember 2022 Will AI Steal Submarines' Stealth? Better detection will make the oceans transparent--and perhaps doom mutually assured destruction Natasha Bajema 16 Jul 2022 11 min read A photo of a submarine in the water under a partly cloudy sky. The Virginia-class fast attack submarine USS Virginia cruises through the Mediterranean in 2010. Back then, it could effectively disappear just by diving. U.S. Navy Submarines are valued primarily for their ability to hide. The assurance that submarines would likely survive the first missile strike in a nuclear war and thus be able to respond by launching missiles in a second strike is key to the strategy of deterrence known as mutually assured destruction. Any new technology that might render the oceans effectively transparent, making it trivial to spot lurking submarines, could thus undermine the peace of the world. For nearly a century, naval engineers have striven to develop ever-faster, ever-quieter submarines. But they have worked just as hard at advancing a wide array of radar, sonar, and other technologies designed to detect, target, and eliminate enemy submarines. The balance seemed to turn with the emergence of nuclear-powered submarines in the early 1960s. In a 2015 study for the Center for Strategic and Budgetary Assessment, Bryan Clark, a naval specialist now at the Hudson Institute, noted that the ability of these boats to remain submerged for long periods of time made them "nearly impossible to find with radar and active sonar." But even these stealthy submarines produce subtle, very-low-frequency noises that can be picked up from far away by networks of acoustic hydrophone arrays mounted to the seafloor. And now the game of submarine hide-and-seek may be approaching the point at which submarines can no longer elude detection and simply disappear. It may come as early as 2050, according to a recent study by the National Security College of the Australian National University, in Canberra. This timing is particularly significant because the enormous costs required to design and build a submarine are meant to be spread out over at least 60 years. A submarine that goes into service today should still be in service in 2082. Nuclear-powered submarines, such as the Virginia-class fast-attack submarine, each cost roughly US $2.8 billion, according to the U.S. Congressional Budget Office. And that's just the purchase price; the total life cycle cost for the new Columbia-class ballistic-missile submarine is estimated to exceed $395 billion. The twin problems of detecting submarines of rival countries and protecting one's own submarines from detection are enormous, and the technical details are closely guarded secrets. Many naval experts are speculating about sensing technologies that could be used in concert with modern AI methodologies to neutralize a submarine's stealth. Rose Gottemoeller, former deputy secretary general of NATO, warns that "the stealth of submarines will be difficult to sustain, as sensing of all kinds, in multiple spectra, in and out of the water becomes more ubiquitous." And the ongoing contest between stealth and detection is becoming increasingly volatile as these new technologies threaten to overturn the balance. We have new ways to find submarines Today's sensing technologies for detecting submarines are moving beyond merely hearing submarines to pinpointing their position through a variety of non-acoustic techniques. Submarines can now be detected by the tiny amounts of radiation and chemicals they emit, by slight disturbances in the Earth's magnetic fields, and by reflected light from laser or LED pulses. All these methods seek to detect anomalies in the natural environment, as represented in sophisticated models of baseline conditions that have been developed within the last decade, thanks in part to Moore's Law advances in computing power. [svg]Airborne laser-based sensors can detect submarines lurking near the surface.IEEE Spectrum According to experts at the Center for Strategic and International Studies, in Washington, D.C., two methods offer particular promise. Lidar sensors transmit laser pulses through the water to produce highly accurate 3D scans of objects. Magnetic anomaly detection (MAD) instruments monitor the Earth's magnetic fields and can detect subtle disturbances caused by the metal hull of a submerged submarine. Both sensors have drawbacks. MAD works only at low altitudes or underwater. It is often not sensitive enough to pick out the disturbances caused by submarines from among the many other subtle shifts in electromagnetic fields under the ocean. Lidar has better range and resolution and can be installed on satellites, but it consumes a lot of power--a standard automotive unit with a range of several hundred meters can burn 25 watts. Lidar is also prohibitively expensive, especially when operated in space. In 2018, NASA launched a satellite with laser imaging technology to monitor changes in Earth's surface--notably changes in the patterns on the ocean's surface; the satellite cost more than $1 billion. Indeed, where you place the sensors is crucial. Underwater sensor arrays won't put an end to submarine stealth by themselves. Retired Rear Adm. John Gower, former submarine commander for the Royal Navy of the United Kingdom, notes that sensors "need to be placed somewhere free from being trolled or fished, free from seismic activity, and close to locations from which they can be monitored and to which they can transmit collected data. That severely limits the options available." One way to get around the need for precise placement is to make the sensors mobile. Underwater drone swarms can do just that, which is why some experts have proposed them as the ultimate antisubmarine capability. Clark, for instance, notes that such drones now have enhanced computing power and batteries that can last for two weeks between charges. The U.S. Navy is working on a drone that could run for 90 days. Drones are also now equipped with the chemical, optical, and geomagnetic sensors mentioned earlier. Networked underwater drones, perhaps working in conjunction with airborne drones, may be useful for not only detecting submarines but also destroying them, which is why several militaries are investing heavily in them. A photo of a plane on a runway.A U.S. Navy P-8 Poseidon aircraft, equipped to detect submarines, awaits refueling in Okinawa, Japan, in 2020. U.S.Navy For example, the Chinese Navy has invested in a fishlike undersea drone known as Robo-Shark, which was designed specifically for hunting submarines. Meanwhile, the U.S. Navy is developing the Low-Cost Unmanned Aerial Vehicle Swarming Technology, for conducting surveillance missions. Each Locust drone weighs about 6 kilograms, costs $15,000, and can be outfitted with MAD sensors; it can skim low over the ocean's surface to detect signals under the water. Militaries study the drone option because it might work. Then again, it very well might not. A photo of a robotic shark. Robo-Shark, a 2.2-meter-long submersible made by Boya Gongdao Robot Technology, of Beijing, is said to be capable of underwater surveillance and unspecified antisubmarine operations. The company says that the robot moves at up to 5 meters per second (10 knots) by using a three-joint structure to wave the caudal fin, making less noise than a standard propeller would. robosea.org Gower considers underwater drones to be "the least likely innovation to make a difference in the decline of submarine stealth." A navy would need a lot of drones, data rates are exceedingly slow, and a drone's transmission range is short. Drones are also noisy and extremely easy to detect. "Not to mention that controlling thousands of underwater drones far exceeds current technological capabilities," he adds. Gower says it could be possible "to use drones and sonar networks together in choke points to detect submarine patrols." Among the strategically important submarine patrol choke points are the exit routes on either side of Ireland, for U.K. submarines; those around the islands of Hainan and Taiwan, for Chinese submarines; in the Barents or Kuril Island chain, for Russian submarines; and the Straits of Juan de Fuca, for U.S. Pacific submarines. On the other hand, he notes, "They could be monitored and removed since they would be close to sovereign territories. As such, the challenges would likely outweigh the gains." Gower believes a more powerful means of submarine detection lies in the "persistent coverage of the Earth's surface by commercial satellites," which he says "represents the most substantial shift in our detection capabilities compared to the past." More than 2,800 of these satellites are already in orbit. Governments once dominated space because the cost of building and launching satellites was so great. These days, much cheaper satellite technology is available, and private companies are launching constellations of tens to thousands of satellites that can work together to image every bit of the Earth's surface. They are outfitted with a wide range of sensing technologies, including synthetic aperture radar (SAR), which scans a scene down below while moving over a great distance, providing results like those you'd get from an extremely long antenna. Since these satellite constellations view the same locations multiple times per day, they can capture small changes in activity. Experts have known for decades about the possibility of detecting submarines with SAR based on the wake patterns they form as they move through the ocean. To detect such patterns, known as Bernoulli humps and Kelvin wakes, the U.S. Navy has invested in the AN/APS-154 Advanced Airborne Sensor, developed by Raytheon. The aircraft-mounted radar is designed to operate at low altitudes and appears to be equipped with high-resolution SAR and lidar sensors. Commercial satellites equipped with SAR and other imaging instruments are now reaching resolutions that can compete with those of government satellites and offer access to customers at extremely affordable rates. In other words, there's lots of relevant, unclassified data available for tracking submarines, and the volume is growing exponentially. One day this trend will matter. But not just yet. Jeffrey Lewis, director of the East Asia Nonproliferation Program at the James Martin Center for Nonproliferation Studies, regularly uses satellite imagery in his work to track nuclear developments. But tracking submarines is a different matter. "Even though this is a commercially available technology, we still don't see submarines in real time today," Lewis says. The day when commercial satellite imagery reduces the stealth of submarines may well come, says Gower, but "we're not there yet. Even if you locate a submarine in real time, 10 minutes later, it's very hard to find again." Artificial intelligence coordinates other sub-detecting tech Though these new sensing methods have the potential to make submarines more visible, no one of them can do the job on its own. What might make them work together is the master technology of our time: artificial intelligence. "When we see today's potential of ubiquitous sensing capabilities combined with the power of big-data analysis," Gottemoeller says, "it's only natural to ask the question: Is it now finally possible?" She began her career in the 1970s, when the U.S. Navy was already worried about Soviet submarine-detection technology. Submarines can now be detected by the tiny amounts of radiation and chemicals they emit, by slight disturbances in the Earth's magnetic fields, and by reflected light from laser or LED pulses. Unlike traditional software, which must be programmed in advance, the machine-learning strategy used here, called deep learning, can find patterns in data without outside help. Just this past year, DeepMind's AlphaFold program achieved a breakthrough in predicting how amino acids fold into proteins, making it possible for scientists to identify the structure of 98.5 percent of human proteins. Earlier work in games, notably Go and chess, showed that deep learning could outdo the best of the old software techniques, even when running on hardware that was no faster. For AI to work in submarine detection, several technical challenges must be overcome. The first challenge is to train the algorithm, which involves acquiring massive volumes and varieties of sensor data from persistent satellite coverage of the ocean's surface as well as regular underwater collection in strategic locations. Using such data, the AI can establish a detailed model of baseline conditions, then feed new data into the model to find subtle anomalies. Such automated sleuthing is what's likeliest to detect the presence of a submarine anywhere in the ocean and predict locations based on past transit patterns. The second challenge is collecting, transmitting, and processing the masses of data in real time. That task would require a lot more computing power than we now have, both in fixed and on mobile collection platforms. But even today's technology can start to put the various pieces of the technical puzzle together. Nuclear deterrence depends on the ability of submarines to hide For some years to come, the vastness of the ocean will continue to protect the stealth of submarines. But the very prospect of greater ocean transparency has implications for global security. Concealed submarines bearing ballistic missiles provide the threat of retaliation against a first nuclear strike. What if that changes? "We take for granted the degree to which we rely upon having a significant portion of our forces exist in an essentially invulnerable position," Lewis says. Even if new developments did not reduce submarine stealth by much, the mere perception of such a reduction could undermine strategic stability. A gray unmanned helicopter, notably lacking a cockpit or any kind of window, is shown hovering against a clear, blue sky. It carries a downward-poinging sensor under its nose. A Northrop Grumman MQ-8C, an uncrewed helicopter, has recently been deployed by the U.S. Navy in the Indo-Pacific area for use in surveillance. In the future, it will also be used for antisubmarine operations. Northrop Grumman Gottemoeller warns that "any perception that nuclear-armed submarines have become more targetable will lead to questions about the survivability of second-strike forces. Consequently, countries are going to do everything they can to counter any such vulnerability." Experts disagree on the irreversibility of ocean transparency. Because any technological breakthroughs will not be implemented overnight, "nations should have ample time to develop countermeasures [that] cancel out any improved detection capabilities," says Matt Korda, senior research associate at the Federation of American Scientists, in Washington, D.C. However, Roger Bradbury and eight colleagues at the National Security College of the Australian National University disagree, claiming that any technical ability to counter detection technologies will start to decline by 2050. Korda also points out that ocean transparency, to the extent that it occurs, "will not affect countries equally. And that raises some interesting questions." For example, U.S. nuclear-powered submarines are "the quietest on the planet. They are virtually undetectable. Even if submarines become more visible in general, this may have zero meaningful effect on U.S. submarines' survivability." Sylvia Mishra, a new-tech nuclear officer at the European Leadership Network, a London-based think tank, says she is "more concerned about the overall problem of ambiguity under the sea." Until recently, she says, movement under the oceans was the purview of governments. Now, though, there's a growing industry presence under the sea. For example, companies are laying many underwater fiber-optic communication cables, Mishra says, "which may lead to greater congestion of underwater inspection vehicles, and the possibility for confusion." A large, cylindrical vehicle is shown just as it has been lowered below the surface of the water, suspended by two green cables.A Snakehead, a large underwater drone designed to be launched and recovered by U.S. Navy nuclear-powered submarines, is shown at its christening ceremony in Narragansett Bay in Newport, R.I.U.S. Navy Confusion might come from the fact that drones, unlike surface ships, do not bear a country flag, and therefore their ownership may be unclear. This uncertainty, coupled with the possibility that the drones could also carry lethal payloads, increases the risk that a naval force might view an innocuous commercial drone as hostile. "Any actions that hold the strategic assets of adversaries at risk may produce new touch points for conflict and exacerbate the risk of war," says Mishra. Given the strategic importance of submarine stealth, Gower asks, "Why would any country want to detect and track submarines? It's only something you'd do if you want to make a nuclear-armed power nervous." Even in the Cold War, when the United States and the U.K. routinely tracked Soviet ballistic-missile submarines, they did so only because they knew their activities would go undetected--that is, without risking escalation. Gower postulates that this was dangerously arrogant: "To actively track second-strike nuclear forces is about as escalatory as you might imagine." "All nuclear-armed states place a great value on their second-strike forces," Gottemoeller says. If greater ocean transparency produces new risks to their survivability, real or perceived, she says, countries may respond in two ways: build up their nuclear forces further and take new measures to protect and defend them, producing a new arms race; or else keep the number of nuclear weapons limited and find other ways to bolster their viability. Ultimately, such considerations have not dampened the enthusiasm of certain governments for acquiring submarines. In September 2021 the Australian government announced an enhanced trilateral partnership with the United States and the United Kingdom. The new deal, known as AUKUS, will provide Australia with up to eight nuclear-powered submarines with the most coveted propulsion technology in the world. However, it could be at least 20 years before the Royal Australian Navy can deploy the first of its new subs. [svg]The Boeing Orca, the largest underwater drone in the U.S. Navy's inventory, was christened in April, in Huntington Beach, Calif. The craft is designed, among other things, for use in antisubmarine warfare. The Boeing Company As part of its plans for nuclear modernization, the United States has started replacing its entire fleet of 14 Ohio-class ballistic-missile submarines with new Columbia-class boats. The replacement program is projected to cost more than $128 billion for acquisition and $267 billion over their full life cycles. U.S. government officials and experts justify the steep cost of these submarines with their critical role in bolstering nuclear deterrence through their perceived invulnerability. To protect the stealth of submarines, Mishra says, "There is a need for creative thinking. One possibility is exploring a code of conduct for the employment of emerging technologies for surveillance missions." There are precedents for such cooperation. During the Cold War, the United States and the Soviet Union set up a secure communications system--a hotline--to help prevent a misunderstanding from snowballing into a disaster. The two countries also developed a body of rules and procedures, such as never to launch a missile along a potentially threatening trajectory. Nuclear powers could agree to exercise similar restraint in the detection of submarines. The stealthy submarine isn't gone; it still has years of life left. That gives us ample time to find new ways to keep the peace. From Your Site Articles * World's Largest Swarm of Miniature Robot Submarines - IEEE ... > * Submarines - IEEE Spectrum > * Scientists Explore Underwater Quantum Links for Submarines ... > * DARPA's Self-Driving Submarine Hunter Steers Like a Human ... > Related Articles Around the Web * The Capacity of the Navy's Shipyards to Maintain Its Submarines ... > * Attack Submarines - SSN > United States Navy > Displayy-FactFiles > * Ballistic Missile Submarines | Commander, Submarine Force, U.S. ... > * Fleet Ballistic Missile Submarines - SSBN > United States Navy ... > Keep Reading |Show less {"imageShortcodeIds":["30133857"]}