https://news.mit.edu/2014/algorithm-recovers-speech-from-vibrations-0804 Skip to content | Massachusetts Institute of Technology MIT Top Menu| * Education * Research * Innovation * Admissions + Aid * Campus Life * News * Alumni * About MIT * More | Search MIT Search websites, locations, and people [ ] See More Results Suggestions or feedback? MIT News | Massachusetts Institute of Technology Subscribe to MIT News newsletter Browse Enter keywords to search for news articles: [ ] Submit Browse By Topics View All - Explore: * Machine learning * Social justice * Startups * Black holes * Classes and programs Departments View All - Explore: * Aeronautics and Astronautics * Brain and Cognitive Sciences * Architecture * Political Science * Mechanical Engineering Centers, Labs, & Programs View All - Explore: * Abdul Latif Jameel Poverty Action Lab (J-PAL) * Picower Institute for Learning and Memory * Media Lab * Lincoln Laboratory Schools * School of Architecture + Planning * School of Engineering * School of Humanities, Arts, and Social Sciences * Sloan School of Management * School of Science * MIT Schwarzman College of Computing View all news coverage of MIT in the media - Subscribe to MIT newsletter - Close Breadcrumb 1. MIT News 2. Extracting audio from visual information Extracting audio from visual information Algorithm recovers speech from the vibrations of a potato-chip bag filmed through soundproof glass. Watch Video Larry Hardesty | MIT News Office Publication Date: August 4, 2014 Press Inquiries Press Contact: Abby Abazorius Email: abbya@mit.edu Phone: 617-253-2709 MIT News Office Media Download [MIT-Visual-Microphone] | Download Image Credits: Image: Christine Daniloff/MIT *Terms of Use: Images for download on the MIT News office website are made available to non-commercial entities, press and the general public under a Creative Commons Attribution Non-Commercial No Derivatives license. You may not alter the images provided, other than to crop them to size. A credit line must be used when reproducing images; if one is not provided below, credit the images to "MIT." Close [placeholder--news-article--image-gallery] Credits: Image: Christine Daniloff/MIT Previous image Next image Researchers at MIT, Microsoft, and Adobe have developed an algorithm that can reconstruct an audio signal by analyzing minute vibrations of objects depicted in video. In one set of experiments, they were able to recover intelligible speech from the vibrations of a potato-chip bag photographed from 15 feet away through soundproof glass. In other experiments, they extracted useful audio signals from videos of aluminum foil, the surface of a glass of water, and even the leaves of a potted plant. The researchers will present their findings in a paper at this year's Siggraph, the premier computer graphics conference. "When sound hits an object, it causes the object to vibrate," says Abe Davis, a graduate student in electrical engineering and computer science at MIT and first author on the new paper. "The motion of this vibration creates a very subtle visual signal that's usually invisible to the naked eye. People didn't realize that this information was there." Joining Davis on the Siggraph paper are Fredo Durand and Bill Freeman, both MIT professors of computer science and engineering; Neal Wadhwa, a graduate student in Freeman's group; Michael Rubinstein of Microsoft Research, who did his PhD with Freeman; and Gautham Mysore of Adobe Research. Reconstructing audio from video requires that the frequency of the video samples -- the number of frames of video captured per second -- be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That's much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second. Commodity hardware In other experiments, however, they used an ordinary digital camera. Because of a quirk in the design of most cameras' sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn't as faithful as that with the high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers' voices, their identities. The researchers' technique has obvious applications in law enforcement and forensics, but Davis is more enthusiastic about the possibility of what he describes as a "new kind of imaging." "We're recovering sounds from objects," he says. "That gives us a lot of information about the sound that's going on around the object, but it also gives us a lot of information about the object itself, because different objects are going to respond to sound in different ways." In ongoing work, the researchers have begun trying to determine material and structural properties of objects from their visible response to short bursts of sound. Video thumbnail Play video Watch how MIT researchers extract audio from the vibrations of a plant, potato-chip bag, and other objects. In the experiments reported in the Siggraph paper, the researchers also measured the mechanical properties of the objects they were filming and determined that the motions they were measuring were about a tenth of micrometer. That corresponds to five thousandths of a pixel in a close-up image, but from the change of a single pixel's color value over time, it's possible to infer motions smaller than a pixel. Suppose, for instance, that an image has a clear boundary between two regions: Everything on one side of the boundary is blue; everything on the other is red. But at the boundary itself, the camera's sensor receives both red and blue light, so it averages them out to produce purple. If, over successive frames of video, the blue region encroaches into the red region -- even less than the width of a pixel -- the purple will grow slightly bluer. That color shift contains information about the degree of encroachment. Putting it together Some boundaries in an image are fuzzier than a single pixel in width, however. So the researchers borrowed a technique from earlier work on algorithms that amplify minuscule variations in video, making visible previously undetectable motions: the breathing of an infant in the neonatal ward of a hospital, or the pulse in a subject's wrist. That technique passes successive frames of video through a battery of image filters, which are used to measure fluctuations, such as the changing color values at boundaries, at several different orientations -- say, horizontal, vertical, and diagonal -- and several different scales. The researchers developed an algorithm that combines the output of the filters to infer the motions of an object as a whole when it's struck by sound waves. Different edges of the object may be moving in different directions, so the algorithm first aligns all the measurements so that they won't cancel each other out. And it gives greater weight to measurements made at very distinct edges -- clear boundaries between different color values. The researchers also produced a variation on the algorithm for analyzing conventional video. The sensor of a digital camera consists of an array of photodetectors -- millions of them, even in commodity devices. As it turns out, it's less expensive to design the sensor hardware so that it reads off the measurements of one row of photodetectors at a time. Ordinarily, that's not a problem, but with fast-moving objects, it can lead to odd visual artifacts. An object -- say, the rotor of a helicopter -- may actually move detectably between the reading of one row and the reading of the next. For Davis and his colleagues, this bug is a feature. Slight distortions of the edges of objects in conventional video, though invisible to the naked eye, contain information about the objects' high-frequency vibration. And that information is enough to yield a murky but potentially useful audio signal. "This is new and refreshing. It's the kind of stuff that no other group would do right now," says Alexei Efros, an associate professor of electrical engineering and computer science at the University of California at Berkeley. "We're scientists, and sometimes we watch these movies, like James Bond, and we think, 'This is Hollywood theatrics. It's not possible to do that. This is ridiculous.' And suddenly, there you have it. This is totally out of some Hollywood thriller. You know that the killer has admitted his guilt because there's surveillance footage of his potato chip bag vibrating." Efros agrees that the characterization of material properties could be a fruitful application of the technology. But, he adds, "I'm sure there will be applications that nobody will expect. I think the hallmark of good science is when you do something just because it's cool and then somebody turns around and uses it for something you never imagined. It's really nice to have this type of creative stuff." Share this news article on: * Twitter * Facebook * LinkedIn * Reddit * Print Press Mentions BBC News In this video, BBC Click's LJ Rich explores how researchers at MIT CSAIL have devised a system that can reconstruct sound from a video recording. "I think what's really different about this technology is that it provides you with a way to image this information," says graduate student Abe Davis. Full story via BBC News - Forbes Steven Rosenbaum highlights PhD student Abe Davis' TED talk in a piece for Forbes. Rosenbaum writes that Davis "has co-created the world's most improbable audio instrument." Full story via Forbes - The Washington Post Rachel Feltman writes for The Washington Post about how MIT researchers have developed new technology that can amplify microscopic movements invisible to the human eye. "MIT researchers recently published a study in which they extracted intelligible audio by analyzing the movements of a nearby bag of chips," Feltman writes. Full story via The Washington Post - CNN Heather Kelly of CNN reports on how MIT researchers have developed a new technique to recreate audio from silent video. "We showed that we can determine pretty reliably the gender of a speaker from low-quality sound we managed to recover from a tissue box," says Dr. Michael Rubinstein. Full story via CNN - PBS NewsHour Colleen Shalby reports for the PBS NewHour on the "visual microphone" developed by MIT researchers that can detect and reconstruct audio by analyzing the sound waves traveling through objects. Full story via PBS NewsHour - Wired "Researchers have developed an algorithm that can use visual signals from videos to reconstruct sound and have used it to recover intelligible speech from a video," writes Katie Collins for Wired about an algorithm developed by a team of MIT researchers that can derive speech from material vibrations. Full story via Wired - ABC News Alyssa Newcomb of ABC News reports on how MIT researchers have developed a new method that can uncover intelligible audio by videotaping everyday objects and translating the sound vibrations back into intelligible sound. Full story via ABC News - Time Magazine Time reporter Nolan Feeney writes about how researchers from MIT have developed a new technique to extract intelligible audio of speech by "videotaping and analyzing the tiny vibrations of objects." Full story via Time Magazine - Bloomberg Businessweek Bloomberg Businessweek reporter Drake Bennett writes about how MIT researchers have developed a technique for extracting audio by analyzing the sound vibrations traveling through objects. Bennett reports that the researchers found that sound waves could be detected even when using cell phone camera sensors. Full story via Bloomberg Businessweek - NPR NPR's Melissa Block examines the new MIT algorithm that can translate visual information into sound. Abe Davis explains that by analyzing sound waves traveling through an object, "you can start to filter out some of that noise and you can actually recover the sound that produced that motion." Full story via NPR - New Scientist Hal Hodson of New Scientist reports on the new algorithm developed by MIT researchers that can turn visual images into sound. "We were able to recover intelligible speech from maybe 15 feet away, from a bag of chips behind soundproof glass," explains Abe Davis, a graduate student at MIT. Full story via New Scientist - Popular Science In a piece for Popular Science, Douglas Main writes on the new technique developed by MIT researchers that can reconstruct speech from visual information. The researchers showed that, "an impressive amount of information about the audio (although not its content) could also be recorded with a regular DSLR that films at 60 frames per second." Full story via Popular Science - The Washington Post Rachel Feltman of The Washington Post examines the new MIT algorithm that can reconstruct sound by examining the visual vibrations of sound waves. "This is a new dimension to how you can image objects," explains graduate student Abe Davis. Full story via The Washington Post - BetaBoston Michael Morisy writes for BetaBoston about an algorithm developed by MIT researchers that can recreate speech by analyzing material vibrations. "The sound re-creation technique typically required cameras shooting at thousands of frames per second," writes Morisy. Full story via BetaBoston - Slate Writing for Slate, Elliot Hannon reports on the new technology developed by MIT researchers that allows audio to be extracted from visual information by processing the vibrations of sound waves as they move through objects. Full story via Slate - Previous item Next item Related Links * Paper: "The Visual Microphone: Passive Recovery of Sound from Video" * The Visual Microphone * Abe Davis * Computer Graphics Group * Computer Science and Artificial Intelligence Laboratory * School of Engineering Related Topics * Computer vision * Imaging * School of Engineering * Electrical Engineering & Computer Science (eecs) * Computer Science and Artificial Intelligence Laboratory (CSAIL) * Research Related Articles Digital illustration featuring a black-and-white photo of a young woman's head and shoulders, superimposed with a network of interconnected nodes running through her head and neck Seeing the human pulse In these frames of video, a new algorithm amplifies the almost imperceptible change in skin color caused by the pumping of the blood. Researchers amplify variations in video, making the invisible visible [placeholder--frontpage--featured-news] Faster computer graphics Previous item Next item More MIT News Group of 10 people posing on the grass in front of the MIT Dome MeepCon comes to MIT Photonics community gathers to further develop open-source software for electromagnetic simulations spanning a broad range of applications. Read full story - George Lordos tinkers with a device On the front lines of space innovation Aeronautics and astronautics PhD candidate George Lordos develops technologies to enable life on Mars. Read full story - Photo of Wes Harris wearing a suit and a bow tie. Wesley Harris elected vice president of National Academy of Engineering The distinguished professor of aeronautics and astronautics will continue to champion diversity in engineering in the new role. Read full story - a donation recipient holds up their mobile bank account Fighting poverty with direct cash payments The alumni-founded GiveDirectly has delivered over $500 million in cash to impoverished people, letting recipients decide how best to meet their needs. Read full story - Graphic showing cell-to-cell adhesins. In a series of panels, multiple polygonal shapes are clustered together with their sides aligned. Living LEGOs Mathematical modeling speeds up the process of programming bacterial systems to self-assemble into desired 2D shapes. Read full story - Photo of Jiahao Dong sitting at a desk with two monitors in Jaramillo lab. A large piece of lab equipment labeled "TriboIndenter" is in the foreground. MIT team reports giant response of semiconductors to light The materials' stiffness increases up to 40 percent, in a reversible effect, the researchers report in a study that also explains the phenomenon's atomic origins. Read full story - * More news on MIT News homepage - More about MIT News at Massachusetts Institute of Technology This website is managed by the MIT News Office, part of the MIT Office of Communications. News by Schools/College: * School of Architecture and Planning * School of Engineering * School of Humanities, Arts, and Social Sciences * MIT Sloan School of Management * School of Science * MIT Schwarzman College of Computing Resources: * About the MIT News Office * MIT News Press Center * Terms of Use * Press Inquiries * Filming Guidelines * RSS Feeds Tools: * Subscribe to MIT Daily/Weekly * Subscribe to press releases * Submit campus news * Guidelines for campus news contributors Massachusetts Institute of Technology MIT Top Level Links: * Education * Research * Innovation * Admissions + Aid * Campus Life * News * Alumni * About MIT * Join us in building a better world. Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA Recommended Links: * Visit * Map (opens in new window) * Events (opens in new window) * People (opens in new window) * Careers (opens in new window) * Contact * Privacy * Accessibility * + Social Media Hub + MIT on Twitter + MIT on Facebook + MIT on YouTube + MIT on Instagram