[HN Gopher] Our New Sam Audio Model Transforms Audio Editing
___________________________________________________________________
Our New Sam Audio Model Transforms Audio Editing
Author : ushakov
Score : 162 points
Date : 2025-12-16 20:25 UTC (7 days ago)
(HTM) web link (about.fb.com)
(TXT) w3m dump (about.fb.com)
| ajcp wrote:
| Given TikToks insane creator adoption rate is Meta developing
| these models to build out a content creation platform to compete?
| mgraczyk wrote:
| I doubt it, although it's possible these models will be used
| for creator tools, I believe the main idea is to use them for
| data labeling.
|
| At the time the first SAM was created, Meta was already
| spending over 2B/year on human labelers. Surely that number is
| higher now and research like this can dramatically increase
| data labeling volume
| embedding-shape wrote:
| > I doubt it, although it's possible these models will be
| used for creator tools, I believe the main idea is to use
| them for data labeling.
|
| How is creating 3D objects and characters (and something
| _resembling_ bones /armature but isn't) supposed to help with
| data labeling? As synthetic data for training other models,
| maybe, but seems like this new release is aimed at improving
| their own tooling for content creators, hard to deny this
| considering their demos.
|
| For the original SAM releases, I agree, that was probably the
| purpose. But these new ones that generate stuff and do
| effects and what not, clearly go beyond that initial scope.
| yjftsjthsd-h wrote:
| > Visual prompting: Click on the person or object in the video
| that's making a sound to isolate their audio.
|
| How does _that_ work? Correlating sound with movement?
| yodon wrote:
| Think about it conceptually:
|
| Could you watch a music video and say "that's the snare drum,
| that's the lead singer, keyboard, bass, that's the truck that's
| making the engine noise, that's the crowd that's cheering, oh
| and that's a jackhammer in the background"? So can AI.
|
| Could you point out who is lead guitar and who is rhythm
| guitar? So can AI.
| scarecrowbob wrote:
| I mean, sometimes I -mixing- a show and I couldn't tell you
| where a specific sound is coming from....
| yodon wrote:
| > sometimes I -mixing- a show and I couldn't tell you where
| a specific sound is coming from
|
| And in those situations it won't work. Is any of this
| really a surprise?
| recursive wrote:
| I thought about it. Still seems kind of pointless.
|
| That doesn't seem any better than typing "rhythm guitar". In
| fact, it seems worse and with extra steps. Sometimes the
| thing making the sound is not pictured. This thing is going
| to make me scrub through the video until the bass player is
| in frame instead of just typing "bass guitar". Then it will
| burn some power inferring that the thing I clicked on was a
| bass.
| yjftsjthsd-h wrote:
| To be fair, it's one of 3 ways to prompt
| janalsncm wrote:
| If it's anything like the original SAM, thousands of hours of
| annotator time.
|
| If I had to do it synthetically, take single subjects with a
| single sound and combine them together. Then train a model to
| separate them again.
| ac2u wrote:
| I wonder if the segmentation would work with a video of a
| ventriloquist and a dummy?
| m3kw9 wrote:
| Can I create a continuous "who farted" detector? Would be great
| at parties
| IncreasePosts wrote:
| Each person's unique fartprint is yet another way big tech will
| be tracking us
| BoorishBears wrote:
| They're already analyzing poop, what's a mic to go with your
| toilet camera?
|
| https://www.kohlerhealth.com/dekoda/
| samat wrote:
| And ads based on a fart! I guess you could throw in some
| spectrography for content aware ads too!! 'Hmm, I sense you
| like onions, you would love French soup in the restaurant
| downstairs today!'
| rmnclmnt wrote:
| Bighead is back! << Fart Alert >>!
| teeray wrote:
| I wonder if this would be nice for hearing aid users for reducing
| the background restaurant babble that overwhelms the people you
| want to hear.
| ks2048 wrote:
| I recently discovered Audacity includes plug-ins for audio
| separation that work great (e.g. split into vocals track and
| instruments track). The model it uses also originated at Facebook
| (demucs).
| tantalor wrote:
| Is "demucs" a pun on demux (demultiplexer)?
| ipsum2 wrote:
| Yes.
| TylerE wrote:
| Audacity is very very very far from state of the art in that
| respect.
| wellthisisgreat wrote:
| What's a good alternative ?
| 5-0 wrote:
| I suppose that depends on the use case.
|
| For mash-ups specifically, using yt-dlp to download music
| and split into stems with Demucs, using the UVR frontend,
| before importing into a DAW is effortless. The catch is
| that you can't expect to get OK-ish separation on anything
| other than vocals and "other", which really isn't a problem
| for mash-ups.
|
| https://github.com/Anjok07/ultimatevocalremovergui
| nartho wrote:
| IS there any DAW plugins that do that ?
| embedding-shape wrote:
| If you're already in the Ableton ecosystem, their newly
| released stem separation is actually very good, at least
| for the small amount of testing I've done so far. Much
| better than demucs, which shouldn't come as a surprise I
| suppose.
| TylerE wrote:
| I use RipX DAW personally. It very cleanly seperates
| vocals, guitar, bass, and drums.
| vhcr wrote:
| This new SAM model actually competes against SOTA models.
|
| https://www.reddit.com/r/LocalLLaMA/comments/1pp9w31/ama_wit.
| ..
| embedding-shape wrote:
| Their answer:
|
| > If you are interested in how well we do compared to
| demucs in particular, we can use the MUSDB18 dataset since
| that is the domain that demucs is trained to work well on.
| There our net win rate against demucs is ~17%, meaning we
| do perform better on the MUSDB18 test set. There are
| actually stronger competitors on both this domain and our
| "in-the-wild" instrument stem separation domain that we
| built for SAM Audio Bench, but we either match or beat all
| of the ones we tested (AudioShake, LalalAI, MoisesAI, etc.)
|
| So ~20% better than demucs, better than the ones they
| tested, but the acknowledge there are better models out
| there even today. So not sure "competes against SOTA
| models" is right, but "getting close to compete against
| SOTA models" might be more accurate.
| embedding-shape wrote:
| > for audio separation that work great
|
| What did you compare it to? Ableton recently launched a audio
| separation feature too, and probably the highest ROI on
| simple/useful/accurate so far I've tried, other solutions been
| lacking in one of the points before.
| yunwal wrote:
| This is hilariously bad with music. Like I can type in the most
| basic thing like "string instruments" which should theoretically
| be super easy to isolate. You can generally one-shot this using
| spectral analysis libraries. And it just totally fails.
| duped wrote:
| what in theory makes those "super easy" to isolate? Humans are
| terrible at this to begin with, it takes years to train one of
| them to do it mildly well. Computers are even worse - blind
| source separation and the cocktail party problem have been the
| white whale of audio DSP for decades (and only very recently
| did tools become passable).
| yunwal wrote:
| The fact that you can do it with spectral analysis libraries,
| no LLM required.
|
| This is much easier than source separation. It would be
| different if I were asking to isolate a violin from a viola
| or another violin, you'd have to get much more specific about
| the timbre of each instrument and potentially understand what
| each instruments part was.
|
| But a vibration made from a string makes a very unique wave
| that is easy to pick out in a file.
| duped wrote:
| Are you making this up? What spectral analysis libraries or
| tools?
|
| String instruments create similar harmonic series to horns,
| winds, and voice (because everything is a string in some
| dimension) and the major differences are in the spectral
| envelope, something that STFT tools are just ok at
| approximating because of the time/frequency tradeoff (aka:
| the uncertainty principle).
|
| This is a very hard problem "in theory" to me, and I'm just
| above casually versed in it.
| 613style wrote:
| He's not making it up and there's no reason for that
| tone. Strings are more straightforward to isolate
| compared to vocals/horns/etc because they produce a near-
| perfect harmonic series in parallel lines in a
| spectrogram. The time/frequency tradeoff exists, but it's
| less of a problem for strings because of their slow
| attack.
|
| You can look up HPSS and python libraries like Essentia
| and Librosa.
| IndySun wrote:
| Hmmm... was 'tone' a pun?
|
| Why mention a strings 'slow attack' as less of a problem?
| No isolation software considers this an easy route.
|
| Vocals are more effectively isolated by virtue of the
| fact they are unique sounding. Strings (and other sounds)
| are the similar in some ways but far more generic. All
| software out there indicates this, including the examples
| mentioned.
| mrob wrote:
| All wind instruments and all bowed string instruments
| produce a perfect harmonic series while emitting a steady
| tone. The most important difference between timbres of
| different instruments is in the attack, where inharmonic
| tones are also generated. Several old synths used this
| principle to greatly increase realism, by adding brief
| samples of attack transients to traditional subtractive
| synthesis, e.g.:
|
| https://en.wikipedia.org/wiki/Linear_arithmetic_synthesis
| dleeftink wrote:
| I might misremember, but iZotope RX and Melodyne were
| pretty useful in this regard.
| jb1991 wrote:
| If you look at the actual harmonics of a string and of
| horn, you will see how wrong you are. There is a reason
| why they sound different to the ear.
|
| It's because of this that you can have a relatively
| inexpensive synthesizer (not sample or PCM based) that
| does a crude job of mimicking these different instruments
| by just changing the harmonics.
| mrob wrote:
| There is one important difference between the harmonics
| of string and wind instruments: it's possible to build a
| wind instrument that suppresses (although not entirely
| eliminates) the even harmonics, e.g. a stopped organ
| pipe. If it sounds like a filtered square wave it's
| definitely a wind instrument. But if it sounds like a
| filtered sawtooth wave it could be either.
| coldtea wrote:
| > _what in theory makes those "super easy" to isolate? Humans
| are terrible at this to begin with,_
|
| Humans are amazing at it. You can discern the different
| instruments way better than any stem separating AI.
| photon_garden wrote:
| I had the same experience. It did okay at isolating vocals but
| everything else it failed or half-succeeded at.
| embedding-shape wrote:
| Like most models released for publicity rather than
| usefulness, they'll do great at benchmarks and single
| specific use cases, but no one seem to be able to release
| actually generalized models today.
| lomase wrote:
| Like everything AI you just have to lie a little and people
| whith 0 clue abot SOTA in audio will think this is amazing.
| hamza_q_ wrote:
| Use Demucs bruh https://github.com/adefossez/demucs
| yunwal wrote:
| Hilarious that this is maintained by facebook and yet SAM
| fails so badly
| throwaw12 wrote:
| This is super cool. Of course, it is possible to separate
| instrument sounds using specialized tools, but can't wait to see
| how people use this model for bunch of other use cases, where its
| not trivial to use those specialized tools:
|
| * remove background noise of tech products, but keep the nature
|
| * isolate the voice of a single person and feed into STT model to
| improve accuracy
|
| * isolating sound of events in games and many more
| 7734128 wrote:
| Finally a way to perhaps remove laugh tracks in the near future.
| sefrost wrote:
| There are examples on YouTube of laughter tracks being removed
| and there are lots of awkward pauses, so I think you'd need to
| edit the video to cut the pauses out entirely.
|
| - https://www.youtube.com/watch?v=23M3eKn1FN0
|
| - https://www.youtube.com/watch?v=DgKgXehYnnw
| embedding-shape wrote:
| Cutting the pauses will change the beats and rhythm of the
| scene, so you probably need to edit some of the voice lines
| and actual scenes too then. In the end, if you're not
| interested in the original performance and work, you might as
| well read the script instead and imagine it however you want,
| read it at the pace you want and so on.
| vintermann wrote:
| And have a video model render an entirely new version for
| you, I guess.
| samuell wrote:
| I tried this to try to extract some speech from an audio track
| with heavy noise from wind (filmed out on a windy sea shore
| without mic windscreen), and the result unfortunately was less
| intelligible than the original.
|
| I got much better results, though still not perfect, with the
| voice isolator in ElevenLabs.
| AkshatJ27 wrote:
| You can try it out in the playground:
| https://aidemos.meta.com/segment-anything/gallery/ There seem to
| be many more fun little demos by meta here like automatic video
| masking, making 3d models from 2d images, etc.
| theflyestpilot wrote:
| sample anything model?
| Oras wrote:
| To try: https://aidemos.meta.com/segment-anything/editor/segment-
| aud...
|
| Github: https://github.com/facebookresearch/sam-audio
|
| I quite like adding effects such as making the isolated speech
| studio-quality or broadcast-ready.
| keepamovin wrote:
| FB has been a pioneer in voice and audio, somehow. A couple of
| years ago FB-Research had a little repo on GitHub that was the
| best noise-removal / voice-isolation out there. I wanted to use
| it in Wisprnote and politely emailed the authors. Never heard
| back (that's okay), but I was so impressed with the perceptual
| quality and "wind removal" (so hard).
| websiteapi wrote:
| I wonder if it works for speaker diarization out of the box. I've
| found that open source speaker diarization that doesn't require a
| lot of tweaking is basically non-existent.
| hamza_q_ wrote:
| Yeah I was frustrated by slow and hard to use OSS diarization
| too; recently released a library to address that, check it out:
| https://github.com/narcotic-sh/senko
|
| Also https://zanshin.sh, if you'd like speaker diarization when
| watching YouTube videos
| websiteapi wrote:
| looks interesting. will check it out.
___________________________________________________________________
(page generated 2025-12-23 23:01 UTC)