[HN Gopher] Show HN: Free AI-based music demixing in the browser
___________________________________________________________________
Show HN: Free AI-based music demixing in the browser
Hi all, I've spent some time working on music demixing or music
source separation algorithms, which take in a mixed song and output
estimates of isolated components (e.g. vocals, drums, bass, other).
I took a popular PyTorch model with good performance (Open-Unmix,
UMX-L weights), reimplemented the inference steps in C++, and
compiled it to WebAssembly for a free client-side music demixer.
Author : sevagh
Score : 116 points
Date : 2023-07-13 12:03 UTC (10 hours ago)
(HTM) web link (sevag.xyz)
(TXT) w3m dump (sevag.xyz)
| henriquecm8 wrote:
| I tried 2 different files, a .flac and a .mp3, both around 3
| minutes and half, and the demixing stopped at 7.5% on both files.
| henriquecm8 wrote:
| I tried on Firefox now, it also stopped at 7.5%, my previous
| attempts were on Edge Chromium.
| sevagh wrote:
| What does the developer console show? After the first few
| layers (STFT -> FC1 -> BN1) comes the LSTM which is a much
| slower step (and could look like the site is stopped).
| wackget wrote:
| Didn't work on Chrome or Brave or Firefox for me.
|
| Console warning:
|
| The AudioContext was not allowed to start. It must be resumed (or
| created) after a user gesture on the page. https://goo.gl/7K7WLu
|
| Download Weights button also did nothing.
| code51 wrote:
| Retry. It'll work. (Brave tested)
| zzzeek wrote:
| wow i always thought Spleeter was the only game in town. I've
| been not that satisfied with its results.
|
| is there a command line version of this tool ?
| noman-land wrote:
| What about Melodyne?
| humanistbot wrote:
| I couldn't get this to work on Firefox 115, but could in Chrome.
| Does this require Chrome?
| sevagh wrote:
| I have been testing and using it on Firefox 113. I wonder if
| it's related to the maximum memory of WASM. I compile it with
| `-s MAXIMUM_MEMORY=4GB`, which is supported by Chrome but maybe
| not Firefox.
|
| Does the developer console say something like "Aborted" or give
| a memory error in Firefox? If your clip is big enough that it
| uses >2GB but <4GB of memory, that could explain why it works
| in Chrome.
| humanistbot wrote:
| Nevermind, got it to work after disabling adblocker
| chaosprint wrote:
| Great work. Thanks for sharing the code!
| nanidin wrote:
| I met a dj few weeks ago that had vocals, drums, and bass all on
| knobs that could do this on the fly in realtime. I bet it works
| in a similar way under the hood.
|
| I believe the program he was using was djPROAI[0].
|
| [0] https://www.algoriddim.com/djay-pro-mac
| rasjani wrote:
| Feature is called "stems" and its also available on most major
| dj apps like Serato and Rekordbox
| tiahura wrote:
| Cool.
|
| What about the ability to do separate out background noise? I'm
| thinking of a project like this where the Beatles Live album had
| screaming fans mixed down?
|
| https://www.wired.com/2017/03/remastering-one-beatles-live-a...
|
| "It doesn't exist as a software program that is easy to use,"
| Clarke says. "It's a lot of Matlab, more like a research tool.
| There's no graphical front end where you can just load a piece of
| audio up, paint a track, and extract the audio. I write manual
| scripts, which I then put into the engine to process."
|
| There are tons of recordings of live performances that could use
| a little AI TLC.
| kbouck wrote:
| Would love to see something like this auto-mix song transitions
| in playlists to be more DJ-like.
| holoduke wrote:
| In a few years from now DJs will have buttons on their pads to
| extract voices, beats, melodies etc realtime. Could result into
| an interesting new style of club music.
| kbouck wrote:
| this already exists in the djay app (paid add-on feature).
| even runs on a smartphone! performs about the same as the app
| this thread is about.
| kbouck wrote:
| _> re: "realtime"_
|
| it separates the components of tracks that it can download
| (and process), not of a live audio feed
| [deleted]
| crtasm wrote:
| DJs have this now, the track still needs to be processed but
| that can happen in advance or be done on-the-fly (VirtualDJ +
| a GPU takes about 10s to process a track and can do so
| seamlessly while the track is playing).
| jareklupinski wrote:
| I started down this road, and have amassed a small collection
| of files made up of just the first ten and last ten seconds of
| demixed tracks of every song in my library
|
| I hoped to put together a system that compares the backs to the
| fronts and lists the output to find cool transitions, but I
| have no idea how to actually "grade" the similarities
|
| Beyond basic BPM matching on the drums tracks, nothing I've
| tried has made for anything really compelling (sounds random...
| :( )
| tristanc wrote:
| Interesting, I attempted to do the same as you but stopped
| just shy of BPM matching.
|
| However I did get sound similarity working using an audio
| tagging neural net [1]. I chopped off the first and last 15
| seconds of every song in my collection and ran them all
| through this analysis which produces a ~520 dimensional
| vector. I then targeted specific endings I wanted to match
| and used Euclidian distance to find the closest matching song
| beginning.
|
| YMMV but I thought it actually worked pretty well, I just
| never got to automating the BPM matching. I can try to look
| for my old script if you're interested :)
|
| [1] https://github.com/fschmid56/EfficientAT
| sevagh wrote:
| OK, so, tangentially related: I tried to do something once -
| I took small chunks of songs generated by SampleRNN in an
| attempt to stitch together the ones that sounded the most
| similar to create a much longer track.
|
| The script [1] uses Essentia Chromaprint [2] to "grade" the
| similarity of audio tracks, and combine the ones with the
| closest chromaprint. No crossfade or BPM matching, just yolo
| concatenation.
|
| I have a track on Soundcloud which uses the above technique
| (mashing together short generated clips by their chromagram),
| trained on Cannibal Corpse [3]
|
| 1: https://github.com/sevagh/1000sharks.xyz/blob/master/sampl
| er...
|
| 2: https://essentia.upf.edu/reference/std_Chromaprinter.html
|
| 3: https://soundcloud.com/user-167126026/1000sharks-domainal-
| sk...
| kbouck wrote:
| my idea was to let an existing app like djay do the beat
| matching (which it does really well!) and demixing (less
| well), and then make a custom app that would act like a midi
| controller, adjusting the vocals, beats, melody while songs
| are transitioning, perhaps working off of cues bookmarked in
| the songs.
| uguuo_o wrote:
| I tried to use it but I had some issues as others in the thread.
|
| I have tried many sources and method over the years and settled
| on spleeter [0]. Works well even for 10+ minute songs, varying
| styles from flamenco to heavy metal.
|
| [0] https://github.com/deezer/spleeter
| potatoman22 wrote:
| I like your site. WebAssembly has opened up client side ML
| inference and it's a really convenient way for devs to serve
| certain models now. Out of curiosity, why did you reimplement
| inference in C to compile to WASM instead of using a python to
| WASM solution?
| sevagh wrote:
| Good question! So, I wasn't even thinking about WASM to begin
| with. When I saw llama.cpp and whisper.cpp on the front page of
| HN, I found the idea exciting - instead of neural networks
| being magic, I wanted to copy the ggml idea of parsing the
| PyTorch weights file myself and rewriting the inference code in
| a lower-level language than Python (or, it's even more accurate
| to say PyTorch, since there is so much matrix heavy lifting
| e.g. broadcasting or reshaping that is done for you
| automatically).
|
| That's when I wrote umx.cpp [1] (which is what this site is
| based on).
|
| On an unrelated project, a friend of mine mentioned WASM, and
| as I looked into it a bit more I thought trying to compile
| umx.cpp to WASM would be a great idea, since I only use Eigen
| (which is a header-only library that only depends on std).
|
| 1: https://github.com/sevagh/umx.cpp
| comex wrote:
| Haven't tried this demo, but in my experience these open-source
| models that split music into four components work absolutely
| fantastically. Not quite perfectly -- if you remove vocals, the
| remaining track may have faint echos of vocals - but astoundingly
| well compared to the state of the art just 5 years ago or so.
|
| However... what if you want more than four components? What if
| you want to split a complex arrangement into a separate component
| for each individual instrument? Does anyone know of any
| interesting research in this area?
| gnabgib wrote:
| Gave it a shot - as the webpage warns, it does take a bit off
| time. The results are indeed impressive, although the
| separation isn't perfect.. each division (drum/bass/vocals) has
| slight echos of each other (bass line occasionally includes a
| slight muffled vocal, vocals include some snare, hi-hat, and
| strings). It's a great starting point though.
| sevagh wrote:
| Demucs [1], one of the leading/SOTA systems, has an
| experimental 6-source model, `htdemucs_6s`, which adds piano
| and guitar:
|
| >We are also releasing an experimental 6 sources model, that
| adds a guitar and piano source. Quick testing seems to show
| okay quality for guitar, but a lot of bleeding and artifacts
| for the piano source.
|
| I also believe Audioshake [2] (a company in the space) is doing
| guitar separation as well.
|
| 1: https://github.com/facebookresearch/demucs 2:
| https://www.audioshake.ai/
| xnx wrote:
| Nice job. I love this trend of fully client-side WASM utilities.
| pcthrowaway wrote:
| Very cool, thank you for sharing, and making it open source! I
| love this!
___________________________________________________________________
(page generated 2023-07-13 23:01 UTC)