https://blog.gingerbeardman.com/2023/08/13/automatically-classifying-the-content-of-sound-files-using-ml/

[?]I  Get Info
by Matt Sephton

Automatically classifying the content of sound files using ML

2023-08-13  *  3 mins read
#audio #machinelearning #script #sfx

Following on from yesterday's extraction of old sound effects, I
quickly realised I needed an easier way to search them as they came
out of Director as unlabelled, numbered files. I can use QuickLook or
a media player to quickly audition them, but how could I easily find
the sample that contains the sound of running water or a horse
trotting?

I wondered if there was a way of using Machine Learning (ML) to
automatically categorise sounds. It seemed like something that should
be possible, especially given the recent explosion in "AI" (really:
ML) tools. I quickly found Google's AudioSet, which sounded like the
perfect dataset:

    AudioSet consists of an expanding ontology of 632 audio event
    classes and a collection of 2,084,320 human-labeled 10-second
    sound clips drawn from YouTube videos. The ontology is specified
    as a hierarchical graph of event categories, covering a wide
    range of human and animal sounds, musical instruments and genres,
    and common everyday environmental sounds.

But the data set is only one half of the solution. You need to use
the dataset to create a model and then run that model against your
own data to get the required results. Thankfully, I found YAMNet:

    YAMNet is a deep net that predicts ~521 audio event classes from
    the AudioSet-YouTube corpus it was trained on.

I guess YAMNet is tracking behind AudioSet in terms of total
categories, but it is good enough for me.

Let's go

I used the script described in this tutorial as a starting point. I'm
not a regular python user but using pip to install tensorflow, along
with any other missing imports, and after that it ...just worked.

Getting your files in order

According to the documentation all sound files need to be at a sample
rate of 16000Hz. After getting some calssification results of
"Silence", I realised they also need to be 16-bit resolution. So I
ran a quick sox command to create compliant copies of all my sounds.
I'll delete these when I'm done. Notice how I decided to trim sounds
to a maximum length of 3 seconds. This helps speed things up and most
sounds can still be recognised with such a short starting section.

find . -iname "*.wav" -exec sox {} -c 1 -r 16000 -b 16 {}_16k.wav trim 0 00:03 \;

Optimisation

Running the classifier works at about real-time, a few seconds per
sound, but I noticed that it was leaving a lot of my CPU unused. This
struck me as a prime candidate for parallelisation, which is pretty
easy on the command line. I used the parallel command to scale up the
classification to use all 10-cores of the M1 Pro CPU in my 2021
MacBook Pro.

find . -iname "*.wav" -exec parallel python3 classify.py {} ::: {} \+

As I type my computer is making short order of the task, whilst
remaining perfectly responsive, if a little warm. Final speed for me
is one sound every ~0.85 seconds.

Python Script

Creating my SFX Library

Soundly is a sort of iTunes for sound effects. It's an app that
enables easy, automatic organisation of files, quick searching of
metadata, painless playback/auditioning, non-destructive edits, and
simple exporting of the final sounds. The free version allows a local
library of 10,000 files which is more than enough for my usage.

As you add your local folder of files it allows you to import a
(semicolon-separated) .csv file containing additional metadata. It's
here that I point it to the file that was generated by the
classifier. The categories are imported as the description of the
sound, and are able to be searched. Perfect!

--
Comments: @gingerbeardman
 
Previous Post
Extracting sounds from Macromedia Director files
---------------------------------------------------------------------
1bit  1bitwoodblocks  3ds  60fps  amiga  animation  apple  appletv  
appstore  arcade  archaeology  art  atarist  audio  automation  
bandkun  basiliskii  becker  birthday  cars  controllers  credits  
critique  dailydriver  deneba  dreamcast  ds  dsiware  electronics  
emulation  extension  famicom  gamedev  gameplay  gba  golf  graphics
  guide  hack  hanafuda  hardware  history  hypercard  
internetarchive  ios  itchio  japanese  koei  koikoi  macintosh  
macos  mahjong  maps  marchintosh  markdown  medium  megadrive  
midjourney  mmm  mod  modes  msx  music  nes  nintendo  nintendo64  
nintendods  palmos  patreon  pc  physics  piece  playdate  
playstation  polarium  preservation  proxy  recommendations  
reverseengineering  review  rss  save  scans  script  sfx  snes  
software  solitaire  sonyreader  sparrowsolitaire  speedcam  system7 
thoruyamamoto  translation  tv  ultrapaint  ux  videogame  videogames
  wii  wiiware  windows  workflow  wwdc