https://github.com/JoinMusic/fish Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + GitHub Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions By size + Enterprise + Teams + Startups By industry + Healthcare + Financial services + Manufacturing By use case + CI/CD & Automation + DevOps + DevSecOps * Resources Topics + AI + DevOps + Security + Software Development Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} JoinMusic / fish Public * Notifications You must be signed in to change notification settings * Fork 5 * Star 135 YouTube video to chords, lyrics, beat and melody. 135 stars 5 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 1 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights JoinMusic/fish This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master BranchesTags Go to file Code Folders and files Name Name Last commit message Last commit date Latest commit History 1 Commit cover cover image image python python README.md README.md View all files Repository files navigation * README Fish YouTube video to chords, lyrics, beat and melody. A transformer-based hybrid multimodal model, various transformer models address different problems in the field of music information retrieval, these models generate corresponding information dependencies that mutually influence each other. An AI-powered multimodal project focused on music, generate chords, beats, lyrics, melody, and tabs for any song. The online experience, See the site here [tnn] [model] U-Net network model for audio source separation, Pitch-Net, Beat-Net, Chord-Net and Segment-Net based on the transformer model. Apart from establishing the correlation between the frequency and time, the most important aspect is to establish the mutual influence between different networks. The entire AI-powered process is implemented in aitabs.py, while the various network structure models can be referenced in the models folder. Note: U-Net and Segment-Net use the stft spectrum of audio as input. Beat-Net uses three spectrograms of drums, bass, and other instruments as input,Chord-Net uses one spectrogram of the background music. Features * Chord, music chord detection, including major, minor, 7, maj7, min7, 6, m6, sus2, sus4, 5, and inverted chords. Determining the key of a song. * Beat, music beat, downbeat detection and tempo (BPM) tracking * Pitch, tracking the pitch of the melody in the vocal track. * Music Structure, music segment boundaries and labels, include intro, verse, chorus, bridge and etc. * Lyrics, music lyrics recognition and automatic lyrics to audio alignment, use ASR (whisper) to recognize the lyrics of the vocal track. The alignment of lyrics and audio is achieved through fine-tuning the wav2vec2 pre-training model. Currently, it supports dozens of languages, including English, Spanish, Portuguese, Russian, Japanese, Korean, Arabic, Chinese, and more. * AI Tabs, Generate playable sheet music, including chord charts and six-line staves, using chords, beats, music structure information, lyrics, rhythm, etc. It supports editing functionalities for chords, rhythm, and lyrics. * Other, audio source separation, speed adjustment, pitch shifting, etc. For more AI-powered feature experiences, see the website: Cover Using a combination of audio STFT, MFCC, and chroma features, with a Transformer model for timbre feature modeling and high-level abstraction, this approach can maximize the avoidance of overfitting and underfitting problems compared to using a single feature, and has better generalization capabilities. With a small amount of data and minimal training, it can achieve better results. The online experience, See the site here [net1] [net2] The model begins by processing the audio signal through a U-Net, which isolates the vocal track. The vocal track is then simultaneously fed into PitchNet and HuBERT (Wav2Vec2). PitchNet is responsible for extracting pitch features, while HuBERT captures detailed features of the vocals. The core of the model is CombineNet, which receives features from the Features module. This module consists of three spectrograms: STFT, MFCC, and Chroma, each extracting different aspects of the audio. These features are enhanced by the TimbreBlock before being passed to the Encoder. During this process, noise is introduced via STFT transformation and combined with the features before entering the Encoder for processing. The processed features are then passed to the Decoder, where they are combined with the previous features to generate the final audio output. CombineNet is based on an encoder-decoder architecture and is trained to generate a mask that is used to extract and replace the timbre, ultimately producing the final output audio. The entire AI-powered process is implemented in run.py, while the various network structure models can be referenced in the models folder. Demo The results of training on a 1-minute speech of Donald Trump are as follows: Train 10 epoch(Hozier's Too Sweet) Train 100 epoch(Hozier's Too Sweet) te10_20s.webm te100_20s.webm You can experience creating your own voice online, See the site here About YouTube video to chords, lyrics, beat and melody. Resources Readme Activity Custom properties Stars 135 stars Watchers 1 watching Forks 5 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 100.0% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.