https://github.com/huggingface/distil-whisper Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} huggingface / distil-whisper Public * Notifications * Fork 13 * Star 677 677 stars 13 forks Activity Star Notifications * Code * Issues 4 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights huggingface/distil-whisper This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/h] Use Git or checkout with SVN using the web URL. [gh repo clone huggin] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit Git stats * 18 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time Distil_Whisper.pdf Update paper November 1, 2023 10:46 README.md Update README.md November 1, 2023 15:13 View code Distil-Whisper 1. Usage 2. Why use Distil-Whisper? !?[?] 3. Approach [?] 4. Acknowledgements README.md Distil-Whisper [Paper] [Models] [wandb] Distil-Whisper is a distilled version of Whisper that is 6 times faster, 49% smaller, and performs within 1% WER on out-of-distribution evaluation sets. Model Link distil-medium.en To be published on November 2nd distil-large-v2 To be published on November 2nd 1. Usage The Distil-Whisper checkpoints will be released on November 2nd with a direct Transformers integration. Instructions for running inference will be provided here: from transformers import WhisperForConditionalGeneration ... 2. Why use Distil-Whisper? !?[?] Distil-Whisper is designed to be a drop-in replacement for Whisper on English speech recognition. Here are 4 reasons for making the switch to Distil-Whisper: 1. Faster inference: 6 times faster inference speed, while performing to within 1% WER of Whisper on out-of-distribution audio: [68747470733a2f2f68756767696e67666163652e636f2f64617461736574732f64697374696c2d776869] 2. Robustness to noise: demonstrated by strong WER performance at low signal-to-noise ratios: [68747470733a2f2f68756767696e67666163652e636f2f64617461736574732f64697374696c2d776869] 3. Robustness to hallucinations: quantified by 1.3 times fewer repeated 5-gram word duplicates (5-Dup.) and 2.1% lower insertion error rate (IER) than Whisper: [68747470733a2f2f68756767696e67666163652e636f2f64617461736574732f64697374696c2d776869] 4. Designed for speculative decoding: Distil-Whisper can be used as an assistant model to Whisper, giving 2 times faster inference speed while mathematically ensuring the same outputs as the Whisper model. 3. Approach [?] To distill Whisper, we copy the entire encoder module and freeze it during training. We copy only two decoder layers, which are initialised from the first and last decoder layers from Whisper. All other decoder layers from Whisper are discarded. [68747470733a2f2f68756767696e67666163652e636f2f64617461736574732f64697374696c2d776869] Distil-Whisper is trained on a knowledge distillation objective. Specifically, it is trained to minimise the KL divergence between the distilled model and the Whisper model, as well as the cross-entropy loss on pseudo-labelled audio data. We train Distil-Whisper on a total of 22k hours of pseudo-labelled audio data, spanning 10 domains with over 18k speakers: [68747470733a2f2f68756767696e67666163652e636f2f64617461736574732f64697374696c2d776869] This diverse audio dataset is paramount to ensuring robustness of Distil-Whisper to different datasets and domains. In addition, we use a WER filter to discard pseudo-labels where Whisper mis-transcribes or hallucinates. This greatly improves WER performance of the downstream distilled model. For full details on the distillation set-up and evaluation results, refer to the Distil-Whisper paper. 4. Acknowledgements * OpenAI for the original Whisper model and codebase * Google's TPU Research Cloud (TRC) programme for Cloud TPU v4s About No description or website provided. Topics audio speech-recognition whisper Resources Readme Activity Stars 677 stars Watchers 45 watching Forks 13 forks Report repository Releases No releases published Packages 0 No packages published Contributors 4 * @sanchit-gandhi sanchit-gandhi Sanchit Gandhi * @patrickvonplaten patrickvonplaten Patrick von Platen * @IbrahimAmin1 IbrahimAmin1 Ibrahim Amin * @amrrs amrrs amrrs Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.