https://github.com/ggerganov/whisper.cpp/releases/tag/v1.4.0 Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} ggerganov / whisper.cpp Public * Notifications * Fork 1.5k * Star 17.2k * Code * Issues 161 * Pull requests 13 * Discussions * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Security * Insights 1. Releases 2. v1.4.0 v1.4.0 Latest Latest Compare Choose a tag to compare [ ] Could not load tags Nothing to show {{ refName }} default (*) View all tags @ggerganov ggerganov released this 30 Apr 16:56 * 3 commits to master since this release v1.4.0 fa8dbdc This commit was signed with the committer's verified signature. [1991296] ggerganov Georgi Gerganov GPG key ID: 449E073F9DC10735 Learn about vigilant mode. Overview This is a new major release adding integer quantization and partial GPU (NVIDIA) support Integer quantization This allows the ggml Whisper models to be converted from the default 16-bit floating point weights to 4, 5 or 8 bit integer weights. The resulting quantized models are smaller in disk size and memory usage and can be processed faster on some architectures. The transcription quality is degraded to some extend - not quantified at the moment. * Supported quantization modes: Q4_0, Q4_1, Q4_2, Q5_0, Q5_1, Q8_0 * Implementation details: #540 * Usage instructions: README * All WASM examples now support Q5 quantized models: https:// whisper.ggerganov.com Here is a quantitative evaluation of the different quantization modes applied to the LLaMA and RWKV large language models. These results can give an impression about the expected quality, size and speed for quantized Whisper models: LLaMA quantization (measured on M1 Pro) Model Measure F16 Q4_0 Q4_1 Q4_2 Q5_0 Q5_1 Q8_0 7B perplexity 5.9565 6.2103 6.1286 6.1698 6.0139 5.9934 5.9571 7B file size 13.0G 4.0G 4.8G 4.0G 4.4G 4.8G 7.1G 7B ms/tok @ 4th 128 56 61 84 91 95 75 7B ms/tok @ 8th 128 47 55 48 53 59 75 7B bits/weight 16.0 5.0 6.0 5.0 5.5 6.0 9.0 13B perplexity 5.2455 5.3748 5.3471 5.3433 5.2768 5.2582 5.2458 13B file size 25.0G 7.6G 9.1G 7.6G 8.4G 9.1G 14G 13B ms/tok @ 4th 239 104 113 160 176 185 141 13B ms/tok @ 8th 240 85 99 97 108 117 147 13B bits/weight 16.0 5.0 6.0 5.0 5.5 6.0 9.0 ref: https://github.com/ggerganov/llama.cpp#quantization RWKV quantization Format Perplexity (169M) Latency, ms (1.5B) File size, GB (1.5B) Q4_0 17.507 76 1.53 Q4_1 17.187 72 1.68 Q4_2 17.060 85 1.53 Q5_0 16.194 78 1.60 Q5_1 15.851 81 1.68 Q8_0 15.652 89 2.13 FP16 15.623 117 2.82 FP32 15.623 198 5.64 ref: ggerganov/ggml#89 (comment) This feature is possible thanks to the many contributions in the llama.cpp project: https://github.com/users/ggerganov/projects/2 GPU support via cuBLAS Using cuBLAS results mainly in improved Encoder inference speed. I haven't done proper timings, but one can expect at least 2-3 times faster Encoder evaluation with modern NVIDIA GPU cards compared to CPU-only processing. Feel free to post your Encoder benchmarks in issue #89. * Implementation details: #834 * Usage instructions: README This is another feature made possible by the llama.cpp project. Special recognition to @slaren for putting almost all of this work together --------------------------------------------------------------------- This release remains in "beta" stage as I haven't verified that everything works as expected. What's Changed * Updated escape_double_quotes() Function by @tauseefmohammed2 in # 776 * examples : add missing #include by @pH5 in #798 * Flush upon finishing inference by @tarasglek in #811 * Escape quotes in csv output by @laytan in #815 * C++11style by @wuyudi in #768 * Optionally allow a Core ML build of Whisper to work with or without Core ML models by @Canis-UK in #812 * add some tips about in the readme of the android project folder by @Zolliner in #816 * whisper: Use correct seek_end when offset is used by @ThijsRay in #833 * ggml : fix 32-bit ARM NEON by @ggerganov in #836 * Add CUDA support via cuBLAS by @ggerganov in #834 * Integer quantisation support by @ggerganov in #540 New Contributors * @tauseefmohammed2 made their first contribution in #776 * @pH5 made their first contribution in #798 * @tarasglek made their first contribution in #811 * @laytan made their first contribution in #815 * @wuyudi made their first contribution in #768 * @Canis-UK made their first contribution in #812 * @Zolliner made their first contribution in #816 * @ThijsRay made their first contribution in #833 Full Changelog: v1.3.0...v1.4.0 Contributors * @pH5 * @tarasglek * @ggerganov * @slaren * @ThijsRay * @Canis-UK * @laytan * @Zolliner * @wuyudi * @tauseefmohammed2 pH5, tarasglek, and 8 other contributors Assets 6 11 egorsmkv, geimist, lin72h, gricowijaya, vadi2, demkom58, mediaexplorer74, sv2, vaionicle, admp, and pixlflip reacted with thumbs up emoji 6 aguglie, lin72h, tim-gromeyer, ThijsRay, araffin, and artob reacted with rocket emoji All reactions * 11 reactions * 6 reactions 16 people reacted 0 Join discussion Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.