https://github.com/Const-me/Whisper Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + For + Enterprise + Teams + Startups + Education + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} Const-me / Whisper Public * Notifications * Fork 2 * Star 22 High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model License MPL-2.0 license 22 stars 2 forks Star Notifications * Code * Issues 1 * Pull requests 0 * Actions * Security * Insights More * Code * Issues * Pull requests * Actions * Security * Insights Const-me/Whisper This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 2 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/C] Use Git or checkout with SVN using the web URL. [gh repo clone Const-] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @Const-me Const-me Sample audio clips, with performance stats ... 5f11a82 Jan 16, 2023 Sample audio clips, with performance stats 5f11a82 Git stats * 19 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time ComLightLib Source codes Jan 16, 2023 ComputeShaders Source codes Jan 16, 2023 Examples Minor, documentation Jan 16, 2023 SampleClips Sample audio clips, with performance stats Jan 16, 2023 Tools Minor, documentation Jan 16, 2023 Whisper Comment Jan 16, 2023 WhisperNet Release notes Jan 16, 2023 .gitignore Source codes Jan 16, 2023 LICENSE Initial commit Jan 7, 2023 Readme.md Minor, documentation Jan 16, 2023 WhisperCpp.sln Source codes Jan 16, 2023 gui-capture.png Readme Jan 16, 2023 gui-load-model.png Readme Jan 16, 2023 gui-transcribe.png Readme Jan 16, 2023 View code [ ] Quick Start Guide Features Developer Guide Build Instructions Other Notes Performance Notes Further Optimisations Missing Features Final Words Readme.md This project is a Windows port of the whisper.cpp implementation. Which in turn is a C++ port of OpenAI's Whisper automatic speech recognition (ASR) model. Quick Start Guide Download WhisperDesktop.zip from the "Releases" section of this repository, unpack the ZIP, and run WhisperDesktop.exe. On the first screen it will ask you to download a model. I recommend ggml-medium.bin (1.42GB in size), because I've mostly tested the software with that model. Load Model Screen The next screen allows to transcribe an audio file. Transcribe Screen There's another screen which allows to capture and transcribe or translate live audio from a microphone. Capture Screen Features * Vendor-agnostic GPGPU based on DirectCompute; another name for that technology is "compute shaders in Direct3D 11" * Plain C++ implementation, no runtime dependencies except essential OS components * Much faster than OpenAI's implementation. On my desktop computer with GeForce 1080Ti GPU, medium model, 3:24 min speech took 45 seconds to transcribe with PyTorch and CUDA, but only 19 seconds with my implementation and DirectCompute. Funfact: that's 9.63 gigabytes runtime dependencies, versus 431 kilobytes Whisper.dll * Mixed F16 / F32 precision: Windows requires support of R16_FLOAT buffers since D3D version 10.0 * Built-in performance profiler which measures execution time of individual compute shaders * Low memory usage * Media Foundation for audio handling, supports most audio and video formats (with the notable exception of Ogg Vorbis), and most audio capture devices which work on Windows (except some professional ones, which only implementing ASIO API). * Voice activity detection for audio capture. The implementation is based on the 2009 article "A simple but efficient real-time voice activity detection algorithm" by Mohammad Moattar and Mahdi Homayoonpoor. * Easy to use COM-style API, idiomatic C# wrapper available on nuget * Pre-built binaries available The only supported platform is 64-bit Windows. Should work on Windows 8.0 or newer, but I have only tested on Windows 10. The library requires a Direct3D 11.0 capable GPU, which in 2023 simply means "any hardware GPU". The most recent GPU without D3D 11.0 support was Intel Sandy Bridge from 2011. Developer Guide Build Instructions 1. Clone this repository 2. Open WhisperCpp.sln in Visual Studio 2022. I'm using the freeware community edition, version 17.4.4. 3. Switch to Release configuration 4. Build and run CompressShaders C# project, in the Tools subfolder of the solution. 5. Build Whisper project to get the native DLL, or WhisperNet for the C# wrapper and nuget package, or the examples. Other Notes If you gonna consume the library in a software built with Visual C++, you probably redistribute Visual C++ runtime DLLs in the form of the .msm merge module. If you do that, right click on the Whisper project, Properties, C/ C++, Code Generation, switch "Runtime Library" setting from Multi-threaded (/MT) to Multi-threaded DLL (/MD), and rebuild: the binary will become smaller. The library includes RenderDoc GPU debugger integration. When launched your program from RenderDoc, hold F12 key to capture the compute calls. If you gonna debug HLSL shaders, use the debug build of the DLL, it includes debug build of the shaders and you'll get better UX in the debugger. The repository includes a lot of code which was only used for development: couple alternative model implementations, compatible FP64 versions of some compute shaders, debug tracing and the tool to compare the traces, etc. That stuff is disabled by preprocessor macros or constexpr flags, I hope it's fine to keep here. Performance Notes I have a limited selection of GPUs in this house. Specifically, I have optimized for nVidia 1080Ti, Radeon Vega 8 inside Ryzen 7 5700G, and Radeon Vega 7 inside Ryzen 5 5600U. The nVidia delivers relative speed 5.8 for the large model, 10.6 for the medium model. The AMD Ryzen 5 5600U APU delivers relative speed about 2.2 for the medium model. Not great, but still, much faster than realtime. I have also tested on nVidia 1650: slower than 1080Ti but pretty good, much faster than realtime. I have also tested on Intel HD Graphics 4000 inside Core i7-3612QM, the relative speed was 0.14 for medium model, 0.44 for small model. That's much slower than realtime, but I was happy to find my software works even on the integrated mobile GPU launched in 2012. I'm not sure the performance is ideal on discrete AMD GPUs, or integrated Intel GPUs, have not specifically optimized for them. Ideally, they might need slightly different builds of a couple of the most expensive compute shaders, mulMatTiled.hlsl and mulMatByRowTiled.hlsl And maybe other adjustments, like the useReshapedMatMul() value in Whisper/D3D/device.h header file. Further Optimisations I have only spent a few days optimizing performance of these shaders. It might be possible to do much better, here's a few ideas. * Newer GPUs like Radeon Vega or nVidia 1650 have higher FP16 performance compared to FP32, yet my compute shaders are only using FP32 data type. Half The Precision, Twice The Fun * In the current version, FP16 tensors are using shader resource views to upcast loaded values, and unordered access views to downcast stored ones. Might be a good idea to switch to byte address buffers, load/ store complete 4-bytes values, and upcast / downcast in HLSL with f16tof32 / f32tof16 intrinsics. * In the current version all shaders are compiled offline, and Whisper.dll includes DXBC byte codes. The HLSL compiler D3DCompiler_47.dll is an OS component, and is pretty fast. For the expensive compute shaders, it's probably a good idea to ship HLSL instead of DXBC, and compile on startup with environment-specific values for the macros. * It might be a good idea to upgrade the whole thing from D3D11 to D3D12. The newer API is harder to use, but includes potentially useful features not exposed to D3D11: wave intrinsics, and explicit FP16 . Missing Features Automatic language detection is not implemented. The original version implements "diarize" feature, they analyze stereo PCM to detect speaker based on the difference between left/ right channels. Despite my version preserves stereo PCM data over the pipeline, it doesn't expose that data. In the current version there's high latency for realtime audio capture. Specifically, depending on voice detection the figure is about 5-10 seconds. At least in my tests, the model wasn't happy when I supplied too short pieces of the audio. I have increased the latency and called it a day, but ideally this needs a better fix for optimal UX. Final Words From my perspective, this is an unpaid hobby project, which I completed over the 2022-23 winter holydays. The code probably has bugs. The software is provided "as is", without warranty of any kind. Thanks to Georgi Gerganov for whisper.cpp implementation, and the models in GGML binary format. I don't program Python, and I don't know anything about the ML ecosystem. I wouldn't even start this project without a good C++ reference implementation, to test my version against. That whisper.cpp project has an example which uses the same GGML implementation to run another OpenAI's model, GPT-2. It shouldn't be hard to support that ML model with the compute shaders and relevant infrastructure already implemented in this project. If you find this useful, I'll be very grateful if you consider a donation to "Come Back Alive" foundation. About High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model Resources Readme License MPL-2.0 license Stars 22 stars Watchers 1 watching Forks 2 forks Releases 2 Version 1.1 Latest Jan 16, 2023 + 1 release Packages 0 No packages published Languages * C++ 59.6% * C 26.6% * HLSL 8.5% * C# 5.3% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.