https://github.com/GRVYDEV/S.A.T.U.R.D.A.Y Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} GRVYDEV / S.A.T.U.R.D.A.Y Public * Notifications * Fork 5 * Star 28 A toolbox for working with WebRTC, Audio and AI License MIT license 28 stars 5 forks Star Notifications * Code * Issues 2 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights GRVYDEV/S.A.T.U.R.D.A.Y This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 10 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/G] Use Git or checkout with SVN using the web URL. [gh repo clone GRVYDE] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @GRVYDEV GRVYDEV add demo link ... 3d4800e Jul 2, 2023 add demo link 3d4800e Git stats * 107 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time client add openai ttt backend July 2, 2023 15:34 images cleanup + readme July 2, 2023 18:56 log implement basic llm interface (#17) July 2, 2023 14:59 models my current version May 31, 2023 00:44 rtc solve it! June 6, 2023 22:43 stt add openai ttt backend July 2, 2023 15:34 tester housekeeping May 30, 2023 10:24 tts implement basic llm interface (#17) July 2, 2023 14:59 ttt fix prompt bug July 2, 2023 19:45 util rename modules June 5, 2023 22:17 web implement basic llm interface (#17) July 2, 2023 14:59 whisper.cpp @ 5e2b340 start prep for whisper inference May 25, 2023 10:01 .gitignore my current version May 31, 2023 00:44 .gitmodules start prep for whisper inference May 25, 2023 10:01 LICENSE Initial commit April 9, 2023 12:10 Makefile cleanup + readme July 2, 2023 18:56 README.md add demo link July 2, 2023 20:28 docker-compose.yml fix client build process June 6, 2023 19:03 go.work add openai ttt backend July 2, 2023 15:34 go.work.sum fix client build process June 6, 2023 19:03 View code [ ] Project S.A.T.U.R.D.A.Y A toolbox for vocal computing built with Pion, whisper.cpp, and Coqui TTS. Build your own personal, self-hosted J.A.R.V.I.S powered by WebRTC Table of Contents About the Project How It Works STT (Speech-to-Text) TTT (Text-to-Text) TTS (Text-to-Speech) Diagram Getting started Prerequisites 1. RTC 2. TTS 3. Client Roadmap Local Inference Ease of Use Building With S.A.T.U.R.D.A.Y Discord Built With Bugs Contributing License Support Me Contact Me README.md Logo Project S.A.T.U.R.D.A.Y A toolbox for vocal computing built with Pion, whisper.cpp, and Coqui TTS. Build your own personal, self-hosted J.A.R.V.I.S powered by WebRTC Stars Badge Forks Badge View Demo * Getting Started * Request Features * Table of Contents 1. About The Project + How It Works + Diagram 2. Getting Started + Prerequisites 3. Roadmap 4. Discord 5. Built With 6. Bugs 7. Contributing 8. License 9. Support Me 10. Contact Me About the Project Project S.A.T.U.R.D.A.Y is a toolbox for vocal computing. It provides tools to build elegant vocal interfaces to modern LLMs. The goal of this project is to foster a community of like minded individuals who want to bring forth the technology we have been promised in sci-fi movies for decades. It aims to be highly modular and flexible while staying decoupled from specific AI Models. This allows for seamless upgrades when new AI technology is released. How It Works Project S.A.T.U.R.D.A.Y is composed of tools. A tool is an abstraction that encapsulates a specific part of the vocal computing stack. There are 2 main constructs that comprise a tool: * Engine - An engine encapsulates the domain specific functionality of a tool. This logic should remain the same regardless of the inference backend used. For example, in the case of the STT tool the engine contains the Voice Activity Detection algorithm along with some custom buffering logic. This allows the backend to be easily changed without needing to re-write code. * Backend - A backend is what actually runs the AI inference. This is usually a thin wrapper but allows for more flexibility and ease of upgrade. A backend can also be written to interface with an HTTP server to allow for easy language inter-op. This project contains 3 main kinds of tools. The 3 main tools are STT, TTT and TTS. STT (Speech-to-Text) STT tools are the ears of the system and perform Speech-to-Text inference on incoming audio. TTT (Text-to-Text) TTT tools are the brains of the system and perform Text-to-Text inference once the audio has been transformed into Text. TTS (Text-to-Speech) TTS tools are the mouth of the system and perform Text-to-Speech inference on the text proved by the TTT tool. Diagram Here is a diagram of how the main demo currently works. Saturday demo diagram Getting started The demo that comes in this repo is your own personal, self-hosted J.A.R.V.I.S like assistant. DISCLAIMER: I have only tested this on M1 Pro and Max processors. We are doing a lot of local inference so the demo requires quite a bit of processing power. Your mileage may very on different operating systems and hardware. If you run into problems please open an issue. In order to run the demo there are some pre-requisites. Prerequisites In order to run the demo, Golang, Python, Make and a C Complier are required. There are 3 processes that need to be running for the demo: * RTC - The RTC server hosts the web page and a WebRTC server. The WebRTC server is what you connect to when you load the page and it is also what the client connects to to start listening to your audio. * Client - The Client is where all of the magic happens. When it is started it joins the RTC server and starts listening to your audio. When you start speaking it will buffer the incoming audio until you stop. Once you stop speaking it will run STT inference on that audio, pass it to the TTT tool to generate a response to your text and then pass that output to the TTS tool to turn that response into speech. There are 2 system libraries needed to use the client pkg-config and opus. On macOS these can be installed with brew: brew install opus pkg-config * TTS - The TTS server is where text from the TTT tool is tranformed into speech. In the demo this uses Coqui TTS. There are 2 system libraries that are needed for this tool mecab and espeak. On macOS they can be installed with brew: brew install mecab espeak NOTE: For now the order in which you start the processes matters. You MUST start the RTC server and the TTS server BEFORE you start the client. 1. RTC From the root of the project run make rtc make run 2. TTS FIRST TIME SETUP: When you run the tts server for the first time you will need to install the dependencies. Consider using a virtual environment for this. cd tts/servers/coqui-tts pip install -r requirements.txt From the root of the project run make tts 3. Client The client requires whisper.cpp and the use of cgo however the make script should take care of this for you. From the root of the project run make client make client Roadmap Local Inference The main thing on the roadmap right now is getting TTT inference to run locally with something like llama.cpp. At the time of publishing this I do not have great internet and cannot download the model weights needed to get this working. Ease of Use The second largest item on my roadmap is continuing to improve the setup and configuration process. Building With S.A.T.U.R.D.A.Y The final thing on my roadmap is to continue to build applications with S.A.T.U.R.D.A.Y. I hope more people will build along with me as this is the #1 way to improve the project and uncover new features that need to be added. Discord Join the Discord to stay up to date! Built With This project is built with the following open source packages: * Pion * whisper.cpp * Coqui TTS Bugs I am very from perfect and there are bound to be bugs and things I've overlooked in the installation process. Please, add issues and feel free to reach out if anything is unclear. Also, we have a Discord. Contributing Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated. 1. Fork the Project 2. Create your Feature Branch: git checkout -b feature/ AmazingFeature 3. Commit your Changes: git commit -m 'Add some AmazingFeature' 4. Push to the Branch: git push origin feature/AmazingFeature 5. Open a Pull Request License MIT Support Me If you like the project and want to financially support it feel free to buy me a coffee Contact Me GitHub @GRVYDEV * Twitter @grvydev * Email grvy@aer.industries About A toolbox for working with WebRTC, Audio and AI Topics audio golang ai webrtc opus opus-codec whisper-cpp Resources Readme License MIT license Stars 28 stars Watchers 4 watching Forks 5 forks Report repository Releases No releases published Packages 0 No packages published Contributors 2 * @GRVYDEV GRVYDEV Garrett Graves * @TryAndErr0r TryAndErr0r Languages * Go 78.0% * JavaScript 11.4% * Dockerfile 2.9% * Python 2.6% * Makefile 2.5% * HTML 2.5% * Shell 0.1% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.