https://github.com/nkasmanoff/pi-card Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} nkasmanoff / pi-card Public * Notifications * Fork 3 * Star 119 * Raspberry Pi Voice Assistant 119 stars 3 forks Branches Tags Activity Star Notifications * Code * Issues 1 * Pull requests 1 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights nkasmanoff/pi-card This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 14 Commits assets assets assistanttools assistanttools sounds sounds .env.example .env.example .gitignore .gitignore README.md README.md assistant.py assistant.py config.py config.py requirements.txt requirements.txt View all files Repository files navigation * README Pi-C.A.R.D [assistant] Table of Contents * Introduction * Usage * Hardware * Setup Introduction Pi-card is an AI powered voice assistant running entirely on a Raspberry Pi. It is capable of doing anything a standard LLM (like ChatGPT) can do in a conversational setting. In addition, if there is a camera equipped, you can also ask Pi-card to take a photo, describe what it sees, and then ask questions about that image. Why Pi-card? Raspberry Pi - Camera Audio Recognition Device. [picard-fac] Please submit an issue or pull request if you can think of a better way to force this ackronym. How does it work? Pi-card runs entirely on your Raspberry Pi. Once the main program is run, the system will listen for your wake word. Once your wake word has been said, you are officially in a conversation. Within this conversation you do not need to constantly repeat the wake word. The system will continue to listen for your commands until you say something like "stop", "exit", or "goodbye". The system has a memory of the conversation while you have it, meaning if you want the assistant to repeat something it said, or elaborate on a previous topic, you can do so. While the system is designed to be entirely local, it is also possible to easily connect it to some external APIs or services if you want to enhance the conversation, or give it control to some external devices. To do so is something I am open to improving, but for now it will be done based on specific keywords to trigger the external service. A good example of this is that for camera purposes, the system will activate the camera if you say "take a photo" or "what do you see". How useful is it? The system is designed to be a fun project that can be a somewhat helpful AI assistant. Since everything is done locally, the system will not be as capable, or as fast, as cloud based systems. However, the system is still capable of a lot of improvements to be made. Why isn't this an app? The main reason for this is that I wanted to create a voice assistant that is completely offline and doesn't require any internet connection. This is because I wanted to ensure that the user's privacy is protected and that the user's data is not being sent to any third party servers. Usage After downloading the repository, installing the requirements, and following the other setup instructions, you can run the main program by running the following command: python assistant.py Once the program is running, you can start a conversation with the assistant by saying the wake word. The default wake word is "hey assistant", but you can change this in the config.py file. Hardware * Raspberry Pi 5 Model B * USB Microphone * Speaker * Camera Setup Software To keep this system as fast and lean as possible, we use cpp implementations of the audio transcription and vision language models. These are done with the wonderful libraries whipser.cpp for the audio transcription and llama.cpp for the vision language model. In both cases, please clone these repositories wherever you like, and add their paths to the config.py file. Once cloned, please go to each repository, and follow the setup instructions to get the models running. Some pointers are given below: For llama.cpp, we are using the vision language model capabilities, which are slightly different from the standard setup. You will need to follow the setup instructions for LlaVA, but update the model to be used to be one better suited for this device, Moondream2 To install Moondream, you'll need to go to HuggingFace model hub, and download the model. I did so using python, with the following commands. Once again, make sure the vision model path is added to the config.py file. from huggingface_hub import snapshot_download model_id="vikhyatk/moondream2" snapshot_download(repo_id=model_id, local_dir=your/local/path, local_dir_use_symlinks=False, revision="main") For whisper.cpp, you will need to follow the quick-start guide in the README. Hardware The hardware setup is quite simple. You will need a Raspberry Pi 5 Model B, a USB microphone, a speaker, and a camera. The USB microphone and speaker can be plugged into the Raspberry Pi's USB ports. The camera can be connected to the camera port on the Raspberry Pi. I used the following hardware for my setup: * Raspberry Pi 5 Kit * USB Microphone * Speaker * Camera * Camera Connector Please note Pi 5's have a new camera port, hence the new camera connector. Feel free to use your own, this is what worked for me. About Raspberry Pi Voice Assistant Resources Readme Activity Stars 119 stars Watchers 1 watching Forks 3 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 100.0% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.