https://github.com/hegelai/prompttools Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} hegelai / prompttools Public * Notifications * Fork 32 * Star 720 Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate). prompttools.readthedocs.io License Apache-2.0 license 720 stars 32 forks Activity Star Notifications * Code * Issues 14 * Pull requests 1 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights hegelai/prompttools This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 11 branches 2 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/h] Use Git or checkout with SVN using the web URL. [gh repo clone hegela] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @steventkrawczyk steventkrawczyk Update README.md ... ba309c6 Aug 1, 2023 Update README.md ba309c6 Git stats * 366 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github Disable comment GHA for now July 31, 2023 15:48 docs Version bump August 1, 2023 10:42 examples Refactor utils functions without BC-breaking (#43) August 1, 2023 10:40 img Add streamlit playground (#33) July 30, 2023 17:42 prompttools Added support for top_p, max_tokens, frequency_penalty and presence_p... August 1, 2023 11:03 scripts Update autoeval notebook July 26, 2023 14:21 test Adding unit testing template July 17, 2023 16:28 .flake8 Adding pre-commit config (#18) July 13, 2023 07:54 .gitignore Add .chroma to .gitignore July 23, 2023 13:21 .pre-commit-config.yaml Adding pre-commit config (#18) July 13, 2023 07:54 .readthedocs.yaml Try req file again July 16, 2023 17:03 CODE_OF_CONDUCT.md Adding Code of Conduct July 18, 2023 20:45 CONTRIBUTING.md Adding pre-commit config (#18) July 13, 2023 07:54 LICENSE Update LICENSE (#27) July 27, 2023 09:36 README.md Update README.md August 1, 2023 15:11 pyproject.toml Version bump August 1, 2023 10:42 requirements.txt Adding streamlit to requirements and adding a README for the playgr... July 31, 2023 14:23 setup.py Add streamlit playground (#33) July 30, 2023 17:42 version.txt Version bump August 1, 2023 10:42 View code [ ] PromptTools Quickstart Playground Documentation Supported Integrations Frequently Asked Questions (FAQs) Contributing Usage and Feedback License README.md [687474707] PromptTools Test and experiment with prompts, LLMs, and vector databases. [6874747073] [6874747073] Total Downloads [6874747073] [6874747073] Welcome to prompttools created by Hegel AI! This repo offers a set of open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. The core idea is to enable developers to evaluate using familiar interfaces like code, notebooks, and a local playground. In just a few lines of codes, you can test your prompts and parameters across different models (whether you are using OpenAI, Anthropic, or LLaMA models). You can even evaluate the retrieval accuracy of vector databases. prompts = ["Tell me a joke.", "Is 17077 a prime number?"] models = ["gpt-3.5-turbo", "gpt-4"] temperatures = [0.0] openai_experiment = OpenAIChatExperiment(models, prompts, temperature=temperatures) openai_experiment.run() openai_experiment.visualize() image To stay in touch with us about issues and future updates, join the Discord. Quickstart To install prompttools, you can use pip: pip install prompttools You can run a simple example of a prompttools locally with the following git clone https://github.com/hegelai/prompttools.git cd prompttools && jupyter notebook examples/notebooks/OpenAIChatExperiment.ipynb You can also run the notebook in Google Colab Playground [playground] If you want to interact with prompttools using our playground interface, you can launch it with the following commands. First, install prompttools: pip install prompttools Then, clone the git repo and launch the streamlit app: git clone https://github.com/hegelai/prompttools.git cd prompttools && streamlit run prompttools/playground/playground.py Documentation Our documentation website contains the full API reference and more description of individual components. Check it out! Supported Integrations Here is a list of APIs that we support with our experiments: LLMs * OpenAI (Completion, ChatCompletion) - Supported * LLaMA.Cpp (LLaMA 1, LLaMA 2) - Supported * HuggingFace (Hub API, Inference Endpoints) - Supported * Anthropic - Supported * Google PaLM - Supported * LangChain - Exploratory Vector Databases and Data Utility * Chroma - Supported * Weaviate - Supported * MindsDB - Supported * Milvus - Exploratory * Pinecone - Exploratory * LanceDB - Exploratory * LlamaIndex - Exploratory If you have any API that you'd like to see being supported soon, please open an issue or a PR to add it. Feel free to discuss in our Discord channel as well. Frequently Asked Questions (FAQs) 1. Will this library forward my LLM calls to a server before sending it to OpenAI, Anthropic, and etc.? + No, the source code will be executed on your machine. Any call to LLM APIs will be directly executed from your machine without any forwarding. 2. Does prompttools store my API keys or LLM inputs and outputs to a server? + No, all data stay on your local machine. No metrics, telemetry, or usage data are collected. As a result, we would love to hear direct feedback from you. Please open an issue or join our Discord. 3. How do I persist my results? + To persist the results of your tests and experiments, you can export your Experiment with the methods to_csv, to_json, to_lora_json, or to_mongo_db. We are building more persistence features and we will be happy to further discuss your use cases, pain points, and what export options may be useful for you. Contributing We welcome PRs and suggestions! Don't hesitate to open a PR/issue or to reach out to us via email. Please have a look at our contribution guide and "Help Wanted" issues to get started! Usage and Feedback We will be delighted to work with early adopters to shape our designs. Please reach out to us via email if you're interested in using this tooling for your project or have any feedback. License We will be gradually releasing more components to the open-source community. The current license can be found in the LICENSE file. If there is any concern, please contact us and we will be happy to work with you. About Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate). prompttools.readthedocs.io Topics python machine-learning deep-learning embeddings developer-tools vector-search large-language-models prompt-engineering llms Resources Readme License Apache-2.0 license Code of conduct Code of conduct Activity Stars 720 stars Watchers 9 watching Forks 32 forks Report repository Releases 1 prompttools 0.0.22 Latest Aug 1, 2023 Packages 0 No packages published Used by 1 * @hegelai @hegelai / prompttools Contributors 5 * * * * * Languages * Python 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.