https://github.com/comet-ml/opik Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + GitHub Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions By size + Enterprise + Teams + Startups By industry + Healthcare + Financial services + Manufacturing By use case + CI/CD & Automation + DevOps + DevSecOps * Resources Topics + AI + DevOps + Security + Software Development + View all Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} comet-ml / opik Public * Notifications You must be signed in to change notification settings * Fork 37 * Star 728 Open-source end-to-end LLM Development Platform License Apache-2.0 license 728 stars 37 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 0 * Pull requests 9 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights comet-ml/opik This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Last commit Last Name Name message commit date Latest commit History 131 Commits .github .github .hooks .hooks apps apps deployment deployment sdks/python sdks/python .dockerignore .dockerignore .gitignore .gitignore .java-version .java-version .pre-commit-config.yaml .pre-commit-config.yaml CLA.md CLA.md CONTRIBUTING.md CONTRIBUTING.md LICENSE LICENSE README.md README.md build_and_run.sh build_and_run.sh hooks-install.sh hooks-install.sh hooks-remove.sh hooks-remove.sh readme-thumbnail.png readme-thumbnail.png readme_header.png readme_header.png version.txt version.txt View all files Repository files navigation * README * Apache-2.0 license Comet Opik logo Opik Open-source end-to-end LLM Development Platform Confidently evaluate, test and monitor LLM applications. Python SDK License Build Website * Slack community * Twitter * Documentation Opik thumbnail What is Opik? Opik is an open-source platform for evaluating, testing and monitoring LLM applications. Built by Comet. You can use Opik for: * Development: + Tracing: Track all LLM calls and traces during development and production (Quickstart, Integrations + Annotations: Annotate your LLM calls by logging feedback scores using the Python SDK or the UI. * Evaluation: Automate the evaluation process of your LLM application: + Datasets and Experiments: Store test cases and run experiments (Datasets, Evaluate your LLM Application) + LLM as a judge metrics: Use Opik's LLM as a judge metric for complex issues like hallucination detection, moderation and RAG evaluation (Answer Relevance, Context Precision + CI/CD integration: Run evaluations as part of your CI/CD pipeline using our PyTest integration * Production Monitoring: Monitor your LLM application in production and easily close the feedback loop by adding error traces to your evaluation datasets. [?] Installation Opik is available as a fully open source local installation or using Comet.com as a hosted solution. The easiest way to get started with Opik is by creating a free Comet account at comet.com. If you'd like to self-host Opik, you can do so by cloning the repository and starting the platform using Docker Compose: # Clone the Opik repository git clone https://github.com/comet-ml/opik.git # Navigate to the opik/deployment/docker-compose directory cd opik/deployment/docker-compose # Start the Opik platform docker compose up --detach # You can now visit http://localhost:5173 on your browser! For more information about the different deployment options, please see our deployment guides: Installation methods Docs link Local instance Local Deployment Kubernetes Kubernetes Get Started To get started, you will need to first install the Python SDK: pip install opik Once the SDK is installed, you can configure it by running the opik configure command: opik configure This will allow you to configure Opik locally by setting the correct local server address or if you're using the Cloud platform by setting the API Key Tip You can also call the opik.configure(use_local=True) method from your Python code to configure the SDK to run on the local installation. You are now ready to start logging traces using the Python SDK. Logging Traces The easiest way to get started is to use one of our integrations. Opik supports: Integration Description Documentation Try in Colab Log traces for all OpenAI LLM Open OpenAI calls Documentation Quickstart In Colab Log traces for all LangChain Open LangChain LLM calls Documentation Quickstart In Colab Log traces for all LlamaIndex Open LlamaIndex LLM calls Documentation Quickstart In Colab Fine-tune and serve Open Predibase open-source Large Language Documentation Quickstart In Models Colab Tip If the framework you are using is not listed above, feel free to open an issue or submit a PR with the integration. If you are not using any of the frameworks above, you can also using the track function decorator to log traces: import opik opik.configure(use_local=True) # Run locally @opik.track def my_llm_function(user_question: str) -> str: # Your LLM code here return "Hello" Tip The track decorator can be used in conjunction with any of our integrations and can also be used to track nested function calls. [?][?] LLM as a Judge metrics The Python Opik SDK includes a number of LLM as a judge metrics to help you evaluate your LLM application. Learn more about it in the metrics documentation. To use them, simply import the relevant metric and use the score function: from opik.evaluation.metrics import Hallucination metric = Hallucination() score = metric.score( input="What is the capital of France?", output="Paris", context=["France is a country in Europe."] ) print(score) Opik also includes a number of pre-built heuristic metrics as well as the ability to create your own. Learn more about it in the metrics documentation. Evaluating your LLM Application Opik allows you to evaluate your LLM application during development through Datasets and Experiments. You can also run evaluations as part of your CI/CD pipeline using our PyTest integration. Contributing There are many ways to contribute to Opik: * Submit bug reports and feature requests * Review the documentation and submit Pull Requests to improve it * Speaking or writing about Opik and letting us know * Upvoting popular feature requests to show your support To learn more about how to contribute to Opik, please see our contributing guidelines. About Open-source end-to-end LLM Development Platform Resources Readme License Apache-2.0 license Activity Custom properties Stars 728 stars Watchers 22 watching Forks 37 forks Report repository Releases 42 0.1.14 Latest Sep 17, 2024 + 41 releases Packages 0 Contributors 24 * * * * * * * * * * * * * * + 10 contributors Languages * Java 50.4% * Python 30.9% * TypeScript 17.9% * SCSS 0.3% * Shell 0.3% * JavaScript 0.1% * Other 0.1% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.