[HN Gopher] Show HN: Opik, an open source LLM evaluation framework
___________________________________________________________________
Show HN: Opik, an open source LLM evaluation framework
Hey HN! I'm Caleb, one of the contributors to Opik, a new open
source framework for LLM evaluations. Over the last few months, my
colleagues and I have been working on a project to solve what we
see as the most painful parts of writing evals for an LLM
application. For this initial release, we've focused on a few core
features that we think are the most essential: - Simplifying the
implementation of more complex LLM-based evaluation metrics, like
Hallucination and Moderation. - Enabling step-by-step tracking,
such that you can test and debug each individual component of your
LLM application, even in more complex multi-agent architectures. -
Exposing an API for "model unit tests" (built on Pytest), to allow
you to run evals as part of your CI/CD pipelines - Providing an
easy UI for scoring, annotating, and versioning your logged LLM
data, for further evaluation or training. It's often hard to feel
like you can trust an LLM application in production, not just
because of the stochastic nature of the model, but because of the
opaqueness of the application itself. Our belief is that with
better tooling for evaluations, we can meaningfully improve this
situation, and unlock a new wave of LLM applications. You can run
Opik locally, or with a free API key via our cloud platform. You
can use it with any model server or hosted model, but we currently
have a built-in integration with the OpenAI Python library, which
means it automatically works not just with OpenAI models, but with
any model served via a compatible model server (ollama, vLLM, etc).
Opik also currently has out-of-the-box integrations with LangChain,
LlamaIndex, Ragas, and a few other popular tools. This is our
initial release of Opik, so if you have any feedback or questions,
I'd love to hear them!
Author : calebkaiser
Score : 60 points
Date : 2024-09-17 13:01 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| tcsizmadia wrote:
| It looks very promising! Congratulations, great tool! I can't
| wait to start experimenting with it. I plan to use it locally,
| with Ollama.
| calebkaiser wrote:
| Awesome, thanks! If you run into any issues, you can open a
| ticket on the repo or ping me directly at caleb[at]comet.com
| hrpnk wrote:
| Is there a reason you didn't just implement OpenTelemetry (OT)
| straight away? Curious about the trade offs to opt for a home-
| grown telemetry inspired by OT instead.
| calebkaiser wrote:
| Good question! It mostly came down to implementation speed, as
| well as some uncertainty about performance/overhead. We will be
| releasing OpenTelemetry compatible ingestion endpoints in the
| near future, but since Opik has so many features that aren't
| related to OT, we decided to move forward without it for the
| initial release. It is a great project though and something we
| will be implementing soon--it will especially be useful for
| building out integrations with frameworks that are
| OpenTelemetry compatible.
| baggiponte wrote:
| Have you seen two "prototypes" of standard for LLM telemetry?
| One is openllmetry, maintained by the folks at TraceLoop.
| Seems the more popular. The other one is openinference IIRC,
| by Arize AI.
| calebkaiser wrote:
| Of the two, the only one I've ever personally explored is
| OpenLLMetry. Extremely cool project. In general, this is
| one of those areas where the field still needs to "shake
| out" a bit.
| kakaly0403 wrote:
| There is a GenAI standard spec from OpenTelemetry for
| tracing LLM based applications. Currently there are 3
| library implementations of this spec - Langtrace,
| OpenLLMetry and OpenLit. Microsoft has an implementation
| for .NET aswell. OpenInference, though opentelemetry
| compatible does not adhere to the standard spec.
| yu3zhou4 wrote:
| Hello! How does it compare to DeepEval (open source)?
| calebkaiser wrote:
| Great question. First, we have a ton of respect for the work
| the DeepEval team is doing. That said, we took a fundamentally
| different approach in building Opik as an open source project.
| With DeepEval, if you want to log your data or use the UI, you
| need to use Confident AI's cloud platform (which as far as I'm
| aware, has no free plan). So, if you want to visualize traces,
| do production monitoring, labeling, etc, you can't just use the
| DeepEval open source library.
|
| All of Opik's functionality, including the UI and logging, is
| available in the open source version. The only "features" that
| are inaccessible from the open source version of Opik are
| things that are actually features of the Comet platform. For
| example, Comet Artifacts allow you to store your datasets as
| versioned assets, preserved as an immutable series of
| snapshots, which automatically track any experiments they've
| been a part of in order to preserve their full data lineage.
| You can use Opik with Artifacts, but that will require a free
| Comet account. Any Opik-specific feature, however, is fully
| available in the open source version.
___________________________________________________________________
(page generated 2024-09-17 23:01 UTC)