Host Your Own Evaluation Framework for LLMs
reddit / selfhosted

----------------------------------------------------------------------
sunglasses-guy · 2023-12-13 18:31

Hi all, I've been working on this evaluation framework for LLM
(generative AI) applications for a few months, and implemented 10+ LLM
evaluation metrics so ML engineers and data scientists don't have to
do the boring work. The best part is more than half of these metrics
are able to run locally on your machine, which is perfect for AI
applications where data might be extremely sensitive.

Since I discovered this morning that my framework is now trending on
Github, I would love to share my work with you all:
[https://github.com/confident-
ai/deepeval](https://github.com/confident-ai/deepeval). Thank you for
your support!

​

----------------------------------------------------------------------
CosineTau · 2023-12-13 23:26

Can you help me understand the utility better?

The Readme makes an intuitive analogy to unit testing. Are we testing
the response to specific prompts against an expected response? For
instance, we are testing to make sure "blue" is the response the
prompt "what color is the sky?"

----------------------------------------------------------------------
sunglasses-guy · 2023-12-18 14:30

It really depends, your example is a case where we would require the
expected response for exact string matching. However, you can also
imagine a use case where we're trying to evaluate a summarization,
which in this case we will only require the 1) prompt (original text),
2) response (summary), and not the expected response.


In fact, different "metrics" require different components in a test
case in order to be able to function. Here's the summarization we
have: [https://docs.confident-ai.com/docs/metrics-
summarization](https://docs.confident-ai.com/docs/metrics-
summarization), it uses LLMs for evaluation but in a very clever way,
do give it a read!