[HN Gopher] Test Driven Development (TDD) for your LLMs? Yes ple...
___________________________________________________________________
Test Driven Development (TDD) for your LLMs? Yes please, more of
that please
Author : lewq
Score : 26 points
Date : 2024-12-04 14:42 UTC (8 hours ago)
(HTM) web link (blog.helix.ml)
(TXT) w3m dump (blog.helix.ml)
| justanotheratom wrote:
| There is a real need for this. I have to admit most of my testing
| right now is vibes based. Problem is, these LLM evaluation
| platforms get in-between me and my LLM.
| Onawa wrote:
| I use promptfoo for our internal evaluations. There are a ton
| of different "assertions" (tests) that you can write, including
| model-graded evaluations using rubrics.
|
| This is far from a solved problem, but there are options out
| there for systematic testing of LLMs.
| benatkin wrote:
| I read this blog post on my iPhone and when I went to the top to
| try and find out more about Helix, it had a giant link to install
| the Substack iOS app, which detracted from the experience. It
| might be a good idea to use a real CMS.
|
| Here's the website: https://tryhelix.ai/
| heavyarms wrote:
| Whenever I see one of these posts, I click just to see if the
| proposed solution to testing the output of an LLM is to use the
| output of an LLM... and in almost all cases it is. It doesn't
| matter how many buzzwords and acronyms you use to describe what
| you're doing, at the end of the day it's turtles all the way
| down.
|
| The issue is not the technology. When it comes to natural
| language (LLM responses that are sentences, prose, etc.) there is
| no actual standard by which you can even judge the output. There
| is no gold standard for natural language. Otherwise language
| would be boring. There is also no simple method for determining
| truth... philosophers have been discussing this for thousands of
| years and after all that effort we now know that...
| -\\_(tsu)_/-... and also, Earth is Flat and Birds Are Not Real.
|
| Take, for example, the first sentence of my comment: "Whenever I
| see one of these posts, I click just to see if the proposed
| solution to testing the output of an LLM is to use the output of
| an LLM... and in almost all cases it is." This is absolutely
| true, in my own head, as my selective memory is choosing to
| remember that one time I clicked on a similar post on HN. But
| beyond the simple question of if it is true or not, even an army
| of human fact checkers and literature majors could probably not
| come up with a definitive and logical analysis regarding the
| quality and veracity of my prose. Is it even a grammatically
| correct sentence structure... with the run-on ellipsis and what
| not... ??? Is it meant to be funny? Or snarky? Who knows
| -\\_(tsu)_/- WFT is that random pile of punctuation marks in the
| middle of that sentence... does the LLM even have a token for
| that?
| senko wrote:
| The output of an LLM is often qualitative, not quantitative,
| and to test that, you _need_ something that can judge the
| quality.
|
| You're not debating philosophy with the LLM, you're just asking
| it if the answer matches (semantically) to the expected one.
|
| I usually test LLM output quality with the following prompt
| (simplified):
|
| "An AI assistant was tasked with {task}. The relevant
| information for their task was {context}. Their answer is
| {answer}. The correct answer should be something like {ground
| truth}. Is their answer correct?"
|
| Then you can spice it up with chain of thought, asking it to
| judge alongside preferred criteria/dimensions and output a
| score, etc... you can go as wild as you'd like. But even this
| simple approach tends to work really well.
|
| > turtles all the way down.
|
| Saying "LLM testing LLM" is bad is like saying "computer
| testing computer" is bad. Yet, automated tests have value. And
| just as the unit tests will not prove your program is bug free,
| LLM evals won't guarantee 100% correctness. But they're
| _incredibly_ useful tool.
|
| In my experience working on pretty complex multi-agent multi-
| step systems, trying to get those to work _without_ an eval
| framework in place is like playing whack-a-mole, only way less
| fun.
| throwawaymaths wrote:
| Its inherently difficult because LLMs are necessarily
| probabilistic and even worse, for any practical use the key step
| irreversibly discards most of the probabilities
___________________________________________________________________
(page generated 2024-12-04 23:01 UTC)