[HN Gopher] Good LLM Validation Is Just Good Validation
___________________________________________________________________
Good LLM Validation Is Just Good Validation
Author : ivanleomk
Score : 53 points
Date : 2023-10-27 16:11 UTC (6 hours ago)
(HTM) web link (jxnl.github.io)
(TXT) w3m dump (jxnl.github.io)
| ivanleomk wrote:
| What if your validation logic could learn and adapt like a human
| but feel exactly like writing normal software. This is the future
| of validation and it's already here.
| ren_engineer wrote:
| home page for Instructor mentions using Marvin for "a more
| comprehensive solution". When should somebody use Instructor vs
| Marvin? Is it just a library vs framework type situation in terms
| of tradeoffs?
| behnamoh wrote:
| I've come to realize you better develop your own little library
| for LLMs instead of depending on complex libraries with hidden
| prompts here and there which you can't control.
|
| People think LLMs are about connecting things (chaining) but
| you'll get that for free if you focus instead on the "content"
| and "behavior" of your LLMs.
| jxnlco wrote:
| The goal of instructor is that i dont write any prompts for
| you, i just make the openai.ChatCompletion.create call easier
| to manage and let you return a pass in a pydantic object and
| return an instance of it,
|
| Whole codebase is 300 lines of code you can copy paste into
| your own repo into function_calls.py :)
| TeMPOraL wrote:
| Yeah, obligatory reminder that "chaining" is just function
| composition. As in:
| SecondQuery(FirstQuery(userInput));
|
| "Advanced" chaining can be as simple as: do {
| partial = FirstQuery(input); choice =
| DeciderQuery(partial); input = choice ?
| Query2A(partial) : Query2B(partial); }
| while(!ValidationQuery(input)); return input; // or
| print it or whatever
|
| It only looks complex when you need to chain things at
| runtime through some interface, but that's the usual problem
| of having to reinvent basic blocks of your language as
| objects or data structures because your language doesn't let
| you treat code and data interchangeably. But if you're just
| experimenting with chains, using the most basic programming
| building blocks directly - functions, conditionals, loops -
| is by far the easiest.
| danenania wrote:
| Or put even simpler: calling a function with the result of
| another function.
|
| I'd question how much value FP abstractions like partials
| and composition are really adding in your example.
| TeMPOraL wrote:
| Oh, I didn't mean to invoke FP abstractions -
| "composition" is used outside of FP, and "partial"...
| that's just bad naming choice on my end. This example was
| meant to be dumb, imperative C-like code.
| danenania wrote:
| Ah sorry, I saw 'partial' and thought you were passing
| around functions. Please disregard my comment :)
| jxnlco wrote:
| creator of instructor here:
|
| My only goal is to make structured output easier, I want to be
| thought of as more like 'requests' it just makes http/io a bit
| easier. Rather than say, a whole app framework.
|
| Instructor only supports openai's function calls -> pydantic
| and then i mostly spend my time writing docs on how to think
| about this pattern.
|
| Marvin has logic for custom apps, chat bots, functions, on
| different llm backends etc.
| techn00 wrote:
| Does anyone know a similar library for js/ts?
| doomroot wrote:
| Zod
| vishhvak wrote:
| You should checkout https://www.symphony.run/docs/functions
| jxnlco wrote:
| I'd say its zod, but it does not have validation in the same
| way instructor does. Typechat has some options tho!
|
| https://github.com/microsoft/typechat/
| bluecrab wrote:
| Something like this for open source models would be great.
___________________________________________________________________
(page generated 2023-10-27 23:01 UTC)