[HN Gopher] Show HN: Fructose, LLM calls as strongly typed funct...
___________________________________________________________________
Show HN: Fructose, LLM calls as strongly typed functions
Hi HN! Erik here from Banana (formerly the serverless GPU
platform), excited to show you what we've been working on next:
Fructose Fructose is a python package to call LLMs as strongly
typed functions. It uses function type signatures to guide the
generation and guarantee a correctly typed output, in whatever
basic/complex python datatype requested. By guaranteeing output
structure, we believe this will enable more complex applications to
be built, interweaving code with LLMs with code. For now, we've
shipped Fructose as a client-only library simply calling gpt-4 (by
default) with json mode, pretty simple and not unlike other
packages such as marvin and instructor, but we're also working on
our own lightweight formatting model that we'll host and/or
distribute to the client, to help reduce token burn and increase
accuracy. We figure, no time like the present to show y'all what
we're working on! Questions, compliments, and roasts welcomed.
Author : edunteman
Score : 125 points
Date : 2024-03-06 18:17 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jsight wrote:
| I love the concept, but I'd really prefer being able to use it
| against local llms (localai, ollama, etc).
| edunteman wrote:
| Seems like a great feature (and honestly allows us to do
| smarter things for strictly structured generation). I'm
| curious, what's your main motivation for local llms vs hosted
| APIs?
| imtringued wrote:
| I'm pretty sure the primary reason is that you don't want an
| update of the hosted LLM to suddenly break your application
| without warning.
| ramses0 wrote:
| Not wanting to get dinged for $20/mo? Ability to use offline
| for local home automation (eg: "given a verbal input request,
| determine the home devices in scope and their on/off state"
| => "given the current devices state and the verbal request,
| generate a list of home assistant actions to perform"), using
| a custom model for the above, etc.
| nate_nowack wrote:
| as with marvin, you can just swap the base url and use any of
| the oss proxy libs that clone the openai api (but since they
| don't do function calling [except for mistral i think], its not
| as good afaik)
| jonathan-adly wrote:
| Very Cool! Would it work for Pydantic out of the box? Or that's
| something coming along?
| edunteman wrote:
| currently don't have pydantic support yet, but we're not too
| opinionated on that. I know it's seemed to emerge as a
| standard, and I imagine useful in the context of running
| fructose in a FastAPI handler, but we led with dataclasses
| because they're language native and achieve much of the same
| thing
| yaj54 wrote:
| Pydantic can serialize both instances and classes/types to
| json and json schema. That seems quite helpful for this use
| case. How are you handling serialization to/from the llm?
| Role-your-own or are there additional libraries for doing
| this with dataclasses?
| bagels wrote:
| How do you guarantee output structure? Does it ever fail to
| conform?
| edunteman wrote:
| It's not 100% yet. Route to that: 1. Clientside, retry strategy
| on failed parse. Not yet implemented, we throw an exception on
| parse fail right now, but soon to be implemented. Not ideal
| because of token burn and latency, but the best quick solution.
| 2. For the custom model we're building, we use strict grammar
| definitions to bias outputs toward the needed structure (or if
| there is only one structurally correct token, outright skip the
| generation of that token and insert it directly).
|
| I've been impressed at how well gpt-4 does with the default
| prompt template we use. Even better if you enable the
| chain_of_thought flavor.
| bagels wrote:
| Basically my experience with homegrown. If you give it a
| typescript template for outputs, and a ton of prose to tell
| it how to respond, you usually get the right responses.
| FrustratedMonky wrote:
| Can this be a F# Type Provider?
| Smaug123 wrote:
| Good Lord that sounds terrifying - I do in _general_ prefer my
| type definitions to be at least approximately deterministic.
| minimaxir wrote:
| This approach may be too high-level "magic" to the point of being
| difficult to work with and iterate upon.
|
| Looking at the prompt templates
| (https://github.com/bananaml/fructose/tree/main/src/fructose/...
| ), they use LangChain-esque "just try to make the output to be
| valid JSON" when APIs such as GPT-4 Turbo which this model uses
| by default now support function calling/structured data natively
| and do a very good job of it
| (https://news.ycombinator.com/item?id=38782678), and libraries
| such as outlines (https://github.com/outlines-dev/outlines) which
| is more complex but can better ensure a dictionary output for
| local LLMs.
| edunteman wrote:
| Many of our early users have said this as well. I don't want
| this to turn into an abstraction monstrosity: the more
| unadulterated the prompt, the better. We're looking to outlines
| as inspiration for doing this logic as part of the model vs the
| client. Thanks for the links!
| biddit wrote:
| I've done a lot of work over the last year wrangling LLM outputs
| - both from the OpenAI API as well as local LLMs.
|
| What are the benefits of using Fructose over LMQL, Guidance or
| OpenAI's function calling?
| edunteman wrote:
| Still learning about the landscape so can't give informed
| opinions. LMQL is a new one for me, will check it out.
|
| What we're mostly going for is composability vs abstraction.
| What's the smallest nugget of lift we can do for you, to make
| it feel natural to implement what you want? In this case it's
| treating the calls as functions and leaning on native python
| features like functions, docstrings, and types, so you can
| still use the python language like closures to do the weird
| things you need.
|
| This is all handwavy, put on my wizard language design hat, so
| take it with a grain of salt. We're just trying things out.
| nerpderp82 wrote:
| Mentioned packages
|
| https://github.com/eth-sri/lmql
|
| https://github.com/outlines-dev/outlines
|
| https://github.com/guidance-ai/guidance
| edunteman wrote:
| here's an awesome post on the landscape
| https://hamel.dev/blog/posts/prompt/
| nerpderp82 wrote:
| I remember reading that, good stuff.
|
| I'd like to see an injectable mitm like proxy that can
| rewrite payloads. Many of these frameworks are useful,
| but when they go off the rails, they hard to modify and
| introspect.
|
| It would be nice if LLMs had a way to speak an annotated
| format, like XML that was able to encode higher level
| information in a coherent manner over "well formed"
| addhoc text.
|
| LLM libraries are in a crazy state right now. It is like
| JS frameworks 2015, a new one that demos well every other
| day.
| edunteman wrote:
| one idea we're cooking is to offer a proxy with a hosted
| reformatting model on-board, to rewrite payloads on their
| way back in the case of type parse failure. fructose, the
| clientside sdk, would be optional
| iAkashPaul wrote:
| TGI just integrated Guidance in 1.4.3, that by itself can support
| both grammar/JSON/Pydantic & tool invocation/function calling.
|
| Langchain & Llamaindex plus Fructose really need to skip the
| structure adherence work & move to chunking/KG generation since
| that's the next pain point to tackle.
| memothon wrote:
| What is TGI? Is that the huggingface Text Generation Inference
| project on GitHub?
| edunteman wrote:
| Yes. TGI is Huggingface's version of LLVM (some nuance, of
| course). LLVM also launched grammar support recently too, so
| we'll be looking into it.
| memothon wrote:
| All of these acronyms are so confusing. I'm assuming LLVM
| isn't the compiler tool, but searching "LLVM ai" doesn't
| give me any good results.
| reliableturing wrote:
| They probably meant vLLM https://docs.vllm.ai/en/latest/
| edunteman wrote:
| ah shoot, yes I meant vLLM, sorry for the confusion, lots
| of comments to reply to :)
| Terretta wrote:
| Why do you have Guidance in caps?
|
| https://github.com/guidance-ai/guidance
|
| or ...
|
| https://huggingface.co/docs/text-generation-inference/concep...
|
| or ... ?
|
| A quick glance through these, they don't seem yet to leverage
| json_object on OpenAI with the word JSON in the prompt, which
| works wonders with the 0125 models.
| politelemon wrote:
| Are you planning to add other types like Claude or Llama2?
| edunteman wrote:
| eventually, but priority goes toward finding an abstraction
| that feels right. We're very likely to break this package API,
| still v0. Sticking with openai till we have more confidence in
| the foundation being correct.
| hedgehog wrote:
| How does Fructose relate or compare to Instructor
| (https://github.com/jxnl/instructor)?
| edunteman wrote:
| Currently, quite comparable and obviously Instructor is more
| mature and feature rich. They're going the "patch the openAI
| client" approach which makes code written still use openAI SDK
| patterns which is pretty smart. Jason seems like he knows what
| he's doing.
|
| We're trying to make it more of a language feature with the
| decorated functions. Plus exploring the hosted formatting model
| direction.
|
| How do you feel this compares? Do you think there's any gaps in
| current tools worth working on?
| dennisy wrote:
| Plus one on this question! Seems very similar.
| creatonez wrote:
| Yet another implementation of https://esolangs.org/wiki/English
| imtringued wrote:
| Since you are going down this route, I would recommend you guys
| to build some sort of unit test driven fine tuning framework,
| where you may provide input output examples expressed as simple
| function calls. You could then let the LLM generate examples and
| check them using the unit tests and keep the valid results to
| build up a valid data set. For bonus points, the unit tests
| themselves could also call the LLM to check if the output passes
| criteria expressed in natural language or not.
| nextos wrote:
| IMHO, in the future programming may look similar to this. Write a
| type declaration for a function with an expressive type system,
| e.g. refinement types. Then use LLMs + SAT/SMT to generate
| provably correct code.
|
| This strikes a happy medium, where machines are assisting
| programmers, making them much more productive. Yet the resulting
| code is understandable as a human has decomposed everything into
| functions, and also robust as it is formally verified.
|
| I am working on a F# proof-of-concept system like this, there are
| other alternatives around implemented in Haskell and other
| languages with varying levels of automation. It is potentially an
| interesting niche for a startup.
| edunteman wrote:
| yeah I had a moment working with fructose where I realized "oh
| this is more like functional programming than I expected"
| obeavs wrote:
| Is the F# POC open source? Link?
| nextos wrote:
| Not yet, it's a bit rough. The LLM I am using requires a bit
| of extra fine-tuning to be really smooth, I need to rent a
| bigger GPU. Besides, I am working on some novel integration
| between transformers and SAT/SMT that will take me some time
| to finish.
| yaj54 wrote:
| Is the theory tied to a specific llm? I'm interpreting it
| as, e.g., the llm writes the code, the solver verifies it,
| repeat until correct. In this situation the two are
| decoupled and the llm would be a drop in and thusly could
| be any local or remote llm. Is there something about your
| approach that doesn't allow this?
|
| (also, +1 for OS link request)
| nextos wrote:
| Decoupling both is the simplest option, but not the one I
| am focusing on. Also note SAT/SMT can also be used for
| synthesis.
|
| In fact, synthesis has a relatively rich history using
| SAT/SMT solvers.
| yaj54 wrote:
| Are you using the SAT/SMT solvers to feed training data
| into a transformer or integrating the solver logic into
| the model code?
| obeavs wrote:
| Our product (phosphor) is built end-to-end in F# so this
| stuff is close to my heart. You might find Moonbit's
| approach to functional AI interesting as well
| https://www.moonbitlang.com/blog/moonbit-ai.
| ravenstine wrote:
| Yeah, functional programming and pure functions seem perfect
| for generative programming. Granted, I think they're perfect
| _in general_ , but as long as human programmers are still stuck
| in the world of object-orientation (the modern sense), then
| they're going to be wasting the time in the LLM feedback loop.
| The LLM should be be able to write a unit of code however it
| wants in a way that is as self-contained as possible. Since an
| LLM can, in theory, quickly "understand" code that most
| software engineers would object to, then we should get out of
| the way of LLMs rather than expect them to be like we are.
| fwip wrote:
| This project seems pretty different to what you've proposed.
| Fructose looks like it's "just" asking an LLM to evaluate a
| function (written in English text), and then jamming whatever
| comes out back into your type system.
|
| Being able to sometimes answer a given question is perhaps a
| first step to writing code that can answer that question
| reliably, but it's a long way from an LLM that does the former
| to one that does the latter.
| nickpsecurity wrote:
| The field that has been doing this is called Program Synthesis.
| Here's an example survey:
|
| https://www.microsoft.com/en-us/research/publication/program...
|
| I've wanted to see the traditional techniques combined with
| modern ML to sort of drive the search and generation process.
| Then, we'd still have the advantages of both formal
| specifications and classic AI (esp traceability). While looking
| for a synthesis link, I stumbled onto one paper trying to mix
| the two approaches:
|
| https://ojs.aaai.org/index.php/AAAI/article/download/5048/49...
| ddellacosta wrote:
| Oh yeah, you just reminded me of this cool talk I saw at
| Strange Loop a while back. Not about the AI parts but re:
| program synthesis in Haskell:
|
| "Type-Driven Program Synthesis" by Nadia Polikarpova
| https://www.youtube.com/watch?v=HnOix9TFy1A
|
| Links to more projects and papers by Prof. Polikarpova:
| https://cseweb.ucsd.edu/~npolikarpova/
|
| I think this is one of the main projects she discusses in the
| talk: https://github.com/nadia-polikarpova/synquid
|
| EDIT: meant to mention this too, which I think has been around
| a bit longer, not that I've ever used it in production:
| https://ucsd-progsys.github.io/liquidhaskell/
| nextos wrote:
| Lots of related work also by A. Solar-Lezama
| https://people.csail.mit.edu/asolar, sometimes in
| collaboration with N. Polikarpova.
| ddellacosta wrote:
| TIL, nice--thanks!
| mattew wrote:
| Good stuff. How does this compare to Instructor? I've been using
| this extensively
|
| https://jxnl.github.io/instructor/
| edunteman wrote:
| answered in different thread. tldr: not that different for now.
| we're likely to do some serverside optimizations, esp. given
| our gpu inference history.
| yaj54 wrote:
| I like your UX a lot more. Modeling the llm calls as actual
| python functions allows them to mesh well with existing code
| organization dev tooling. And using a decorator to
| "implement" a function just feels like a special kind of
| magic. I'd need more ability to use my own "prompt templates"
| to use this as a lib but I'm definitely going to try using
| this general pattern.
| itfollowsthen wrote:
| > not unlike other packages such as marvin
|
| This feels pretty much identical to Marvin? Like the entire API?
|
| From a genuine place of curiosity: I get that your prompts are
| different, but like why in the name of open source would you just
| not contribute to these libraries instead of starting your own
| from scratch?
| nate_nowack wrote:
| yeah this seems to be pretty much the same interface as `fn`
| from marvin, except w/o pydantic (see
| https://github.com/PrefectHQ/marvin?tab=readme-ov-
| file#-buil...)
| edunteman wrote:
| Thanks for asking, and I'd agree. I'd give the same answer as
| the folks asking about instructor: we built this in a week and
| are sharing it early, this package API happens to have landed
| on what Marvin is doing, we're likely to change over time,
| especially leaning toward running our own models as part of it.
| yieldcrv wrote:
| fyi: LM Studio can host a server that uses the OpenAI api to
| whatever model you are using locally
|
| So as long as this library can be directed to localhost or
| configured, it can use any LLM
| msp26 wrote:
| So what is this actually putting into the prompt to guide
| generation? I dislike libraries that come with a lot of pointless
| abstraction.
|
| I'm about to write something that generates typescript code from
| pydantic models. If this just works out of box, it would make me
| very happy.
|
| I'll take a look through the repo tomorrow, sorry if my response
| is a little lazy, I just got off work.
| AmanKishore wrote:
| This is much nicer than calling GPT in the middle of my code.
| Honestly the aesthetics of Fructose just make the code so much
| neater
| edunteman wrote:
| Thanks!
| shykes wrote:
| I love this emerging space at the intersection of programming and
| LLMs. It goes beyond having the LLMs generate code: that's an
| obvious and amazing use case, but it's far from the only one.
|
| Another project I'm excited about in this area is GPTScript,
| which launched last week: http://github.com/gptscript-
| ai/gptscript.
| babyshake wrote:
| Does anyone else get bothered by how this seemingly results in
| code that won't compile?
|
| Instead of this:
|
| @ai() def describe(animals: list[str]) -> str: """ Given a list
| of animals, use one word that'd describe them all. """
|
| it would seem a lot more intuitive to do this:
|
| def describe(animals: list[str]) -> str: return ai("""Given a
| list of animals, use one word that'd describe them all.""",
| animals)
| edunteman wrote:
| Yeah the pyright doesn't like the annotated return type not
| being honored by the empty stub function. I wonder if there's a
| way to trick it.
|
| For your suggestion, the decorator would still be required to
| overload the function execution with the remote call, otherwise
| you'd just be calling the function body, but we have considered
| special wrapper return types to help play better with pyright
| (and also give programmatic access to debug details of the
| call), but that'd add bloat to the package and subtract from
| the more native python feel we're aiming for.
| ilaksh wrote:
| I find it grating that all of these types of things say "LLMs"
| when in fact they literally only work with OpenAI. There are
| hundreds of variations of LLM models. When it works with only
| gpt-4-turbo or gpt-3.5-turbo, it's inaccurate to say it's a tool
| for LLMs in general.
| vineetch wrote:
| So you're saying they should ensure compatibility with all LLMs
| on Day 0 so you can avoid a personal "grating" feeling. It's
| called an MVP.
| judicious wrote:
| Definitely very excited to see this be a thing. Genuinely liked
| the approach to make function calls strongly typed and rely on
| functional programming principles.
|
| During my senior year, I worked on a research project very
| similar to this and I'm glad to see this out there for everyone.
| I'd love to connect with the team if possible!
___________________________________________________________________
(page generated 2024-03-06 23:00 UTC)