[HN Gopher] AutoChain, lightweight and testable alternative to L...
___________________________________________________________________
AutoChain, lightweight and testable alternative to LangChain
Author : yilu331
Score : 118 points
Date : 2023-07-18 16:47 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| yilu331 wrote:
| AutoChain takes inspiration from LangChain and AutoGPT and aims
| to solve both problems by providing a lightweight and extensible
| framework for developers to build their own agents using LLMs
| with custom tools and automatically evaluating different user
| scenarios with simulated conversations. Experienced user of
| LangChain would find AutoChain is easy to navigate since they
| share similar but simpler concepts.
|
| The goal is to enable rapid iteration on generative agents, both
| by simplifying agent customization and evaluation.
|
| If you have any questions, please feel free to reach out to Yi Lu
| yi.lu@forethought.ai
| pieter wrote:
| I'd really like libraries to put the language they're for in the
| description.
|
| Especially anything AI-related seems to just assume everybody
| uses Python.
| capableweb wrote:
| Yeah, I'm getting bothered by all React libraries not saying
| what language it's for too!
|
| Jokes aside, quick glance at the language pane on GitHub shows
| what languages are being used.
| yilu331 wrote:
| One of the valuable things AutoChain provides is simulated
| conversation evaluation. Testing and tuning those agent is very
| hard and expensive. checkout the evaluation output from
| AutoChain, for example.
|
| "conversation_history": "user: I want to
| change the shipping address for my order", "assistant:
| Can you please provide the order ID?", "user: The order
| ID is 456", "assistant: The order with ID 456 has not
| yet shipped. I can update the shipping address for you. Please
| provide the new shipping address.", "user: The new
| shipping address is 234 Spear St, San Francisco",
| "assistant: The shipping address for your order with ID 456 has
| been successfully updated to 234 Spear St, San Francisco.",
| "user: Thank you for updating the shipping address for my order",
| "assistant: Thank your for contacting"
| "is_agent_helpful": { "rating": 5,
| "reason": "The conversation reaches the expected outcome. The
| user successfully provides the order ID and the new shipping
| address, and the assistant updates the shipping address for the
| order with ID 456. The assistant confirms the successful update
| and thanks the user for contacting." },
| "actions_took": { "tool":
| "check_order_status", "tool_input": {
| "order_id": "456" }, "tool_output":
| { "status_code": 200, "order_id":
| "456", "order_status": "not_shipped",
| "tracking_url": "example.com/456",
| "shipping_address": "301 ivy street san francisco ca"
| } }, { "tool":
| "change_shipping_address", "tool_input":
| { "order_id": "456",
| "new_address": "234 Spear St, San Francisco" },
| "tool_output": { "status_code": 200,
| "order_id": "456", "shipping_address": "234 Spear
| St, San Francisco" } }
| "num_turns": 8, "expected_outcome": "found order status
| and changed shipping address"
| delusional wrote:
| If you're dealing with a set amount of functions (like updating
| shipping address), aren't we just building chatbots again? I
| thought we all agreed that we'd much rather just have a bespoke
| button for updating the shipping address than having to deal
| with a chatbot. The conversation presented here is especially
| vexing as it sounds so mechanical that there's hardly any
| reason to use any AI at all.
| yilu331 wrote:
| LLM today makes it a lot easier to build chatbot without
| decision tree and handle more complex scenarios. What you
| would need to do is providing it sufficient tools to use and
| a scope for what it can do, so that the chat bot doesn't have
| to be mechanical
| EGreg wrote:
| Why do people need LangChain or AutoChain? I don't really get it
| in the first place. Genuinely want to get more information.
|
| It seems to me that this can be easily added ad-hoc by
| programmers in their projects. But the biggest reason I see may
| be standardization of some kind? For interoperable building
| blocks maybe
| minimaxir wrote:
| Relevant discussion:
| https://news.ycombinator.com/item?id=36725982
| yilu331 wrote:
| great discussion. there are a lot of problems with LangChain
| for sure. hopefully we are trying to make it easier for
| exploring different ideas
| reallymental wrote:
| LangChain is just too much, personal solutions are great, until
| you need to compare metrics or methodologies of prompt
| generation. Then the onus is on these n-parties who are sharing
| their resources to ensure that all of them used the same
| templates, they were generated the same way, with the only diff
| being the models these prompts were run on.
|
| So maybe a simpler library like Microsoft's Guidance
| (https://github.com/microsoft/guidance)? It does this really
| well.
|
| Also LangChain has a lot of integrations, that just pop up as
| soon as new API for anything LLM pops up, so that helps with
| new user onboarding as well.
| msp26 wrote:
| I wouldn't rely on guidance. I've been waiting for them to
| approve a commit for something as basic as token counts for
| OAI models for weeks now. It's just not actively developed
| enough for serious use cases.
| reallymental wrote:
| I wouldn't count anything out as of yet, the trend is
| usually that people move towards simpler tools that are
| also widely used by others.
|
| Kind of a chicken-and-egg problem (to be widely used, it
| helps to be simple, but not all simple-to-use products make
| it to being widely used). TensorFlow vs PyTorch anyone?
|
| Now, which will be more likely to be widely adopted? No one
| knows, just keep using both, learning both linguistics and
| hopefully one of them gets enough traction and all your
| effort put into building something with these two doesn't
| go to waste, fingers crossed.
| kmod wrote:
| Same reason javascript needs an is-even library
| jondwillis wrote:
| This has been discussed ad-nauseum in the comment section of
| anything pertaining to LangChain over the past few weeks or
| couple of months.
|
| It's simple: a bit of cargo culting and the real advantage that
| LangChain provides in its interoperability with various other
| useful systems.
| yilu331 wrote:
| with openai function calling, it is definitely getting easier
| and easier to build llm agent into a prototype. There are a few
| benefits of using AutoChain. 1) get started easily; plug and
| play different ideas before building into a product 2)
| AutoChain introduces a novel evaluation framework for
| evaluating LLM agent's ability to perform multi-turn
| conversation under different user scenarios.
|
| In addition, when it comes to prototyping for a specific use
| case, we found it is often more than just calling the model but
| also the orchestration process matters, for example, when
| should LLM agent stop answering questions, fix input argument,
| ask a custom clarifying questions and more.
|
| Hope AutoChain makes your exploration easier and more robust!
| J_Shelby_J wrote:
| I use Langchain for the slack/discord bots I built for my
| current company.
|
| Langchain is a swiss army knife of tools. And while that can be
| a good thing, it's mostly a bad thing when you're trying to
| build something for production. It's also not very flexible...
| when you're trying to do anything beyond the pre-built stuff,
| you have to edit the package itself... which means maintaining
| your own forked version or submitting a PR with your changes.
| For example, I improved upon the main text splitter Langchain
| recommends (https://github.com/ShelbyJenkins/langchain/blob/ca1
| 4d3028a57...), but I just haven't gotten around to doing
| everything required for a PR to add the feature (tests, docs,
| notebooks). So now I have two repos to maintain...
|
| But I also really don't want to take the time to set up a web
| scraper for a dozen different document sources when Langchain
| has tools that will work with a minimal amount of tweaking. So,
| I'm happy to use it for things like that, and happy to
| recommend it for it's community built pocket-knife tools!
| supermdguy wrote:
| I used it to prototype an app, and it was nice to be able to
| get started quickly without worrying about writing prompts or
| writing a vector store integration. But once I settled on the
| components I was using it made more sense to use a custom
| implementation.
| yilu331 wrote:
| exactly, i found the same thing which motivated me to create
| a lightweight version of it for rapid iteration. There are a
| lot of customizations i found needed when i try to wrap a
| practical prototype for LLM agent
___________________________________________________________________
(page generated 2023-07-18 23:00 UTC)