[HN Gopher] Show HN: Nomadic - Minimize RAG Hallucinations with ...
___________________________________________________________________
Show HN: Nomadic - Minimize RAG Hallucinations with 1
Hyperparameter Experiment
Hey HN! Mustafa, Lizzie, and Varun here from NomadicML
(https://nomadicml.com). We're excited to show you Nomadic
(https://github.com/nomadic-ml/nomadic): a platform focused on
parameter search to continuously optimize AI systems. Here's a
simple demo notebook where you get the best-performing,
statistically significant configurations for your RAG -- and
improve hallucination metrics by 4X in just 5 minutes -- with a
single Nomadic experiment: https://tinyurl.com/4xmaryyw Our
lightweight library is now live on PyPI (`pip install nomadic`).
Try one of the README examples :) Input your model, define an
evaluation metric, specify the dataset, and choose which parameters
to test. Nomadic emerged from our frustration with existing HPO
(hyperparameter optimization) solutions. We heard over and over
that for the sake of deploying fast, folks resort to setting HPs
through a single, expensive grid search or better yet, intuition-
based "vibes". From fine-tuning to inference, small tweaks to HPs
can have a huge impact on performance. We wanted a tool to make
that "drunken wander" systematic, quick, and interpretable. So we
started building Nomadic - our goal is to create the best parameter
search platform out there for your ML systems to keep your
hyperparameters, prompts, and all aspects of your AI system
production-grade. We started aggregating top parameter search
techniques from popular tools and research (Bayesian Optimizations,
cost-frugal flavors). Among us: Built Lyft's driver earnings
platform, automated Snowflake's just-in-time compute resource
allocation, became a finalist for the INFORMS Wagner Prize (top
prize in industrial optimization), and developed a fintech fraud
screening system for half a million consumers. You might say we
love optimization. If you're building AI agents / applications
across LLM safety, fintech, support, or especially compound AI
systems (multiple components > monolithic models), and want to
deeply understand your ML system's best levers to boost performance
as it scales - get in touch. Nomadic is being actively developed.
Up next: Supporting text-to-SQL pipelines (TAG) and a Workspace UI
(preview it at https://demo.nomadicml.com). We're eager to hear
honest feedback, likes, dislikes, feature requests, you name it. If
you're also a optimization junkie, we'd love for you to join our
community here https://discord.gg/PF869aGM
Author : mustafabal
Score : 50 points
Date : 2024-09-05 18:14 UTC (4 hours ago)
| varunkrishnan17 wrote:
| Hi I'm Varun - one of the cofounders of Nomadic!
|
| Been a pleasure to work with Mustafa and Lizzie on this!
| Hopefully you can solve a pain point I personally have had for so
| long - how can you easily verify that your model continues to
| perform well?
| rnvarma wrote:
| For my company, we don't have complex chains, but generally are
| giving a large context and looking to get structured outputs.
| Curious how this could help with that? We don't currently use any
| eval frameworks.
| varunkrishnan17 wrote:
| That's a great use case of Nomadic! We support many Eval
| frameworks in the optimization, but one is a LLM-as-a-Judge
| model, where you can input custom weights based on your metrics
| of interest! Adhering to a proper structure could be one of
| them :-)
| altairmn wrote:
| Our customers use our platform to build low-latency voice and
| video pipelines. They utilize RAG in voice bots to improve
| response accuracy.
|
| Is it possible to programmatically interface with Nomadic's
| hyperparameter search through an authenticated endpoint, with the
| ability to generate user-specific tokens for secure access?"
| mustafabal wrote:
| Certainly!
|
| The Nomadic SDK supports 1st-party integrations with various
| open & closed-source ML/LLM providers. These are done through
| authenticated endpoints for interfacing securely with your
| models. Also, as noted in the Custom Evaluation section of our
| docs (https://docs.nomadicml.com/features/evaluations), you can
| provide your custom objective_functions and detail your model
| access logic, which may include custom authentication & access
| rules. A sample of this is present in our "Basic RAG" cookbook
| (link: https://colab.research.google.com/drive/1rv2f-qxgoN_eVDF
| u6Um...).
|
| When integrated with the upcoming Nomadic Workspace, you can
| obtain your Nomadic API key and sync your local Nomadic models,
| experiments & experiment results with our managed service. The
| demo of this model/experiment/experiment result visualizatio is
| live at https:demo.nomadicml.com, please check it out and let
| us know your thoughts!
| add-sub-mul-div wrote:
| Lots of grassroots interest in this from a flood of new accounts
| created in the last few hours.
|
| ========
|
| baileyw6 2 hours ago [flagged] [dead] | prev | next [-] excellent
| work!
|
| r0sh 3 hours ago [flagged] [dead] | prev | next [-] cracked team!
|
| mlw14 3 hours ago [flagged] [dead] | prev | next [-] Interesting
| library, is it like unit testing for RAGs? Can't wait to try it
| out!
|
| lncheine 2 hours ago [flagged] [dead] | prev | next [-]
| Interesting library, can't wait to try it out!
|
| Linda_ll 2 hours ago [flagged] [dead] | prev | next [-] Congrats
| on the launch! Excited for what's to come :)
|
| bmountain17 3 hours ago [flagged] [dead] | prev | next [-] Great
| new platform to boast AI performance, can't wait to try the
| Python library!
|
| jjBailey 1 hour ago [flagged] [dead] | prev | next [-] Cool
| library, I'll test it out
|
| sidkapoor39 3 hours ago [flagged] [dead] | prev | next [-]
| Congrats on the launch! Excited to see how this streamlines
| Hyperparameter optimization. Keep up the great work!
|
| brucetry 1 hour ago [flagged] [dead] | prev | next [-] Ver
| interesting, similar to unit test for RAGs? Love to try it out
|
| jjBailey 1 hour ago [flagged] [dead] | prev | next [-] Very
| interesting library!! Can't wait to try it!
|
| luxxxxx 1 hour ago [flagged] [dead] | prev | next [-] Interesting
| library! Is it like unit testing for RAGs? Can't wait to try it
| out!
|
| kangjl888 2 hours ago [flagged] [dead] | prev | next [-] Huge
| congratulations to the NomadicML team on the launch of Nomadic!
| The platform looks like a game-changer for optimizing AI systems,
| excited to see how it transforms hyperparameter search for the
| community.
|
| nishsinha2345 21 minutes ago [flagged] [dead] | prev | next [-]
| Excited to try out this library! would this help make unit
| testing easier? Or be used instead of unit testing?
|
| greysongy5 19 minutes ago [flagged] [dead] | prev | next [-] Wow,
| this seems like it would really help automated RAG testing. What
| are the top use cases today?
|
| sidvijay10 5 minutes ago [flagged] [dead] | prev [-] We're
| looking for a RAG testing framework for searching UGC. So far
| we've just been running evals manually w/o a library. Will try
| out Nomadic and see if it's more convenient.
| _eric_z_lin wrote:
| This looks like a really useful tool for keeping AI systems
| optimized, especially as models and data evolve over time. I'm
| curious, have you considered how Nomadic might integrate into
| CI/CD pipelines? It seems like it could be valuable for
| automatically re-tuning parameters and ensuring performance
| doesn't degrade with new model versions or data updates. Any
| plans for features that would support this kind of continuous
| optimization workflow?
___________________________________________________________________
(page generated 2024-09-05 23:01 UTC)