[HN Gopher] Designing Pareto-optimal RAG workflows with syftr
       ___________________________________________________________________
        
       Designing Pareto-optimal RAG workflows with syftr
        
       Author : roma_glushko
       Score  : 49 points
       Date   : 2025-05-28 14:01 UTC (3 days ago)
        
 (HTM) web link (www.datarobot.com)
 (TXT) w3m dump (www.datarobot.com)
        
       | roma_glushko wrote:
       | A new OSS framework uses multi-objective Bayesian Optimization to
       | efficiently search for Pareto-optimal RAG workflows, balancing
       | cost, accuracy, and latency across configurations that would be
       | impossible to test manually.
        
         | roma_glushko wrote:
         | Useful links:
         | 
         | Github: https://github.com/datarobot/syftr
         | 
         | Paper: https://arxiv.org/abs/2505.20266
        
       | diabolicalrobot wrote:
       | I am a member of the syftr team. Please feel free to ask
       | questions.
        
         | djoldman wrote:
         | Given section A7 in your paper:
         | https://arxiv.org/pdf/2505.20266
         | 
         | ...would it be accurate to say that syftr finds Pareto-optimal
         | choices across cost, accuracy, and latency, where accuracy is
         | decided by an LLM whose assessments are 90% correlated to that
         | of human labelers?
         | 
         | Are there 3 objectives: cost, accuracy, and latency or 2: cost
         | and accuracy?
        
       | andriyvel wrote:
       | looks interesting!
        
         | jstummbillig wrote:
         | and exhausting.
        
       | charcircuit wrote:
       | It sounds impossible to be paretooptimal for complicated
       | problems. How do you know GPT-4o-mini would be optimal. I feel
       | like there is always room on the table for a potential
       | GPT-5o-mini to be more optimal. The solution space of possible
       | gen ai models is gigantic, so we can only improve our solution
       | over time and never find the most optimal one.
        
         | brianbelljr wrote:
         | Yes, maybe theoretically. Practically though you will have to
         | ship your agent with the LLMs that are available today and you
         | will need to pick one. I don't think the authors were trying to
         | solve for like "best forever",probably wasn't their intent. For
         | that I think you would need some kind of proof which sort of
         | says that some kind of theoretical maximum is reached, and a
         | proof like that is not a thing in _most_ applied computer
         | science fields.
        
       | simianwords wrote:
       | Interesting but I'm a bit lost. You are optimising but how do you
       | know the ground truth of "good" and "bad"? Do you manually run
       | the workflow and then decide based on a predefined metric?
       | 
       | Or do you rely on generic benchmarks?
        
         | viraptor wrote:
         | https://github.com/datarobot/syftr/blob/main/docs/datasets.m...
         | 
         | You need custom QA pairs for custom scenarios.
        
       ___________________________________________________________________
       (page generated 2025-05-31 23:00 UTC)