[HN Gopher] Evaluating RAG for large scale codebases
       ___________________________________________________________________
        
       Evaluating RAG for large scale codebases
        
       Author : GavCo
       Score  : 35 points
       Date   : 2025-02-14 08:29 UTC (14 hours ago)
        
 (HTM) web link (www.qodo.ai)
 (TXT) w3m dump (www.qodo.ai)
        
       | jimminyx wrote:
       | Conceptually, LLM-as-a-judge doesn't feel like it should work --
       | it's like asking a student to grade their own homework. it's very
       | unintuitive for me that it actually seems to work pretty well
        
         | petesergeant wrote:
         | LLM as a judge isn't telling you if something is right or
         | wrong, it's telling you if a given generation is normal or an
         | aberration, according solely to the data the model was trained
         | on. This is one reason LLMs prefer their answers to answers
         | from other LLMs.
         | 
         | "LLMs as a judge" is more about addressing the failure mode of
         | auto-regressive (one-token at a time) generation letting an LLM
         | lead itself astray due to its previous choices, rather than
         | telling you any general truth.
         | 
         | Finally I'd note that in every maths challenge I ever completed
         | as a student, you were _strongly_ advised to go back and check
         | your own work at the end if you had time left over, and for me
         | this usually led to me catching things I 'd missed the first
         | time.
        
           | taurknaut wrote:
           | > Using LLMs to judge correctness
           | 
           | It seems their pr is willing to make much stronger claims
           | than you will.
        
             | procaryote wrote:
             | PR's view of truth is "can I convince legal we probably
             | won't lose if we're sued for saying this", which is a
             | fairly weak form of truth
        
         | tinco wrote:
         | No, it's like a teacher's assistant using a answer sheet
         | provided by the professor to grade exams. A common practice at
         | the university I went to and accurate enough in most cases.
         | 
         | The problem of matching an answer to a given answer is a lot
         | simpler than generating an answer. Especially for an LLM which
         | has language transformation as one of its core competences.
        
         | mirekrusin wrote:
         | ...pretty well according to student.
         | 
         | It's student writing judge evaluation questions for himself as
         | judge, judging judge (himself) and evaluating himself later
         | using own judgement (as judge).
         | 
         | 1 student all the way down.
         | 
         | If it thinks eating concrete makes you stronger, it's going to
         | think that and give green light end to end.
        
         | dinfinity wrote:
         | One of the most effective things to do in coding (and many
         | other things) is to always do a self-review: Before you turn
         | over a changeset for review, review it as if you were reviewing
         | some one else's code.
         | 
         | It sounds superfluous, but it works very well as it saves time
         | and frustration by allowing you to quickly fix the stuff that
         | wouldn't pass review anyway.
         | 
         | The LLM as a judge-concept works in a similar way. Instead of
         | "give a good answer", the task and perspective is "does this
         | make sense?", which is very different.
        
       | namanyayg wrote:
       | Who's data are they training on? Are they storing and using all
       | customer data?
        
         | mirekrusin wrote:
         | It's mentioned in the article that they used their own repos.
        
       | 33a wrote:
       | If the self evaluation makes it better, then why not do the self
       | evaluation as part of the normal RAG workflow?
        
       ___________________________________________________________________
       (page generated 2025-02-14 23:01 UTC)