[HN Gopher] $500 GPU outperforms Claude Sonnet on coding benchmarks
       ___________________________________________________________________
        
       $500 GPU outperforms Claude Sonnet on coding benchmarks
        
       Author : yogthos
       Score  : 21 points
       Date   : 2026-03-26 17:31 UTC (5 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | memothon wrote:
       | I'm always skeptical because you can make it pass the benchmarks,
       | then you use it and it is not practically useful unlike an
       | extremely general model.
       | 
       | Cool work though, really excited for the potential of slimming
       | down models.
        
         | yogthos wrote:
         | You obviously have to try it out to see how it works for you,
         | but the trick they use is pretty clever. When you ask an AI to
         | write code, it doesn't always get it right. Sometimes the code
         | has bugs, sometimes it misunderstands the problem entirely. A
         | naive way to address that is to generate a few solutions and
         | test each one. The odds that at least one works go way up.
         | ATLAS generates multiple attempts, running each through a test
         | suite. Each retry also gets told what went wrong with the
         | previous attempt, so it can try to avoid the same mistake.
         | 
         | But this can be pretty slow since you have to run the code in
         | an isolated environment, check the outputs, wait for it to
         | finish. Doing that for every candidate quickly adds up. So
         | ATLAS has another shortcut for avoiding unnecessary testing.
         | Instead of simply generating solutions and testing all of them,
         | it tries to predict which one is most likely correct before
         | running any tests.
         | 
         | ATLAS also asks the model for an embedding of what it just
         | wrote which acts as a fingerprint. Two similar pieces of code
         | will produce similar fingerprints. A well-written, confident
         | solution will produce a different fingerprint than a confused,
         | buggy one.
         | 
         | These fingerprints get fed into a separate, much smaller neural
         | network called the Cost Field. This little network was trained
         | ahead of time on examples where they already knew which
         | solutions were correct and which were wrong. It learned to
         | assign a score to each fingerprint. Correct solutions get a low
         | score and incorrect ones get a high one.
         | 
         | So the process is to generate multiple solutions, get their
         | fingerprints, score each one, and pick the lowest. Only that
         | one gets tested. The Cost Field picks correctly about 88% of
         | the time according to the repo.
        
       ___________________________________________________________________
       (page generated 2026-03-26 23:01 UTC)