[HN Gopher] How We Broke Top AI Agent Benchmarks: And What Comes...
___________________________________________________________________
How We Broke Top AI Agent Benchmarks: And What Comes Next
Author : Anon84
Score : 149 points
Date : 2026-04-11 19:15 UTC (3 hours ago)
(HTM) web link (rdi.berkeley.edu)
(TXT) w3m dump (rdi.berkeley.edu)
| ggillas wrote:
| This is a phenomenal paper on exploits and hopefully changes the
| way benchmarking is done.
|
| From the paper: We achieved near-perfect scores on all of them
| without solving a single task. The exploits range from the
| embarrassingly simple (sending {} to FieldWorkArena) to the
| technically involved (trojanizing binary wrappers in Terminal-
| Bench), but they all share a common thread: the evaluation was
| not designed to resist a system that optimizes for the score
| rather than the task.
| operatingthetan wrote:
| >hopefully changes the way benchmarking is done.
|
| Yeah the path forward is simple: check if the solutions
| actually contain solutions. If they contain exploits then that
| entire result is discarded.
| Leynos wrote:
| Also, fuzz your benchmarks
| siva7 wrote:
| Could it really be that not only we vibeslop all apps
| nowadays but also don't care to even check how ai solved a
| benchmark it claimed solved?
| operatingthetan wrote:
| Probably a more interesting benchmark is one that is scored
| based on the LLM finding exploits in the benchmark.
| SpicyLemonZest wrote:
| Frontier model developers _try_ to check for memorization.
| But until AI interpretability is a fully solved problem,
| how can you really know whether it actually didn 't
| memorize or your memorization check wasn't right?
| retinaros wrote:
| Every ai labs train on the test set. That is a big part of
| why we see benchmark climbing from 1% to 30% after a few
| models iterations
| ZeroGravitas wrote:
| In human multiple choice tests they sometimes use negative
| marking to discourage guessing. It feels like exploits should
| cancel out several correct solutions.
| lambda wrote:
| Unfortunately, very few LLM benchmarks do this. LLMs get
| such high scores on many benchmarks because there's no
| difference between answering "I don't know" as giving a
| made up answer, and made up answers can improve the score
| some of the time, so by chasing higher benchmark numbers on
| these kinds of benchmarks, the labs are prioritizing
| guessing over accuracy.
|
| The Artificial Analysis Omniscience benchmark does penalize
| guessing, so it actually helps you determine which LLMs are
| likely to just guess rather than telling you they don't
| know. Only a very few of the frontier models actually score
| higher than 0 on this, where 0 means that it's equally
| likely to return a correct answer as it is to return a
| hallucination on factual questions.
| zer00eyz wrote:
| 2024: Industry group invalidates 2,600 official Intel CPU
| benchmarks -- SPEC says the company's compiler used unfair
| optimizations to boost performance
| https://www.tomshardware.com/pc-components/cpus/spec-invalid...
|
| 2003: Nvidia accused of cheating in 3DMark 03
| https://www.gamespot.com/articles/nvidia-accused-of-cheating...
|
| It's almost like the benchmarks were designed with zero
| understanding of the history of benchmark manipulation.
|
| I like what LLM's are doing and providing. But the industry as
| a whole seems to live in a vacuum that ignores so much of the
| hard lessons that have been learned over the last 50 years of
| computing. It is doing itself a disservice.
| irishcoffee wrote:
| > It's almost like the benchmarks were designed with zero
| understanding of the history of benchmark manipulation.
|
| I wonder if this common? We should call it Goodharts law
| while someone does the research on how common this is.
|
| For real, I've assumed from the jump these things were all
| gamed, with the amount of money on the line.
| bee_rider wrote:
| What was the cheat in the 2024 Intel situation? The
| TomsHardware article and the Phoronix article they linked
| were quite vague. (Not to say I have any doubts, just
| curious, hadn't heard of this one).
| SlinkyOnStairs wrote:
| > hopefully changes the way benchmarking is done
|
| The purpose of a system is what it does.
|
| AI companies want adcopy, not legitimate benchmarks. Even this
| very paper will be twisted into a means to that end. "Oooo, AI
| is exploiting our benchmarks. Scary alignment problem!!!one!
| Our AI is so good we can't contain it, INVEST NOW!"
| charcircuit wrote:
| I always assumed that these benchmarks would happen in a sandbox.
| I'm surprised that no one realized this sooner.
| ModernMech wrote:
| I'm surprised anyone took them seriously in the first place.
| operatingthetan wrote:
| We need good benchmarks or we are just left following the
| hype train.
| subulaz wrote:
| a LOT of the people who love benchmarks are middle management
| hard-selling GenAI/LLM as magic tech sauce to vaguely
| technical executives who only want to know about the money
| aka headcount savings they so desperately desire.
|
| their collective butts are already glued to the hype train as
| they chase numbers they (often) manufactured to justify the
| latest round of tech spend.
|
| lots of good use cases out there - like the incredible
| progress with medical imaging analysis or complex system
| models for construction - and lots of crap use cases that
| need benchmarks to cosplay relevance.
| tredre3 wrote:
| What else can people do? Try the dozen of commercial
| offerings themselves? Okay I suppose that's doable, you task
| one engineer to try them one by one for one month. But then
| the next model drops and you start all over again...
|
| But then what about local models? You have hundreds of
| variations to test yourself. It's simply not doable unless
| it's your full time hobby.
|
| You need benchmarks to at least separate the cream from the
| crop, so you're left with only a few choices to test
| yourself.
| lnrd wrote:
| I'm honestly confused by the design of SWE-bench and why is
| considered reliable.
|
| It's based on existing GitHub PRs and Issues, the full dataset is
| on HuggingFace and is one year old now. All frontier models 100%
| have those issues and PRs in their training data so obviously
| they are good at reproducing fixes for them when confronted with
| the same codebase and similar requests. Am I missing something?
| How is this considered the most reliable benchmark?
| SpicyLemonZest wrote:
| Frontier model developers do not consider SWE-bench to be
| reliable. OpenAI announced in February
| (https://openai.com/index/why-we-no-longer-evaluate-swe-
| bench...) that they consider it hopelessly contaminated,
| advocating for a new version SWE-bench Pro that was published
| more recently. (They seem to believe that even the publicly
| accessible part of the SWE-bench Pro problem set will be more
| resistant to training set contamination issues in the future,
| for reasons that to be honest I don't really understand.)
| oliver236 wrote:
| what are the point of benchmarks?
| andai wrote:
| If there was not benchmark, number would not go up.
| esafak wrote:
| Are you serious? To help you pick a model.
| danslo wrote:
| If only the blog itself wasn't written by AI?
|
| >No reasoning. No capability. Just exploitation of how the score
| is computed.
|
| _shudder_
| gaythread wrote:
| Modern day HN is overrun with AI posts.
| cpldcpu wrote:
| Yes, marks of AI all over the place. Also the SVGs.
|
| >No solution written, 100% score.
|
| Its weird. Turns out that hardest problem for LLMs to really
| tackle is long-form text.
| basch wrote:
| Maybe in one shot.
|
| In theory I would expect them to be able to ingest the corpus
| of the new yorker and turn it into a template with sub-
| templates, and then be able to rehydrate those templates.
|
| The harder part seems to be synthesizing new connection from
| two adjacent ideas. They like to take x and y and create x+y
| instead of x+y+z.
| sidpatil wrote:
| Someone here mentioned a whole ago that the labs deliberately
| haven't tried to train these characteristics out of their
| models, because leaving them in makes it easier to identify,
| and therefore exclude, LLM-generated text from their training
| corpus.
| blymphony wrote:
| But it's odd that these characteristics are the same across
| models from different labs. I find it hard to believe that
| researchers across competing companies are coordinating on
| something like that.
| alexchantavy wrote:
| I wonder what college freshman-level writing classes are
| teaching about writing voice and AI. The tell-tale patterns are
| pretty frustrating to read.
| stefan_ wrote:
| Whatever classes these guys took, they skipped the one on
| scientific misconduct.
| jmward01 wrote:
| Not really on the topic, but I have wondered if we need a
| different type of test to help find model architecture potential.
| Standardized training sets followed by testing to see the
| potential curves of a model. train on x, test, add y, test, add
| z, test. At each increment you see how well the model is
| absorbing the information and extrapolate how well that
| architecture may do if more fully trained.
| jgalt212 wrote:
| The real question is how to close to VW and Deiselgate are these
| offenses? And what exposure do these companies have? I would
| assume securities fraud, if only because Matt Levine says
| everything is securities fraud.
| SoKamil wrote:
| The more research on this topic is created, the more knowledge
| how to game them will be stored in future training data. And
| since it comes from university, it is ranked higher in data
| corpus. It sounds like a self fulfilling prophecy.
| abirch wrote:
| Damned old Goodhart's Law: "When a measure becomes a target, it
| ceases to be a good measure".
|
| https://en.wikipedia.org/wiki/Goodhart%27s_law
| lukev wrote:
| I think we should all consider the _possibility_ that _part_ of
| the reason Anthropic hasn 't immediately released Mythos is that
| it would be slightly disappointing relative to the benchmark
| scores.
| eiens wrote:
| The models don't get better on every dimension as they scale up
| - there's trade offs.
|
| I'm convinced specialised models are the way but this means
| writing off the investment in existing assets which they won't
| do for obvious reasons.
| bbcc90 wrote:
| Yes good evals are really hard - that's not really news.
|
| This team is doing a good job. They use problems that were
| created in last 30days to avoid training set leakage.
| https://swe-rebench.com/
| czhu12 wrote:
| I wonder if this puts into question the mythos benchmark which
| smashed basically all coding benchmarks to a staggering degree.
| mzelling wrote:
| This is an interesting catalog of vulnerabilities, but I'm not
| sure how groundbreaking the main insight is.
|
| Evaluating AI models has always relied largely on trust. If you
| want to game the benchmarks, you can. Simply train on your test
| data.
|
| When an AI agent has autonomous control over the same computing
| environment where its scores are recorded, it's not surprising
| that it can, in principle, falsify its scores. A more interesting
| question would be whether agents behave in this way
| automatically, without manual tuning by the researcher.
|
| That said, the main takeaway of "don't trust the number, trust
| the methodology" is valid. It's already a truism for researchers,
| and spreading the word to non-researchers is valuable.
| glerk wrote:
| Benchmaxxing like this is trivial to implement, but also trivial
| to detect since submitting to these benchmarks usually involves
| publishing complete execution traces. You can be sure that
| hardcoding a curl command that fetches the test results will get
| caught.
|
| There have been more subtle attempts, recently OpenBlocks was
| caught cheating (hiding previous run traces inside the harness)
| and removed from the TerminalBench leaderboard [1]
|
| Contamination is a big issue and imo static tests like this can't
| avoid Goodhart's law. But blatant cheating will eventually get
| caught and the reputational damage is deterrent enough.
|
| [1]: https://x.com/seconds_0/status/2032643333575815291
___________________________________________________________________
(page generated 2026-04-11 23:00 UTC)