[HN Gopher] Qodo CLI agent scores 71.2% on SWE-bench Verified
___________________________________________________________________
Qodo CLI agent scores 71.2% on SWE-bench Verified
Author : bobismyuncle
Score : 130 points
Date : 2025-08-12 11:05 UTC (11 hours ago)
(HTM) web link (www.qodo.ai)
(TXT) w3m dump (www.qodo.ai)
| gronky_ wrote:
| I've been running a bunch of coding agents on benchmarks recently
| as part of consulting, and this is actually much more impressive
| than it seems at first glance.
|
| 71.2% puts it at 5th, which is 4 points below the leader (four
| points is a lot) and just over 1% lower than Anthropic's own
| submission for Claude Sonnet 4 - the same model these guys are
| running.
|
| But the top rated submissions aren't running production products.
| They generally have extensive scaffolding or harnesses that were
| built *specifically for SWE bench*, which kind of defeats the
| whole purpose of the benchmark.
|
| Take for example Refact which is at #2 with 74.4%, they built a
| 2k lines of code framework around their agent specifically for
| SWE bench (https://github.com/smallcloudai/refact-bench/). It's
| pretty elaborate, orchestrating multiple agents, with a debug
| agent that kicks in if the main agent fails. The debug agent
| analyzes the failure and gives insights to the main agent which
| tries again, so it's effectively multiple attempts per problem.
|
| If the results can be reproduced "out-of-the-box" with their
| coding agent like they claim, it puts it up there as one of the
| top 2-3 CLI agents available right now.
| szundi wrote:
| According to your experience with this model, is it just
| trained for the benchmark or these points are actually
| representing the performance?
| energy123 wrote:
| What are the typical context lengths in SWE-bench problems?
| Does it partly measure performance in the 64-128k context
| range?
| dimitri-vs wrote:
| IIRC the SWE bench dataset gives you the full repo snapshot +
| the issue text, the evaluation pipelines typically run some
| kind of retriever (eg. grep, BM25) to pick a subset of files
| to place in the model's context. They provided context is
| usually limited up to ~50k tokens.
| whymauri wrote:
| This is what the rows look like:
|
| https://huggingface.co/datasets/princeton-nlp/SWE-
| bench_Veri...
|
| Its up to your retrieval system/model to selectively hunt for
| relevant context. Here's a few critiques of the benchy:
|
| https://x.com/brhydon/status/1953648884309536958
| thinkingtoilet wrote:
| This is classic Goodhart's law. "When a measure becomes a
| target, it ceases to be a good measure"
|
| https://en.wikipedia.org/wiki/Goodhart%27s_law
| ambicapter wrote:
| It's really not that hard to _not_ build a custom bench setup
| to game the benchmark instead of just using your product
| straight out of the box, though.
| jasonjmcghee wrote:
| Right. Building a custom setup is blatant- that will wildly
| overfit.
|
| But let's say a group uses it as a metric as part of CI and
| each new idea / feature they create runs against SWE bench.
| Maybe they have parameterized bits and pieces they adjust,
| maybe they have multiple candidates datasets for fine
| tuning, maybe they're choosing between checkpoints.
|
| This will also end up overfitting - especially if done
| habitually. It might be a great metric and result in a more
| powerful overall model. Or it might not.
| VikingCoder wrote:
| Right, other than financial pressure. Which is, of course,
| immense.
| clutchdude wrote:
| Also see the VW dieselgate and numerous other "gaming the
| system" examples.
| kelipso wrote:
| A specific setup for the benchmark is just plain cheating,
| not Goodhart's law.
| eddd-ddde wrote:
| I think multiple attempts are completely understandable and
| even expected? How is that defeating the purpose of the
| benchmark?
| gronky_ wrote:
| It's a pass@1 benchmark. When submitting you need to check a
| box that there was only 1 attempt per problem. See here for
| example: https://github.com/SWE-bench/experiments/pull/219
|
| Building multiple attempts into your agent is stretching the
| rules, even if technically it's acceptable
| terminalshort wrote:
| From my perspective as a potential user the number of
| attempts is the number of times I have to tell it what to
| do. If you have an agent that makes a single attempt and is
| 60% accurate vs another that makes 5 attempts and is 80%
| accurate, why would you care that each individual attempt
| of the 2nd model is less accurate than the first?
| mcintyre1994 wrote:
| I think it depends on "But the top rated submissions
| aren't running production products" It sounds like
| they're shipping a product without the debug agent/try-
| again logic, and that's just for the benchmark, so you
| wouldn't get the performance they get as a user.
| gronky_ wrote:
| This ok from your perspective then?
|
| def make_pass@1_agent(agent, n): def
| retry_agent(problem): for attempt in
| range(n): result = agent(problem)
| if result.success: return result
| return result return retry_agent
| gronky_ wrote:
| Keep in mind that this isn't about users - the top agents
| on the leaderboard aren't running an actual product on
| the benchmark.
|
| If they are running their production product as is, then
| of course whatever is built into the product is fine.
| DougBTX wrote:
| Absolutely fine, as long as the success flag is predicted
| by the model ensemble under test. That's how Claude Code
| works for example, it will continue to iterate until
| success (or it will give up with failure at a certain
| point).
| terminalshort wrote:
| Definitely wouldn't have written the code that way, but
| yes, if (and this is a massive "if") the agent has an
| accurate and meaningful way to determine which way to set
| the success boolean. The obvious caveat would be if n
| needed to be large enough to set the costs higher than I
| am willing to pay for the additional performance or it
| makes it take longer than I'm willing to wait.
|
| Think of the agent like an employee. If he delivers the
| code within the expected time and to the expected quality
| standards, his process of getting there means almost
| nothing. Do I care if he tried 4 different approaches
| along the way and threw out the first 3? Not a bit.
| whymauri wrote:
| Papers have been doing rollouts that involve a model
| proposing N solutions and then self-reviewing to choose
| the best one (prior to the verifier). So far, I think
| that's been counted as one pass.
| radarsat1 wrote:
| I was thinking about this recently with respect to how
| many agent systems now let you specify a smaller/faster
| model for easier tasks and a bigger model for harder
| tasks.
|
| It's interesting to think about what the trade-offs are.
| Assuming the system can properly classify a task as easy
| or hard (big "if" but I guess there are ways), there is
| nonetheless more to think about, depending on your
| pricing plan.
|
| For subscription pricing, I guess you don't really care
| which model runs and in fact it's hard to find a reason
| to ever run the smaller model, so choosing between the
| models is more in the provider's interests for cost
| efficiency.
|
| But for pay-per-use pricing, But if you have a bigger
| model that can get the answer right 80% of the time, and
| a smaller model that can handle smaller changes and get
| things right 60% of the time but correct its mistakes,
| then the system should try to run it on as many tasks as
| possible to save you money.. but in the end if ends up
| having to make a lot of corrections, then maybe you end
| up needing _more_ total requests than the larger model.
| In that case maybe it 's actually _cheaper_ to run the
| larger model, if it takes fewer requests.
|
| So I wonder how that kind of trade-off could be
| effectively calculated. I guess if you can figure out
| when "retries" happen you can count them and do some
| statistics on which model is more likely to work out in
| fewer shots. It's pretty complicated though, when you
| start to think about it in detail.
|
| I do wonder if even having BOTH the smaller and bigger
| model make hypotheses, and try the smaller model's idea
| first, then if it fails, try the bigger model's idea,
| might be the way to go.
| oblio wrote:
| https://github.com/auchenberg/volkswagen
| Roritharr wrote:
| Finally someone mentions Refact, I was in contact with the
| team, rooting for them really.
| bluelightning2k wrote:
| Just looked them up. Their pricing is around buying "coins"
| with no transparency as to what that gets. Hard pass
| Roritharr wrote:
| You realize that you can self-host their stuff?
| https://github.com/smallcloudai/refact
| terminalshort wrote:
| Is there something in this multi-agent approach that makes the
| setup more specific to just the test at hand and less general
| to real engineering tasks? If not, then this multi-agent system
| will just become what you get out of the box in a future
| product. Multiple attempts per problem (as long as there's no
| human intervention or selection between them) is a perfectly
| fine approach for agents because that's not an issue from the
| perspective of an engineer using the product. A single agent is
| already a multi-step usage of LLMs and it sounds like this is
| just another meta level of that.
| ai-christianson wrote:
| One thing with SWE bench is making sure there's zero leakage of
| information into the LLM context.
|
| I.e. the agent cannot even know which tests are failing.
|
| It has to both fix the issue based just on the issue text _and
| fix it in the specific way the unit test, which it cannot see,
| expects_.
|
| For this reason I find the benchmark a little disconnected from
| the reality of software engineering.
| orangebread wrote:
| I've been using Warp for the past few weeks and it's been
| incredibly impressive over other agentic coding
| services/platforms. Curious how Qodo stacks up.
| lightbendover wrote:
| When I tried warp I was convinced that was where the industry
| was going (agents as terminal replacement), but it felt a bit
| too heavy to me so I haven't been using it lately. Still think
| all things will converge on terminal and browser replacement.
| rs186 wrote:
| So this is from the same company that wrote a blog post with
| sentences that don't even make sense:
|
| https://news.ycombinator.com/item?id=44833929, my comment
| https://news.ycombinator.com/item?id=44835939
| khalic wrote:
| We need some international body to start running these tests... I
| just can't trust these numbers any longer. We need a platform for
| this, something at least we can get some peer reviews
| redman25 wrote:
| That sounds like an interesting idea to me. It would at least
| resolve the problem of companies gaming the metric.
|
| Another approach might be the LiveBench approach where new
| tests are released on a regular basis.
| mupuff1234 wrote:
| I'm curious how do these LLM wrapper companies think they'll
| survive long term - especially coding related wrappers.
|
| I could understand focusing on a niche business use case, but
| coding is a main focus of the foundation models themselves.
| M4R5H4LL wrote:
| Labeling them as "wrappers" and "niche business" indicates a
| strong cognitive bias already. Value can be created on both
| sides of the equation.
| dgfitz wrote:
| How so? They are wrappers, and it is niche.
| choilive wrote:
| Wrappers are a bit pejorative and reductive - everything is
| a wrapper around something else.
| dgfitz wrote:
| If everything is a wrapper around something else, how can
| the description be a pejorative?
| itamarcode wrote:
| Unlike most SWE bench submissions, Qodo Command one uses the
| product directly.
|
| I think that the next step is getting an official "checked" mark
| by the SWE bench team
| whymauri wrote:
| I feel like the bash only SWE Bench Verified (a.k.a model +
| mini-swe-agent) is the closest thing to measuring the inherent
| ability of the model vs. the scaffolding.
|
| https://github.com/SWE-agent/mini-swe-agent
| NitpickLawyer wrote:
| There's swe-rebench, where they take "bugs/issues" by date,
| and you can drag a slider on their top scores to see issues
| solved _after_ the model was released (obviously only truly
| working for open models).
| OldGreenYodaGPT wrote:
| Was using their bot for code review for last 2 years but just
| dropped it for BugBot
| esafak wrote:
| If Qodo is reading: please compare your efficiency too. Run some
| tasks on various agents using the same models, and report the
| cost.
| zuzuen_1 wrote:
| Does anyone have a benchmark on the effectiveness of using
| embeddings for mapping bug reports to code files as opposed to
| extensive grepping as Qodo, Cursor and a number of tools I use do
| to localize faults?
| zuzuen_1 wrote:
| I would be more interested in Qodo's performance on the swe-
| bench-multilingual benchmark. Swe-bench-verified only includes
| bugs related to python breakages.
|
| The best submission is swe-bench-multilingual is Claude 3.7
| Sonnet which solves ~43% of the issues in the dataset.
| OldfieldFund wrote:
| do we know anything about the size of the model? I can't find the
| answer.
| afro88 wrote:
| If Qodo are reading this: please introduce a plan that isn't for
| teams or enterprise. A "pro" plan for individuals who want more
| than 250 credits per month.
___________________________________________________________________
(page generated 2025-08-12 23:01 UTC)