[HN Gopher] AI2: Open Coding Agents
___________________________________________________________________
AI2: Open Coding Agents
Author : publicmatt
Score : 92 points
Date : 2026-01-27 17:17 UTC (5 hours ago)
(HTM) web link (allenai.org)
(TXT) w3m dump (allenai.org)
| jauntywundrkind wrote:
| Awesome stuff. Output speed looks crazy fast too.
|
| I wonder if this indeed will start prompting more language
| specific work.
|
| Afaik training still requires not just looking at sample code but
| also being able to write loss functions being able to have
| problems the AI can work at. That seems hard.
|
| One random thought, are there training styles of just deleting
| some code from "good" projects then making the AI make it work
| again?
| CuriouslyC wrote:
| The technique people use is to capture PR diffs from public
| repos and extract the tests then use that to see if agents can
| reconstruct the patch that satisfies the tests.
| ahmadyan wrote:
| Claims in the article are incorrect. They conveniently ignore
| Meta CWM models, which are open-sourced [1] and open-weight [2]
| and are at 65% SWE-bench verified (with TTS) and 54% pass@1 and
| the same size (32B dense). So claims like "surpassing prior open-
| source state-of-the-art coding models of comparable sizes and
| context lengths" and conveniently leaving out the previous OSS
| SOTA out of your eval tables are ... sketch.
|
| [1]https://github.com/facebookresearch/cwm
| [2]https://huggingface.co/facebook/cwm
| philipkglass wrote:
| The difference is that the Allen Institute models have open
| training data, not just open code and weights. Meta doesn't
| share the training data you would need to reproduce their final
| models. For many uses open-weight models are nearly as good,
| but for advancing research it's much better to have everything
| in the open.
| kevmo314 wrote:
| Reading their paper, it wasn't trained from scratch, it's a
| fine tune of a Qwen3-32B model. I think this approach is
| correct, but it does mean that only a subset of the training
| data is really open.
| mhitza wrote:
| The linked open weight disallows commercial, and is only
| licensed for research purpose
| ethan_l_shen wrote:
| Hey! These are great observations. So first, while TTS can
| improve performance, we wanted to evaluate the raw capability
| of our model. This meant generating only one rollout per
| evaluation instance, which follows other papers in the space
| like SWE-smith and BugPilot. In addition, TTS adds extra
| inference cost and is reliant on how rollouts are ranked, two
| confounding factors for deployable models where memory and
| inference speed are extremely important.
|
| Following that line of reasoning, context length is another
| very large confounding factor. Longer context lengths improve
| performance - but also result in enormous increases in KV cache
| size and memory requirements. We decide to control for this in
| our paper and focus at the 32K context length for 32B size
| models, a context length that already pushes the bounds of what
| can be "deployable" locally.
|
| Still, we evaluate at 64K context length using YARN and are
| able to outperform CWM's 54% performance (non TTS), which it
| achieves using 128K context, a substantial increase over what
| we use. This is also pretty significant because we only ever
| train at 32K context, but CWM trains for a full 128K.
| khimaros wrote:
| it's great to see this kind of progress in reproducible weights,
| but color me confused. this claims to be better and smaller than
| Devstral-Small-2-24B, while clocking in at 32B (larger) and
| scoring more poorly?
| ethan_l_shen wrote:
| Hey! We are able to outperform Devstral-Small-2-24B when
| specializing on repositories, and come well within the range of
| uncertainty with our best SERA-32B model. That being said, our
| model is a bit larger than Devstral 24B. Could you point out
| what in the paper gave the impression that we were smaller? If
| theres something unclear we would love to revise
| khimaros wrote:
| "SERA-32B is the first model in Ai2's Open Coding Agents
| series. It is a state-of-the-art open-source coding agent
| that achieves 49.5% on SWE-bench Verified, matching the
| performance of much larger models like Devstral-Small-2
| (24B)" from https://huggingface.co/allenai/SERA-32B
| ethan_l_shen wrote:
| Ah great catch I don't know how we missed that. Thanks!
| Will fix.
| augusteo wrote:
| The ahmadyan comparison is fair. Meta's CWM models hitting 65% vs
| SERA's 54% is a meaningful gap.
|
| But the interesting number here isn't accuracy. It's the $400 to
| reproduce top open-source performance. That's the part that
| matters for teams building internal tooling.
|
| We've been running agents on proprietary codebases at work. The
| pain isn't model quality. It's customization. Most off-the-shelf
| agents don't understand your repo structure, your conventions,
| your test patterns. If you can fine-tune a 32B model on your own
| codebase for a few hundred dollars, that changes the economics
| completely.
|
| But codebases changes everyday, so finetuning will have to be
| continuously done!
|
| Probably not worth it versus something like Claude Code.
|
| Curious whether anyone's tried this on non-Python codebases. Most
| SWE-Bench stuff is Python-heavy.
| storystarling wrote:
| The fine-tuning overhead is definitely a factor, but for
| smaller shops the hard constraint is usually inference VRAM.
| Running a 32B model locally or on a rented GPU is surprisingly
| expensive if you aren't saturating it. Even at 4-bit
| quantization you are looking at dual 3090s or an A6000 to get
| decent tokens per second. The $400 training cost is impressive
| but the hosting bill is what actually kills the margin compared
| to per-token APIs.
| nickandbro wrote:
| Great work! Really respect AI2. they open source everything. The
| model, the weights, the training pipeline, inference stack, and
| corpus
| Imustaskforhelp wrote:
| Hey this looks great? Is it available on Openrouter.
|
| I wish if AI2 could release a more denser model on Openrouter for
| free than the 8B model as I was using Devstral model for agentic
| purposes.
|
| If we can get an agentic good 32B like model on openrouter for
| ~free, then I feel like it will be very interesting to see how
| things would go imo.
|
| Good luck with AI2! The premise of truly open source models is
| really interesting and I feel like it could help bring more
| innovation in the space imo!
| ripped_britches wrote:
| One claim in article is definitely very wrong or at least needs
| to be narrowed. Claude is the only closed agent harness and there
| are about two dozen open ones. Many models may be closed, but
| when people say agent they are generally referring to the
| harness, not the underlying model.
| janmue wrote:
| "Strong closed-weight coding agents like Devstral Small 2 are an
| important point of comparison."
|
| Devstral Small 2 is an open-weights model:
| https://huggingface.co/mistralai/Devstral-Small-2-24B-Instru...
___________________________________________________________________
(page generated 2026-01-27 23:00 UTC)