[HN Gopher] Claude Code daily benchmarks for degradation tracking
___________________________________________________________________
Claude Code daily benchmarks for degradation tracking
Author : qwesr123
Score : 475 points
Date : 2026-01-29 13:59 UTC (9 hours ago)
(HTM) web link (marginlab.ai)
(TXT) w3m dump (marginlab.ai)
| qwesr123 wrote:
| FYI the MarginLab Claude Code degradation tracker is showing a
| statistically significant ~4% drop in SWE-Bench-Pro accuracy over
| the past month
| beardsciences wrote:
| Very interesting. I would be curious to understand how granular
| these updates are being applied to CC + what might be causing
| things like this. I feel like I can notice a very small
| degradation but have compensated with more detailed prompts
| (which I think, perhaps naively, is offsetting this issue).
| chrisjj wrote:
| > more detailed prompts (which I think, perhaps naively, is
| offsetting this issue).
|
| Is exacerbating this issue ... if the load theory is correct.
| goldenarm wrote:
| I really like the idea, but a "+-14.0% significance threshold" is
| meaningless here.
|
| The larger monthly scale should be the default, or you should get
| more samples.
| zacmps wrote:
| Could you elaborate what you think the problems are? I guess
| they should be using some form of multiple comparison
| correction?
| goldenarm wrote:
| The daily scale is not statistically significant and is
| meaningless. You should lower the confidence interval by
| either increasing the scale or the evaluations.
| turnsout wrote:
| This is probably entirely down to subtle changes to CC
| prompts/tools.
|
| I've been using CC more or less 8 hrs/day for the past 2 weeks,
| and if anything it feels like CC is getting better and better at
| actual tasks.
|
| _Edit: Before you downvote, can you explain how the model could
| degrade WITHOUT changes to the prompts? Is your hypothesis that
| Opus 4.5, a huge static model, is somehow changing? Master system
| prompt changing? Safety filters changing?_
| fragebogen wrote:
| I was going to ask, are all other variables accounted for? Are
| we really comparing apples to apples here? Still worth doing
| obviously, as it serves a good e2e evaluations, just for
| curiosity's sake.
| FfejL wrote:
| Honest, good-faith question.
|
| Is CC getting better, or are you getting better at using it?
| And how do you know the difference?
|
| I'm an occasional user, and I can definitely see improvements
| in my prompts over the past couple of months.
| turnsout wrote:
| Good-faith answer: I can't be certain. But I've been using CC
| since its release, and Cursor before that (and actually going
| all the way back to GPT3 to do codegen in the Playground).
| After getting used to the CC workflow, the way that I use it
| has been pretty consistent. To be specific, I use basically
| the same AGENTS.md with small modifications for each project,
| and I live almost exclusively in Plan mode and the best model
| (currently Opus 4.5).
|
| My initial prompting is boilerplate at this point, and looks
| like this:
|
| (Explain overall objective / problem without jumping to a
| solution)
|
| (Provide all the detail / file references / past work I can
| think of)
|
| (Ask it "what questions do you have for me before we build a
| plan?")
|
| And then go back and forth until we have a plan.
|
| Compared to my work with CC six months ago, it's just much
| more capable, able to solve more nuanced bugs, and less
| likely to generate spaghetti code.
| rob wrote:
| I agree with you, it's personally hard to tell.
|
| For me I've noticed it getting nothing but better over the
| past couple months, but I've been working on my workflows and
| tooling.
|
| For example, I used to use plan mode and would put everything
| in a single file and then ask it to implement it in a new
| session.
|
| Switching to the 'superpowers' plugin with its own skills to
| brainstorm and write plans and execute plans with batches and
| tasks seems to have made a big improvement and help catch
| things I wouldn't have before. There's a "get shit done"
| plugin that's similar that I want to explore as well.
|
| The code output always looks good to me for the most part
| though and I've never thought that it's getting dumber
| anything, so I feel like a lot of the improvements I see are
| because of a skill issue on my part trying to use everything.
| Obviously it doesn't help there's a new way to do things
| every two weeks though.
| billylo wrote:
| That's why benchmarks are useful. We all suffer from the
| shortcomings of human perception.
| gpm wrote:
| Benchmarks shortcomings are no worse... they inevitably
| measure something that is only close to the thing you
| actually care about, not the thing you actually care about.
| It's entirely plausible that this decreased benchmark score
| is because Anthropic's initial prompting of the model was
| overtuned to the benchmark and as they're gaining more
| experience with real world use they are changing the prompt
| to do better at that and consequentially worse at the
| benchmark.
| billylo wrote:
| I wonder how best we can measure the usefulness of models
| going forward.
|
| Thumbs up or down? (could be useful for trends) Usage
| growth from the same user over time? (as an approximation)
| Tone of user responses? (Don't do this... this is the wrong
| path... etc.)
| turnsout wrote:
| Benchmarks measure what they measure. But your subjective
| experience also matters.
| arcanemachiner wrote:
| The easiest way would be to quantize the model, and serve
| different quants based on the current demand. Higher volumes ==
| worse quant == more customers served per GPU
| gpm wrote:
| I upvoted, but
|
| > Edit: Before you downvote, can you explain how the model
| could degrade WITHOUT changes to the prompts?
|
| The article actually links to this fine postmortem by anthropic
| that demonstrates one way this is possible - software bugs
| affecting inference:
| https://www.anthropic.com/engineering/a-postmortem-of-three-...
|
| Another way this is possible is the model reacting to
| "stimuli", e.g. the hypothesis at the end of 2023 that the
| (then current) ChatGPT was getting lazy because it was finding
| out the date was in december and it associated winter with
| shorter lazier responses.
|
| A third way this is possible is the actual conspiracy version -
| Anthropic might make changes to make inference cheaper at the
| expense of the quality of the responses. E.g. quantizing
| weights further or certain changes to the sampling procedure.
| fragebogen wrote:
| Would love to see this idea expanded to ever alleged SoTA model
| currently in production. Any speculation as to why this
| degradation occurs?
| embedding-shape wrote:
| Anecdote, I don't have any proof and it's just a feeling. But
| around afternoon in GMT+1 compared to the morning/midday, there
| seems to be a change in the quality of responses, which seems
| to line up with when the US wakes up. I consistently get (what
| feels like) worse responses in both Codex and Claude Code in
| the afternoon/night compared to morning/midday, so much that I
| usually give up then try the same prompt next morning and get
| better results. But I guess that might as well be about me
| being more tired in the night than morning too, as I said,
| haven't measured this.
| jzig wrote:
| It's the afternoon slump. The AI needs a cup of coffee and to
| doomscroll for half an hour!
| embedding-shape wrote:
| Or a load balancing technique :) Either way, it kicks me
| off to do other things so maybe it isn't so bad after all.
| sciencejerk wrote:
| Why is this happening?
| giwook wrote:
| https://www.anthropic.com/engineering/a-postmortem-of-three-...
| observationist wrote:
| >>> We never reduce model quality due to demand, time of day,
| or server load. The problems our users reported were due to
| infrastructure bugs alone.
|
| Just ignore the continual degradation of service day over
| day, long after the "infrastructure bugs" have reportedly
| been solved.
|
| Oh, and I've got a bridge in Brooklyn to sell ya, it's a
| _great_ deal!
| alias_neo wrote:
| > We never reduce model quality due to demand, time of day,
| or server load
|
| Forgive me, but as a native English speaker, this sentence
| says exactly one thing to me; We _do_ reduce model quality,
| just not for these listed reasons.
|
| If they don't do it, they could put a full stop after the
| fifth word and save some ~~tokens~~ time.
| chrisjj wrote:
| Moreover the assurance re _model_ quality is not re
| _results_ quality.
| observationist wrote:
| Yes, Dario is responsible for some of the weaseliest of
| corporate weasel wording I've ever seen, and he's got
| some incredible competition in that arena. Those things
| aren't the reason, they're just strongly coincidental
| with the actual reason, which is to slow the burn rate
| and extend the runway.
| Trufa wrote:
| I have absolutely no insight knowledge, but I think it's not a
| bad assumption to have that, it's costly to run the models,
| when they release a new model they assume that cost and give
| per user more raw power, when they've captured the new users
| and wow factor, they start reducing costs by reducing the
| capacity they provide to users. Rinse and repeat.
| bn-l wrote:
| That is absolutely scummy.
| Uehreka wrote:
| There are frequently claims that Anthropic is somehow diluting
| or dumbing down models in some subtle way. Unfortunately it's
| tough to validate these claims without a body of regularly
| checked evals. This test set should hopefully help settle
| whether Anthropic is actually making changes under the hood or
| whether the changes are all in people's heads.
| observationist wrote:
| They're "optimizing" costs wherever possible - reducing compute
| allocations, quantizing models, doing whatever they can to
| reduce the cost per token, but vehemently insisting that no
| such things are occurring, that it's all in the users' heads,
| and using the weaseliest of corporate weasel speak to explain
| what's happening. They insist it's not happening, then they say
| something like "oh, it happened but it was an accident", then
| they say "yes, it's happening, but it's actually good!" and "we
| serve the same model day by day, and we've always been at war
| with Eastasia."
|
| They should be transparent and tell customers that they're
| trying to not lose money, but that'd entail telling people why
| they're paying for service they're not getting. I suspect it's
| probably not legal to do a bait and switch like that, but this
| is pretty novel legal territory.
| emp17344 wrote:
| It's entirely possible it's not happening, and this phenomenon
| of "model degradation" is just user hype meeting reality.
| Dowwie wrote:
| Simply search user prompts for curse words and then measure
| hostility sentiment. User hostility rises as agents fail to meet
| expectations.
| Trufa wrote:
| I'm glad I'm not the only one.
| sejje wrote:
| One time I cussed Claude out so hard that it actually quit
| his doom-loop and fixed the thing.
|
| It's the only time cussing worked, though.
| bn-l wrote:
| I don't know. My gut feeling is it seems to help.
| mrbananagrabber wrote:
| I uh might be skewing that as I generally just use a lot of
| curse words with Claude by default
| ctxc wrote:
| I feel bad about it but sometimes it's so daft, I can't even xD
|
| It's not my fault, they set high standards!
| smotched wrote:
| there are many times where I just do it myself and it thinks it
| did well.
| preuceian wrote:
| Maybe im overlooking something obvious but how do you 'simply'
| scan the content of Claude users their prompts?
| gordonhart wrote:
| GP was making a joke, but Anthropic could implement this if
| they wanted to. Not a bad metric actually if you can measure
| it cheaply enough.
| mhl47 wrote:
| Or there are global events that stress people out .. or their
| expectations change over time. Not that simple ;)
| nateberkopec wrote:
| Good thing expectations are perfectly constant!
| mbm wrote:
| This might be strangely effective.
| silverlight wrote:
| There was a moment about a week ago where Claude went down for
| about an hour. And right after it came back up it was clear a lot
| of people had given up and were not using it.
|
| It was probably 3x faster than usual. I got more done in the next
| hour with it than I do in half a day usually. It was definitely a
| bit of a glimpse into a potential future of "what if these things
| weren't resource constrained and could just fly".
| yoavsha1 wrote:
| I had that exact same feeling during the US holidays where I
| got to enjoy 2x usage limits and everything just seemed to work
| well
| cmrdporcupine wrote:
| I had terrible results during the holidays -- it wasn't slow
| but it was clear they were dealing with the load by
| quantizing in spots because there were entire chunks of days
| when the results from it were so terrible I gave up and
| switched to using Gemini or Codex via opencode.
| svdr wrote:
| I would also regret it if they become that fast; right now I
| can really take a moment to enjoy the hard work the model is
| doing for me.
| nlh wrote:
| Noticed the exact same thing a few days ago. So much so that I
| went on twitter and HN to search for "claude speed boost" to
| see if there was a known new release. Felt like the time I
| upgraded from a 2400 baud modem to a 14.4 as a kid - everything
| was just lightning fast (for a brief shining moment).
| dajonker wrote:
| Wouldn't be surprised if they slowly start quantizing their
| models over time. Makes it easier to scale and reduce operational
| cost. Also makes a new release have more impact as it will be
| more notably "better" than what you've been using the past couple
| of days/weeks.
| YetAnotherNick wrote:
| Benchmarks like ARG AGI are super price correlated and cheap to
| run. I think it's very easy to prove that the models are
| degrading.
| rustyhancock wrote:
| Oooff yes I think that is exactly the kind of shenanigans they
| might pull.
|
| Ultimately I can understand if a new model is coming in without
| as much optimization then it'll add pressure to the older
| models achieving the same result.
|
| Nice plausible deniability for a convenient double effect.
| kilroy123 wrote:
| It sure feels like they do this. They claim they don't, but
| using it every day for 5-10 hours a day. You notice when
| something changes.
|
| This last week it seems way dumber than before.
| eli wrote:
| I would be surprised tbh.
|
| Anthropic does not exactly act like they're constrained by
| infra costs in other areas, and noticeably degrading a product
| when you're in tight competition with 1 or 2 other players with
| similar products seems like a bad place to start.
|
| I think people just notice the flaws in these models more the
| longer they use them. Aka the "honeymoon-hangover effect," a
| real pattern that has been shown in a variety of real world
| situations.
| Roark66 wrote:
| I haven't noticed much difference in Claude, but I swear gemini
| 3 pro preview was better in the first week or two and later
| started feeling like they quantized it down to hell.
| 9cb14c1ec0 wrote:
| I don't think so. There are other knobs they can tweak to
| reduce load that affect quality less than quantizing. Like
| trimming the conversation length without telling you, reducing
| reasoning effort, etc.
| mgraczyk wrote:
| We never do anything that reduce model intelligence like that
| kristianp wrote:
| Open weights models such as GPT-OSS, Kimi K2.x are trained with
| 4 bit layers. So it wouldn't come as a surprise if the closed
| models do similar things. If I compare Kimi K2.5 and Opus 4.5
| on openrouter, output tokens are about 8x more expensive for
| Opus, which might indicate Opus is much larger and doesn't
| quantize, but the claude subscription plans muddy the waters on
| price comparison a lot.
| ofirpress wrote:
| [SWE-bench co-author here] It seems like they run this test on a
| subset of 50 tasks, and that they only run the test once per day.
| So a lot of the movement in accuracy could be attributed to that.
| I would run on 300 tasks and I'd run the test suite 5 or 10 times
| per day and average that score. Lots of variance in the score can
| come from random stuff like even Anthropic's servers being
| overloaded.
| mohsen1 wrote:
| Hope you don't mind the unrelated question:
|
| How do you pay for those SWE-bench runs?
|
| I am trying to run a benchmark but it is too expensive to run
| enough runs to get a fair comparison.
|
| https://mafia-arena.com
| ofirpress wrote:
| Benchmarks can get costly to run- you can reach out to
| frontier model creators to try and get them to give you free
| credits, but usually they'll only agree to that once your
| benchmark is pretty popular.
| mohsen1 wrote:
| yes I reached out to them but as you say it's a chicken-
| and-egg problem.
|
| Thanks!
| Dolores12 wrote:
| so basically they know requests using your API key should
| be treated with care?
| swyx wrote:
| they could but you can also have some trust in anthropic
| to have some integrity there, these are earnest people.
|
| "trust but verify" ofc .
| https://latent.space/p/artificialanalysis do api keys but
| also mystery shopper checks
| mrandish wrote:
| > these are earnest people.
|
| I agree.
|
| I'll also add that when my startup got acquired into a
| very large, well-known valley giant with a sterling rep
| for integrity and I ended up as a senior executive - over
| time I got a first-hand education on the myriad ways
| genuinely well-intentioned people can still end up being
| the responsible party(s) presiding over a system doing
| net-wrong things. All with no individual ever meaning to
| or even consciously knowing.
|
| It's hard to explain and I probably wouldn't have
| believed myself before I saw and experienced it. Standing
| against an overwhelming organizational tide is stressful
| and never leads to popularity or promotion. I _think_ I
| probably managed to move on before directly compromising
| myself but preventing that required constant vigilance
| and led to some inter-personal and 'official' friction.
| And, frankly, I'm not really sure. It's entirely possible
| I bear direct moral responsibility for a few things I
| believe no good person would do as an exec in a good
| company.
|
| That's the key take-away which took me a while to process
| and internalize. In a genuinely good organization with
| genuinely good people, it's not "good people get
| pressured by constraints and tempted by extreme
| incentives, then eventually slip". I still talk with
| friends who are senior execs there and sometimes they
| want to talk about whether something is net good or bad.
| I kind of dread the conversation going there because it's
| inevitably incredibly complex and confusing.
| Philosopher's trolley car ethics puzzles pale next to
| these multi-layered, messy conundrums. But who else are
| they going to vent to who might understand? To be clear,
| I still believe that company and its leadership to be one
| of the most moral, ethical and well-intentioned in the
| valley. I was fortunate to experience the best case
| scenario.
|
| Bottom line: if you believe earnest, good people being in
| charge is a reliable defense against the organization
| doing systemically net-wrong things - you don't
| comprehend the totality of the threat environment. And
| that's okay. Honestly, you're lucky. Because the reality
| is infinitely more ambiguously amoral than white hats vs
| black hats - at the end of the day the best the 'very
| good people' can manage is some shade of middle gray. The
| saddest part is that good people still care, so they
| _want_ to check the shade of their hat but no one can see
| if it 's light enough to at least tell yourself "I did
| good today."
| pluralmonad wrote:
| Someone posted this here the other day and it uses
| _Demons_ to discuss exactly your point.
|
| https://possessedmachines.com/
| mrandish wrote:
| Wow. Only one page in and already bookmarked to absorb
| later. Thanks for the link.
| epolanski wrote:
| The last thing a proper benchmark should do is reveal it's
| own API key.
| sejje wrote:
| That's a good thought I hadn't had, actually.
| plagiarist wrote:
| IMO it should need a third party running the LLM anyway.
| Otherwise the evaluated company could notice they're
| receiving the same requests daily and discover
| benchmarking that way.
| jabedude wrote:
| But that's removing a component that's critical for the
| test. We as users/benchmark consumers care that the
| service as provided by Anthropic/OpenAI/Google is
| consistent over time given the same model/prompt/context
| plagiarist wrote:
| Might as well have the free tokens, then, especially if
| it is an open benchmark they are already aware of. If
| they want to game it they cannot be stopped from doing so
| when it's on their infra.
| mrandish wrote:
| With the insane valuations and actual revenue at stake,
| benchmarkers should assume they're assessing in an
| adversarial environment. Whether from intentional gaming,
| training to the test, or simply from prioritizing things
| likely to make results look better, targeting benchmarks
| will almost certainly happen.
|
| We already know large graphics card manufacturers tuned
| their drivers to recognize specific gaming benchmarks.
| Then when that was busted, they implemented detecting
| benchmarking-like behavior. And the money at stake in
| consumer gaming was comparatively tiny compared to
| current AI valuations. The cat-and-mouse cycle of measure
| vs counter-measure won't stop and should be a standard
| part of developing and administering benchmark services.
| cedws wrote:
| Agreed, this benchmark would be much more useful ran multiple
| times a day. That could reveal degredation in line with load
| patterns.
| bredren wrote:
| For CC, I suspect it also need to be testing and labeling
| separate runs against subscription, public API and Bedrock-
| served models?
|
| It's a terrific idea to provide this. ~Isitdownorisitjustme
| for LLMs would be the parakeet in the coalmine that could at
| least inform the multitude of discussion threads about
| suspected dips in performance (beyond HN).
|
| What we could also use is similar stuff for Codex, and
| eventually Gemini.
|
| Really, the providers themselves should be running these
| tests and publishing the data.
|
| The availability status information is no longer sufficient
| to gauge the service delivery because it is by nature non-
| deterministic.
| swyx wrote:
| i recall another project here on HN maybe 4-6 months ago that
| would run tests 4x a day or something. not sure how to find
| them again
| Davidzheng wrote:
| but degradation from servers being overloaded would be the type
| of degradation this SHOULD measure no? Unless it's only
| intended for measuring their quietly distilling models (which
| they claim not to do? idk for certain)
| cmrdporcupine wrote:
| I've personally witnessed large variability in behaviour even
| within a given session -- which makes sense as there's
| nothing stopping Anthropic from shuttling your
| context/session around load balanced through many different
| servers, some of which might be quantized heavily to manage
| load and others not at all.
|
| I don't know if they do this or not, but the nature of the
| API is such you could absolutely load balance this way. The
| context sent at each point is not I believe "sticky" to any
| server.
|
| TLDR you could get a "stupid" response and then a "smart"
| response _within_ a single session because of heterogeneous
| quantization / model behaviour in the cluster.
| epolanski wrote:
| I've defended opus in the last weeks but the degradation is
| tangible. It feels like it degraded by a generation tbh.
| cmrdporcupine wrote:
| it's just extremely variable
| megabless123 wrote:
| noob question: why would increased demand result in decreased
| intelligence?
| vidarh wrote:
| It would happen if they quietly decide to serve up more
| aggressively distilled / quantised / smaller models when
| under load.
| chrisjj wrote:
| They advertise the Opus 4.5 model. Secretly substituting
| a cheaper one to save costs would be fraud.
| kingstnap wrote:
| Old school Gemini used to do this. It was super obvious
| because mid day the model would go from stupid to
| completely brain dead. I have a screenshot of Google's
| FAQ on my PC from 2024-09-13 that says this (I took it to
| post to discord):
|
| > How do I know which model Gemini is using in its
| responses?
|
| > We believe in using the right model for the right task.
| We use various models at hand for specific tasks based on
| what we think will provide the best experience.
| chrisjj wrote:
| > We use various models at hand for specific tasks based
| on what we think will provide the best experience
|
| ... for Google :)
| vidarh wrote:
| If you use the API, you pay for a specific model, yes,
| but even then there are "workarounds" for them, such as
| someone else pointed out by reducing the amount of time
| they let it "think".
|
| If you use the subscriptions, the terms specifically says
| that beyond the caps they can limit your "model and
| feature usage, at our discretion".
| chrisjj wrote:
| Sure. I was separating the model - which Anthropic
| promises not to downgrade - and the "thinking time" -
| which Anthropic _doesn 't_ promise not to downgrade. It
| seems the latter is very likely the culprit in this case.
| seunosewa wrote:
| Or just reducing the reasoning tokens.
| Wheaties466 wrote:
| from what I understand this can come from the batching of
| requests.
| chrisjj wrote:
| So, a known bug?
| embedding-shape wrote:
| No, basically, the requests are processed in batches,
| together, and the order they're listed in matters for the
| results, as the grid (tiles) that the GPU is ultimately
| processing, are different depending on what order they
| entered at.
|
| So if you want batching + determinism, you need the same
| batch with the same order which obviously don't work when
| there are N+1 clients instead of just one.
| chrisjj wrote:
| Sure, but how can that lead to increased demand resulting
| in decreased intelligence? That is the effect we are
| discussing.
| embedding-shape wrote:
| Small subtle errors that are only exposed at certain
| execution parts could be one. You might place things
| differently onto the GPU depending on how large the batch
| is, if you've found one way to be faster batch_size<1024,
| but another when batch_size>1024. As number of concurrent
| incoming requests goes up, you increase batch_size. Just
| one possibility, guess there could be a multitude of
| reasons, as it's really hard to reason about until you
| sit with the data in front of you. vLLM has had bugs with
| these sort of thing too, so wouldn't surprise me.
| chrisjj wrote:
| Wouldn't you think that was as likely to increase as
| decrease intelligence, so average to nil in the
| benchmarks?
| embedding-shape wrote:
| No, I'm not sure how that'd make sense. Either you're
| making the correct (expected) calculations, or you're
| getting it wrong. Depending the type of wrong or how
| wrong, could go from "used #2 in attention instead of #1"
| so "blue" instead of "Blue" or whatever, to completely
| incoherent text and garbled output.
| chrisjj wrote:
| I accept errors are more likely to decrease
| "intelligence". But I don't see how increased load,
| through batching, is any more likely to increase than
| decrease errors.
| awestroke wrote:
| I've seen some issues with garbage tokens (seemed to come
| from a completely different session, mentioned code I've
| never seen before, repeated lines over and over) during
| high load, suspect anthropic have some threading bugs or
| race conditions in their caching/inference code that only
| happen during very high load
| exitb wrote:
| An operator at load capacity can either refuse requests, or
| move the knobs (quantization, thinking time) so requests
| process faster. Both of those things make customers
| unhappy, but only one is obvious.
| codeflo wrote:
| This is intentional? I think delivering lower quality
| than what was advertised and benchmarked is borderline
| fraud, but YMMV.
| chrisjj wrote:
| There is no level of quality advertised, as far as I can
| see.
| pseidemann wrote:
| What is "level of quality"? Doesn't this apply to any
| product?
| chrisjj wrote:
| In this case, it is benchmark performance. See the root
| post.
| denysvitali wrote:
| If there's no way to check, then how can you claim it's
| fraud? :)
| mcny wrote:
| Personally, I'd rather get queued up on a long wait time
| I mean not ridiculously long but I am ok waiting five
| minutes to get correct it at least more correct
| responses.
|
| Sure, I'll take a cup of coffee while I wait (:
| lurking_swe wrote:
| i'd wait any amount of time lol.
|
| at least i would KNOW it's overloaded and i should use a
| different model, try again later, or just skip AI
| assistance for the task altogether.
| bpavuk wrote:
| > I think delivering lower quality than what was
| advertised and benchmarked is borderline fraud
|
| welcome to the Silicon Valley, I guess. everything from
| Google Search to Uber is fraud. Uber is a classic example
| of this playbook, even.
| copilot_king wrote:
| If you aren't defrauding your customers you will be left
| behind in 2026
| rootnod3 wrote:
| That number is a sliding window, isn't it?
| TedDallas wrote:
| Per Anthropic's RCA linked in Ops post for September 2025
| issues:
|
| "... To state it plainly: We never reduce model quality
| due to demand, time of day, or server load. ..."
|
| So according to Anthropic they are not tweaking quality
| setting due to demand.
| rootnod3 wrote:
| And according to Google, they always delete data if
| requested.
|
| And according to Meta, they always give you ALL the data
| they have on you when requested.
| entropicdrifter wrote:
| >And according to Google, they always delete data if
| requested.
|
| However, the request form is on display in the bottom of
| a locked filing cabinet stuck in a disused lavatory with
| a sign on the door saying 'Beware of the Leopard'.
| groundzeros2015 wrote:
| What would you like?
| AlexandrB wrote:
| An SLA-style contractually binding agreement.
| edmundsauto wrote:
| I bet this is available in large enterprise agreements.
| How much are you willing to pay for it?
| Onavo wrote:
| Priced in.
| cmrdporcupine wrote:
| I guess I just don't know how to square that with my
| actual experiences then.
|
| I've seen sporadic drops in reasoning skills that made me
| feel like it was January 2025, not 2026 ... inconsistent.
| root_axis wrote:
| I wouldn't doubt that these companies would deliberately
| degrade performance to manage load, but it's also true
| that humans are notoriously terrible at identifying
| random distributions, even with something as simple as a
| coin flip. It's very possible that what you view as
| degradation is just "bad RNG".
| cmrdporcupine wrote:
| yep stochastic fantastic
|
| these things are by definition hard to reason about
| quadrature wrote:
| LLMs sample the next token from a conditional probability
| distribution, the hope is that dumb sequences are less
| probable but they will just happen naturally.
| tempaccount420 wrote:
| It's more like the choice between "the" and "a" than
| "yes" and "no".
| mattmanser wrote:
| Funny how those probabilities consistently at 2pm UK time
| when all the Americans come online...
| chrisjj wrote:
| That's about model quality. Nothing about output quality.
| stefan_ wrote:
| Thats what is called an "overly specific denial". It
| sounds more palatable if you say "we deployed a newly
| quantized model of Opus and here are cherry picked
| benchmarks to show its the same", and even that they
| don't announce publicly.
| direwolf20 wrote:
| They don't advertise a certain quality. You take what
| they have or leave it.
| sh3rl0ck wrote:
| I'd wager that lower tok/s vs lower quality of output
| would be two very different knobs to turn.
| botacode wrote:
| Load just makes LLMs behave less deterministically and likely
| degrade. See: https://thinkingmachines.ai/blog/defeating-
| nondeterminism-in...
|
| They don't have to be malicious operators in this case. It
| just happens.
| bgirard wrote:
| > malicious
|
| It doesn't have to be malicious. If my workflow is to send
| a prompt once and hopefully accept the result, then
| degradation matters a lot. If degradation is causing me to
| silently get worse code output on some of my commits it
| matters to me.
|
| I care about -expected- performance when picking which
| model to use, not optimal benchmark performance.
| Aurornis wrote:
| Non-determinism isn't the same as degradation.
|
| The non-determinism means that even with a temperature of
| 0.0, you can't expect the outputs to be the same across
| API calls.
|
| In practice people tend to index to the best results
| they've experienced and view anything else as
| degradation. In practice it may just be randomness in
| either direction from the prompts. When you're getting
| good results you assume it's normal. When things feel off
| you think something abnormal is happening. Rerun the
| exact same prompts and context with temperature 0 and you
| might get a different result.
| bonoboTP wrote:
| This has nothing to do with overloading. The suspicion is
| that when there is too much demand (or they just want to
| save costs), Anthropic sometimes uses a less capable
| (quantized, distilled, etc) version of the model. People
| want to measure this so there is concrete evidence
| instead of hunches and feelings.
|
| To say that this measurement is bad because the server
| might just be overloaded completely misses the point. The
| point is to see if the model sometimes _silently_
| performs worse. If I get a response from "Opus", I want
| a response from Opus. Or at least want to be told that
| I'm getting slightly-dumber-Opus this hour because the
| server load is too much.
| F7F7F7 wrote:
| "Just drink the water, it's all water."
| novaleaf wrote:
| this is about variance of daily statistics, so I think
| the suggestions are entirely appropriate in this context.
| altcognito wrote:
| Explain this though. The code is deterministic, even if it
| relies on pseudo random number generation. It doesn't just
| happen, someone has to make a conscious decision to force a
| different code path (or model) if the system is loaded.
| FL33TW00D wrote:
| It takes a different code path for efficiency.
|
| e.g
|
| if (batch_size > 1024): kernel_x else: kernel_y
| chrisjj wrote:
| Not deterministic.
| https://thinkingmachines.ai/blog/defeating-
| nondeterminism-in...
| pertymcpert wrote:
| Floating point math isn't associative for operations that
| are associative in normal math.
| measurablefunc wrote:
| That would just add up to statistical noise instead of
| 10% degradation over a week.
| kevin_thibedeau wrote:
| Catastrophic error accumulation can produce more profound
| effects than noise.
| measurablefunc wrote:
| Just to make sure I got this right. They serve millions
| of requests a day & somehow catastrophic error
| accumulation is what is causing the 10% degradation & no
| one at Anthropic is noticing it. Is that the theory?
| minimaltom wrote:
| Its not deterministic. Any individual floating point
| mul/add is deterministic, but in a GPU these are all
| happening in parallel and the accumulation is in the
| order they happen to complete.
|
| When you add A then B then C, you get a different answer
| than C then A then B, because floating point,
| approximation error, subnormals etc.
| bonoboTP wrote:
| It can be made deterministic. It's not trivial and can
| slow it down a bit (not much) but there are environment
| variables you can set to make your GPU computations
| bitwise reproducible. I have done this in training models
| with Pytorch.
| jmalicki wrote:
| For all practical purposes any code reliant on the output
| of a PRNG is non-deterministic in all but the most
| pedantic senses... And if the LLM temperature isn't set
| to 0 LLMs are sampling from a distribution.
|
| If you're going to call a PRNG deterministic then the
| outcome of a complicated concurrent system with no
| guaranteed ordering is going to be deterministic too!
| bonoboTP wrote:
| How is this related to overloading? The nondeterminism
| should not be a function of overloading. It should just
| time out or reply slower. It will only be dumber if it
| gets rerouted to a dumber, faster model eg quantized.
| gmueckl wrote:
| No, this isn't right. There are totally legitimate use
| cases for PRNGs as sources of random number sequences
| following a certain probability distribution where
| freezing the seed and getting reproducibility is actually
| required.
| jmalicki wrote:
| And for a complicated concurrent system you can also
| replay the exact timings and orderings as well!
| make3 wrote:
| There's a million algorithms to make LLM inference more
| efficient as a tradeoff for performance, like using a
| smaller model, using quantized models, using speculative
| decoding with a more permissive rejection threshold, etc
| etc
| strongpigeon wrote:
| The question I have now after reading this paper (which was
| really insightful) is do the models really get _worse_
| under load, or do they just have a higher variance? It
| seems like the latter is what we should expect, not it
| getting worse, but absent load data we can 't really know.
| stefan_ wrote:
| The primary (non malicious, non stupid) explanation given
| here is batching. But I think you would find looking at
| large-scale inference the batch sizes being ran on any
| given rig are fairly static - there is a sweet spot for any
| given model part ran individually between memory
| consumption and GPU utilization, and generally GPUs do
| badly at job parallelism.
|
| I think the more likely explanation is again with the
| extremely heterogeneous compute platforms they run on.
| hatmanstack wrote:
| That's why I'd love to get stats on
| load/hardware/location of where my inference is running.
| Looking at you Trainiuim.
| make3 wrote:
| It's very clearly a cost tradeoff that they control and
| that should be measured.
| epolanski wrote:
| Stilll relevant over time.
| dana321 wrote:
| "Lots of variance in the score can come from random stuff like
| even Anthropic's servers being overloaded"
|
| Aha, so the models do degrade under load.
| seunosewa wrote:
| The degradation may be more significant within the day than at
| the same time every day.
| GoatInGrey wrote:
| Sure, but it's still useful insight to see how it performs
| over time. Of course, cynically, Anthropic could game the
| benchmark by routing this benchmark's specific prompts to an
| unadulterated instance of the model.
| chrisjj wrote:
| > Lots of variance in the score can come from random stuff like
| even Anthropic's servers being overloaded.
|
| Are you suggesting result accuracy varies with server load?
| rootnod3 wrote:
| Sorry what?
|
| "You can't measure my Cloud Service's performance correctly if
| my servers are overloaded"?
|
| "Oh, you just measured me at bad times each day. On only 50
| different queries."
|
| So, what does that mean? I have to pick specific times during
| the day for Claude to code better?
|
| Does Claude Code have office hours basically?
| copilot_king wrote:
| > Does Claude Code have office hours basically?
|
| Yes. Now pay up or you will be replaced.
| rootnod3 wrote:
| Verily, my vichyssoise of verbiage veers most verbose, so
| let me run that thing out of tokens fast.
| johnsmith1840 wrote:
| This has been happening for years. Tgere's a great paper from
| microsoft on Deepspeed AI inference.
|
| Basically the paper showed methods for how to handle heavy
| traffic load by changing model requirements or routing to
| different ones. This was awhile ago and I'm sure it's
| massively more advanced now.
|
| Also why some of AI's best work for me is early morning and
| weekends! So yes, the best time to code with modern LLM
| stacks is when nobody else is. It's also possibly why we go
| through phases of "they neutered the model" some time after a
| new release.
| swyx wrote:
| chill out, ofir does not work for anthropic. he's just saying
| there's inherent variability in LLMs and you need to at least
| 30x the samples that OP is doing in order to make any form of
| statistically significant conclusions.
| kuboble wrote:
| I wonder if my great experience with claude are partly due to
| the fact that my working hours don't overlap with the US west
| coast
| bhk wrote:
| According to Anthropic: "We never reduce model quality due to
| demand, time of day, or server load."
|
| https://www.anthropic.com/engineering/a-postmortem-of-three-...
| embedding-shape wrote:
| They've had issues before with things like "TPU top-k error -
| Claude sometimes dropped the best next token"
| (https://www.anthropic.com/engineering/a-postmortem-of-
| three-...) so what's going on might not be intentional even.
| mgraczyk wrote:
| That issue did not have any time of day dependence
| nikcub wrote:
| > I would run on 300 tasks and I'd run the test suite 5 or 10
| times per day and average that score.
|
| assume this is because of model costs. anthropic could either
| throw some credits their way (would be worthwhile to dispel the
| 80 reddit posts a day about degrading models and quantization)
| or OP could throw up a donation / tip link
| phist_mcgee wrote:
| Then you'd get people claiming that the benchmarks were 'paid
| for' by anthropic
| ghm2199 wrote:
| In medicine there is a concept of reporting adverse effects of
| medication or interventions which are then collectively studied
| for Public Health [MedWatch][VAERS][EudraVigilance] and in
| academia. We should have something like that for all coding
| agents(and agents in other fields too), given how widely its
| deployed and affect on "health" in general(not only human). Call
| it the AI "health" of things benchmark.
|
| I would imagine a sort of hybrid qualities of volunteer efforts
| like wikipedia, new problems like advent of code and benchmarks
| like this. The goal? It would be to study the collective effort
| on the affects of usage to so many areas where AI is used.
|
| [MedWatch](https://www.fda.gov/safety/medwatch-fda-safety-
| information-a...)
|
| [VAERS](https://www.cdc.gov/vaccine-safety-
| systems/vaers/index.html)
|
| [EudraVigilance](https://www.ema.europa.eu/en/human-regulatory-
| overview/resea...)
| antirez wrote:
| Why I do not believe this shows Anthropic serves folks a worse
| model:
|
| 1. The percentage drop is too low and oscillating, it goes up and
| down.
|
| 2. The baseline of Sonnet 4.5 (the obvious choice for when they
| have GPU busy for the next training) should be established to see
| Opus at some point goes Sonnet level. This was not done but
| likely we would see a much sharp decline in certain days /
| periods. The graph would look like dominated by a "square wave"
| shape.
|
| 3. There are much better explanations for this oscillation: A)
| They have multiple checkpoints and are A/B testing, CC asks you
| feedbacks about the session. B) Claude Code itself gets updated,
| as the exact tools version the agent can use change. In part it
| is the natural variability due to the token sampling that makes
| runs not equivalent (sometimes it makes suboptimal decisions
| compared to T=0) other than not deterministic, but this is the
| price to pay to have some variability.
| eterm wrote:
| 4. The graph starts January 8.
|
| Why January 8? Was that an outlier high point?
|
| IIRC, Opus 4.5 was released late november.
| littlestymaar wrote:
| Or maybe, juste maybe, that's when they started testing...
| eterm wrote:
| Wayback machine has nothing for this site before today, and
| article is "last updated Jan 29".
|
| A benchmark like this ought to start fresh from when it is
| published.
|
| I don't entirely doubt the degradation, but the choice of
| where they went back to feels a bit cherry-picked to
| demonstrate the value of the benchmark.
| littlestymaar wrote:
| Which makes sense, you gotta wait until you get enough
| data before you can communicate on the said data...
|
| If anything it's coherent with the fact that they very
| likely didn't have data earlier than January the 8th.
| pertymcpert wrote:
| People were away for the holidays. What do you want them to
| do?
| littlestymaar wrote:
| > 1. The percentage drop is too low and oscillating, it goes up
| and down.
|
| How do you define "too low", they make sure to communicate
| about the statistical significance of their measurements,
| what's the point if people can just claim it's "too low" based
| on personal vibes...
| levkk wrote:
| I believe the science, but I've been using it daily and it's
| been getting worse, noticeably.
| warkdarrior wrote:
| Is it possible that your expectations are increasing, not
| that the model is getting worse?
| GoatInGrey wrote:
| Possible, though you eventually run into types of issues
| that you recall the model just not having before. Like
| accessing a database or not following the SOP you have it
| read each time it performs X routine task. There are also
| patterns that are much less ambiguous like getting caught
| in loops or failing to execute a script it wrote after ten
| attempts.
| merlindru wrote:
| yes but i keep wondering if that's just the game of
| chance doing its thing
|
| like these models are nondeterministic right? (besides
| the fact that rng things like top k selection and
| temperature exist)
|
| say with every prompt there is 2% odds the AI gets it
| massively wrong. what if i had just lucked out the past
| couple weeks and now i had a streak of bad luck?
|
| and since my expectations are based on its previous
| (lucky) performance i now judge it even though it isn't
| different?
|
| or is it giving you consistenly worse performance, not
| able to get it right even after clearing context and
| trying again, on the exact same problem etc?
| emp17344 wrote:
| Any chance you're just learning more about what the model is
| and is not useful for?
| jerf wrote:
| I dunno about everyone else but when I learn more about
| what a model is and is not useful for, my subjective
| experience improves, not degrades.
| emp17344 wrote:
| Not when the product is marketed as a panacea.
| data-ottawa wrote:
| There are some days where it acts staggeringly bad, beyond
| baselines.
|
| But it's impossible to actually determine if it's model
| variance, polluted context (if I scold it, is it now closer
| in latent space to a bad worker, and performs worse?),
| system prompt and tool changes, fine tunes and AB tests,
| variances in top P selection...
|
| There's too many variables and no hard evidence shared by
| Anthropic.
| acuozzo wrote:
| No because switching to the API with the same prompt
| immediately fixes it.
|
| There's little incentive to throttle the API. It's $/token.
| davidee wrote:
| I have to concur. And to the question about understanding
| what its good and bad at; no, tasks that it could accomplish
| quickly and easily just a month ago, now require more
| detailed prompting and constant "erroneous direction
| correction."
|
| It's almost as if, as tool use and planning capabilities have
| expanded, Claude (as a singular product) is having a harder
| time coming up with simple approaches that just work, instead
| trying to use tools and patterns that complicate things
| substantially and introduce much more room for errors/errors
| of assumption.
|
| It also regularly forgets its guidelines now.
|
| I can't tell you how many times it's suggested significant
| changes/refactors to functions because it suddenly forgets
| we're working in an FP codebase and suggests inappropriate
| imperative solutions as "better" (often choosing to use
| language around clarity/consistency when the solutions are
| neither).
|
| Additionally, it has started taking "initiative" in ways it
| did not before, attempting to be helpful but without
| gathering the context needed to do so properly when stepping
| outside the instruction set. It just ends up being much
| messier and inaccurate.
|
| I have to regularly just clear my prompt and start again with
| guardrails that have either: already been established, or
| have not been needed previously / are only a result of the
| over-zealousness of the work its attempting to complete.
| conception wrote:
| I assume, after any compacting of the context window that
| the session is more or less useless at that point I've
| never had consistent results after compacting.
| TIPSIO wrote:
| I too suspect the A/B testing is the prime suspect: context
| window limits, system prompts, MAYBE some other questionable
| things that should be disclosed.
|
| Either way, if true, given the cost I wish I could opt-out or
| it were more transparent.
|
| Put out variants you can select and see which one people flock
| to. I and many others would probably test constantly and
| provide detailed feedback.
|
| All speculation though
| make3 wrote:
| It would be very easy for them to switch the various (compute)
| cost vs performance knobs down depending on load to maintain a
| certain latency; you would see oscillations like this,
| especially if the benchmark is not always run exactly at the
| same time every day.
|
| & it would be easy for them to start with a very costly
| inference setup for a marketing / reputation boost, and slowly
| turn the knobs down (smaller model, more quantized model, less
| thinking time, fewer MoE experts, etc)
| IshKebab wrote:
| > We model tests as Bernoulli random variables and compute 95%
| confidence intervals around daily, weekly, and monthly pass
| rates. Statistically significant differences in any of those time
| horizons are reported.
|
| Doesn't really work like that. I'd remove the "statistically
| significant" labelling because it's misleading.
| sroerick wrote:
| My personal conspiracy theory is that they choose who to serve a
| degraded model to based on social graph analysis and sentiment
| analysis, maximizing for persuasion while minimizing compute.
| arcanemachiner wrote:
| Sounds more like a sound business plan than a conspiracy
| theory.
| copilot_king wrote:
| It sounds like fraud to me
| arcanemachiner wrote:
| Does it say anywhere in their terms of service that they
| guarantee the quality of the model, or promise not to
| modify it?
|
| https://www.anthropic.com/legal/consumer-terms
|
| https://www.anthropic.com/legal/commercial-terms
| copilot_king wrote:
| IMO this strategy seems inspired by TikTok's approach for
| retaining new uploaders.
|
| TikTok used to give new uploaders a visibility boost (i.e., an
| inflated number of likes and comments) on their first couple of
| uploads, to get them hooked on the the service.
|
| In Anthropic/Claude's case, the strategy is (allegedly) to give
| new users access to the premium models on sign-up, and then
| increasingly cut the product with output from cheaper models.
|
| Of course, your suggestion (better service for users who know
| how to speak Proper English) would be the cherry on top of this
| strategy.
|
| From what I've seen on HackerNews, Anthropic is all-in on
| social media manipulation and social engineering, so I suspect
| that your assumption holds water.
| stared wrote:
| Does it benchmark the underlying code (Opus 4.5) or Claude Code
| harness? If the second, I would love to see CC versions involved.
|
| I would be curious to see on how it fares against a constant
| harness.
|
| There were thread claiming that Claude Code got worse with
| 2.0.76, with some people going back to 2.0.62.
| https://github.com/anthropics/claude-code/issues/16157
|
| So it would be wonderful to measure these.
| Jcampuzano2 wrote:
| Claude Code. They mention they are using claude codes CLI in
| the benchmark, and claude code changes constantly.
|
| I wouldn't be surprised if the thing this is actually testing
| is benchmarking just claude codes constant system prompt
| changes.
|
| I wouldn't really trust this to be able to benchmark opus
| itself.
| jampa wrote:
| I am using API mode, and it's clear that there are times when the
| Claude model just gives up. And it is very noticeable because the
| model just does the most dumb things possible.
|
| "You have a bug in line 23." "Oh yes, this solution is bugged,
| let me delete the whole feature." That one-line fix I could make
| even with ChatGPT 3.5 can't just happen. Workflows that I use and
| are very reproducible start to flake and then fail.
|
| After a certain number of tokens per day, it becomes unusable. I
| like Claude, but I don't understand why they would do this.
| arcanemachiner wrote:
| Robbing Peter to pay Paul. They are probably resource-
| constrained, and have determined that it's better to supply a
| worse answer to more people than to supply a good answer to
| some while refusing others. Especially knowing that most people
| probably don't need the best answer 100% of the time.
| chrisjj wrote:
| > Especially knowing that most people probably don't need the
| best answer 100% of the time.
|
| More: probably don't know if they've got a good answer 100%
| of the time.
|
| It is interesting to note that this trickery is workable only
| where the best answers are sufficiently poor. Imagine they
| ran almost any other kind of online service such email, stock
| prices or internet banking. Occasionally delivering only half
| the emails would trigger a customer exodus. But if normal
| service lost a quarter of emails, they'd have only customers
| who'd likely never notice half missing.
| bn-l wrote:
| Right. You can launder quantization that way by muddying the
| waters of discourse about the model.
| DanielHall wrote:
| I encountered the same situation too; Claude has 'become lazy'.
| WhitneyLand wrote:
| First off, this is a cool project, look forward to some
| interesting insights.
|
| I would suggest adding some clarification to note that longer
| measure like 30 pass rate is raw data only while the
| statistically significant labels apply only to change.
|
| Maybe something like Includes all trials, significance labels
| apply only to confidence in change vs baseline.
| taf2 wrote:
| any chance we can get something like this for codex cli that'd be
| cool too compare
| esafak wrote:
| Finally someone did it! We need this for all models.
| Topfi wrote:
| I have yet to experience any degradation in coding tasks I use to
| evaluate Opus 4.5, but I did see a rather strange and
| reproducible worsening in prompt adherence as part of none coding
| tasks since the third week of January.
|
| Very simple queries, even those easily answered via regular web
| searching, have begun to consistently not result accurate results
| with Opus 4.5, despite the same prompts previously yielding
| accurate results.
|
| One of the tasks that I already thought was fully saturated as
| most recent releases had no issues in solving it was to request a
| list of material combinations for fabrics used in bag
| constructions that utilise a specific fabric base. In the last
| two weeks, Claude has consistently and reproducibly provided
| results which deviate from the requested fabric base, making the
| results inaccurate in a way that a person less familiar with the
| topic may not notice instantly. There are other queries of this
| type for other topics I am nerdily familiar with to a sufficient
| degree to notice such deviations from the prompt like motorcycle
| history specific queries that I can say this behaviour isn't
| limited to the topic of fabrics and bag construction.
|
| Looking at the reasoning traces, Opus 4.5 even writes down the
| correct information, yet somehow provides an incorrect final
| output anyways.
|
| What makes this so annoying is that in coding tasks, with
| extensive prompts that require far greater adherence to very
| specific requirements in a complex code base, Opus 4.5 does not
| show such a regression.
|
| I can only speculate what may lead to such an experience, but for
| none coding tasks I have seen regression in Opus 4.5 whereas for
| coding I did not. Not saying there is none, but I wanted to point
| it out as such discussions are often primarily focused on coding,
| where I find it can be easier to see potential regressions where
| their are none as a project goes on and tasks become inherently
| more complex.
|
| My coding benchmarks are a series of very specific prompts
| modifying a few existing code bases in some rather obscure ways,
| with which I regularly check whether a model does severely
| deviate from what I'd seen previously. Each run starts with a
| fresh code base with some fairly simple tasks, then gets
| increasingly complex with later prompts not yet being implemented
| by any LLM I have gotten to test. Partly that originated from my
| subjective experience with LLMs early on, where I found a lot of
| things worked very well but then as the project went on and I
| tried more involved things with which the model struggled, I felt
| like the model was overall worse when in reality, what had
| changed were simply the requirements and task complexity as the
| project grew and easier tasks had been completed already. In this
| type of testing, Opus 4.5 this week got as far and provided a
| result as good as the model did in December. Of course, past
| regressions were limited to specific users, so I am not saying
| that no one is experiencing reproducible regressions in code
| output quality, merely that I cannot reproduce them in my
| specific suite.
| epolanski wrote:
| I definitely noticed a degradation, it feels regressed by a
| generation.
| dudeinhawaii wrote:
| I've noticed a degradation in Opus 4.5, also with Gemini-3-Pro.
| For me, it was a sudden rapid decline in adherence to specs in
| Claude Code. On an internal benchmark we developed,
| Gemini-3-Pro also dramatically declined. Going from being
| clearly beyond every other model (as benchmarks would lead you
| to believe) to being quite mediocre. Delivering mediocre
| results in chat queries and coding also missing the mark.
|
| I didn't "try 100 times" so it's unclear if this is an
| unfortunate series of bad runs on Claude Code and Gemini CLI or
| actual regression.
|
| I shouldn't have to benchmark this sort of thing but here we
| are.
| acuozzo wrote:
| Write your work order with phases (to a file) and, between
| each phase, give it a non-negotiable directive to re-read the
| entire work order file.
|
| Claude-Code is terrible with context compaction. This solves
| that problem for me.
| fernvenue wrote:
| That will be great if there's RSS support.
| rplnt wrote:
| The chart would benefit from having weekends highlighted. Or have
| another chart averaged by a weekday.
| copilot_king wrote:
| This strategy seems inspired by TikTok's approach for retaining
| new uploaders.
|
| TikTok used to give new uploaders a visibility boost (i.e., an
| inflated number of likes and comments) on their first couple of
| uploads, to get them hooked on the the service.
|
| In Anthropic/Claude's case, the strategy is (allegedly) to give
| new users access to the premium models on sign-up, and then
| increasingly cut the product with output from cheaper models.
| chrisjj wrote:
| Yes, but the difference is TikTok didn't sell a particular
| service version.
|
| Anthropic did sell a particular model version.
| dmos62 wrote:
| Lack of transparency as regards "thinking power"-consistency is a
| big gripe of mine with LLM providers. It's even worse with
| ChatGPT and the like. E.g. I had to learn the hard way that at
| >45k input tokens ChatGPT 5.2 Thinking Extended bumps its
| intelligence down so hard that it can't follow basic instructions
| (or it somehow truncates the input, losing the instructions). It
| sucks to lose confidence in an otherwise great tool. I would 100x
| prefer being forced to back-off, or getting a straight-no, than
| getting silently downgraded. Transparency is a big deal.
| judahmeek wrote:
| Sounds like you ran into the Maximum Effective Context Window:
| https://arxiv.org/abs/2509.21361?context=cs.AI
| dmos62 wrote:
| Interesting article. Not sure it's the same phenomenon. What
| I experienced was like a day and night difference when you go
| from 44.5k to 45.5k. Didn't notice any fluctuation to suggest
| that it's no a hard 45000 limit. I ran many many queries,
| similar problem space, but the problems varied a lot.
| parquor wrote:
| Does this use a claude subscription or key, and has the account
| been used for anything else that day?
|
| On HN a few days ago there was a post suggesting that Claude gets
| dumber throughout the day:
| https://bertolami.com/index.php?engine=blog&content=posts&de...
| sd9 wrote:
| I'm sure there is not enough data here for this to be
| statistically significant (it seems to oscillate too much and not
| show real trends or step changes) - BUT
|
| If this measure were hardened up a little, it would be really
| useful.
|
| It feels like an analogue to an employee's performance over time
| - you could see in the graphs when Claude is "sick" or
| "hungover", when Claude picks up a new side hustle and starts
| completely phoning it in, or when it's gunning for a promotion
| and trying extra hard (significant parameter changes). Pretty
| neat.
|
| Obviously the anthropomorphising is not real, but it is cool to
| think of the model's performance as being a fluid thing you have
| to work with, and that can be measured like this.
|
| I'm sure some people, most, would prefer that the model's
| performance were fixed over time. But come on, this is way more
| fun.
| elmean wrote:
| I KNEW I WASNT CRAZY
| wendgeabos wrote:
| Codex is doing better. Why is everyone silent on Codex?
| https://marginlab.ai/trackers/codex/
| CharlesW wrote:
| Benchmark wins don't necessarily translate to "real world" wins
| vs. Claude Code.
| bn-l wrote:
| Codex writes disgusting shit code.
| drc500free wrote:
| What makes the level they chose a "baseline," against which it
| would be appropriate to do statistical tests?
| PlatoIsADisease wrote:
| Pretty sure someone at Google, OpenAI, and Anthropic met up at a
| park, leaving their phones in their car, and had a conversation
| that January 2026, they were all going to silently degrade their
| models.
|
| They were fighting an arms race that was getting incredibly
| expensive and realized they could get away with spending less
| electricity and there was nothing the general population could do
| about it.
|
| Grok/Elon was left out of this because he would leak this idea at
| 3am after a binge.
| kittikitti wrote:
| This is why I run my own models. All the inference providers do
| sneaky things behind the scenes. They will limit the output
| tokens, turn off attention layers, lower reasoning, or just use a
| completely different model. I'm actually surprised that Claude
| Code experienced this, as I've experienced this the least from
| API and coding agents.
| Rastonbury wrote:
| would be interesting to see what scores it's get when it is
| actually degraded via the status page, it gets degraded pretty
| often, so there's at least something to compare or to know at
| what point Anthropic declares degradation
| mannanj wrote:
| I wonder when I experience noticeably degraded model quality, ie
| opus, is it because my usage falls in the highest buckets and I'm
| being shadow limited or served worse versions of opus or is it
| because of actual server load/burden?
|
| It wouldn't be the first time companies have secret shadow
| algorithms running to optimize things and wouldn't it be obvious
| to limit power users as matter of cost/profit and not tell them.
| (See history of "Shadow ban" though that's for different reasons)
| crazygringo wrote:
| > _We model tests as Bernoulli random variables and compute 95%
| confidence intervals around daily, weekly, and monthly pass
| rates. Statistically significant differences in any of those time
| horizons are reported._
|
| They're going to need to provide a lot more detail on their
| methodology, because that doesn't make a lot of sense. From their
| graphs, they seem to be calculating the confidence interval
| around the previous value, then determining whether the new value
| falls outside of it. But that's not valid for establishing the
| statistical significance of a _difference_. You need to calculate
| the confidence interval _of the difference itself_ , and then see
| if _all the values within that confidence interval remain
| positive_ (if it excludes 0). This is because _both_ the old
| _and_ new measurement have uncertainty. Their approach seems to
| be only considering uncertainty for one of them.
|
| They should also really be more specific about the time periods.
| E.g. their graphs only show performance over the past 30 days,
| but presumably the monthly change is comparing the data from 60
| to 31 days ago, to the data from 30 days ago until yesterday? In
| which case the weekly graph really ought to be displaying the
| past _two_ months, not one month.
| steveBK123 wrote:
| New to me, but I am starting to infer that for those "in the
| know" it is common knowledge on HN that LLMs are purposely
| degraded over time to manage capacity/cost or fudge benchmarks...
|
| How do you actually use these in production pipelines in practice
| then?
|
| Are LLMs even well suited for some of the document parsing / data
| scrubbing automation people are throwing at them now?
| mrandish wrote:
| Benchmark tracking of cloud AI performance is going to be crucial
| going forward. Vendors are selling a service that by its nature
| is _very_ difficult for customers to gauge day to day. How will I
| know if a code revision is ~2.5% less good today than it would
| have been yesterday? Or if queries during peak load hours use one
| less 'expert' in their MoE?
|
| Yet vendor's costs to deliver these services are skyrocketing,
| competition is intense and their ability to subsidize with
| investor capital is going away. The pressure on vendors to reduce
| costs by dialing back performance a few percent or under-
| resourcing peak loads will be overwhelming. And I'm just a
| hobbyist now. If I was an org with dozens or hundreds of devs I'd
| want credible ways to verify the QoS and minimum service levels
| I'm paying for are being fulfilled long after a vendor has won
| the contract.
| biddit wrote:
| Call it what you will. But the experience is like you have a
| reliable coworker, but he randomly decides to take bong hits.
|
| "No no yeah bro no I'm good like really the work's done and all
| yeah sorry I missed that let me fix it"
| bn-l wrote:
| I hope the author sees this:
|
| You have to test inter-day variation. Many have noticed a sudden
| drop off at certain times.
| trq_ wrote:
| Hi everyone, Thariq from the Claude Code team here.
|
| Thanks for reporting this. We fixed a Claude Code harness issue
| that was introduced on 1/26. This was rolled back on 1/28 as soon
| as we found it.
|
| Run `claude update` to make sure you're on the latest version.
| isaacdl wrote:
| Anywhere we can read more about what a "harness issue" means?
| What was the impact of it?
| hu3 wrote:
| Hi. Do you guys have internal degradation tests?
| awestroke wrote:
| [flagged]
| dang wrote:
| Please don't post shallow dismissals or cross into personal
| attack in HN discussions.
|
| https://news.ycombinator.com/newsguidelines.html
| stbtrax wrote:
| I assume so to make sure that they're rendering at 60FPS
| conception wrote:
| You joke but having CC open in the terminal hits 10% on my
| gpu to render the spinning thinking animation for some
| reason. Switch out of the terminal tab and gpu drops back
| to zero.
| gpm wrote:
| That sounds like an issue with your terminal more than an
| issue with CC...
| reissbaker wrote:
| Surely you mean 6fps
| vmg12 wrote:
| It happened before 1/26. I noticed when it started modifying
| plans significantly with "improvements".
| persedes wrote:
| What would be cool if this somehow could do a comparison by
| provider. E.g. in the last outages anthropic models running on
| vertex were apparently less affected than those deployed
| elsewhere. (Not saying that one is better than the other, but
| would be a neat read out).
| account266928 wrote:
| Please try to make this statistically rigorous. There's lots of
| advice in this thread (intraday variation, etc) but if Im reading
| this right it looks like the CI includes the baseline value yet
| you still label this as failing.
|
| Wouldn't this just be "our test isn't powerful enough to find a
| signal if there were one here?"
|
| People will see this and derive strong conclusions that the data
| don't support and you, `qwesr123`, or "JB" from your blogs, will
| be responsible.
| _zachs wrote:
| This is super important - even if it's not currently the best
| measure of degradation yet. Anecdotally, Opus 4.5 has gotten so
| bad for me it's almost adding time to my workflow instead saving
| it. It'd be nice to have more 3rd party measurements like this to
| hold Anthropic accountable.
| snissn wrote:
| they should run their test against a control baseline such as an
| open source hosted model to see the overall drift in their test
| motoboi wrote:
| I'd love to see, based on the level of non-determinism perfomance
| on the benchmark how many times you need to run the benchmark for
| the change to be relevant (or statistically significant if you
| want).
|
| That would be a nice paper.
| willturman wrote:
| Could this be (partially?) explained by Model Collapse [1], i.e.
| iteratively training on data that includes an ever increasing
| amount of AI slop?
|
| [1] https://thebullshitmachines.com/lesson-16-the-first-step-
| fal...
___________________________________________________________________
(page generated 2026-01-29 23:00 UTC)