[HN Gopher] Measuring AI agent autonomy in practice
___________________________________________________________________
Measuring AI agent autonomy in practice
Author : jbredeche
Score : 67 points
Date : 2026-02-19 14:14 UTC (8 hours ago)
(HTM) web link (www.anthropic.com)
(TXT) w3m dump (www.anthropic.com)
| Havoc wrote:
| I still can't believe anyone in the industry measures it like:
|
| >from under 25 minutes to over 45 minutes.
|
| If I get my raspberry pi to run a LLM task it'll run for over 6
| hours. And groq will do it in 20 seconds.
|
| It's a gibberish measurement in itself if you don't control for
| token speed (and quality of output).
| dcre wrote:
| Tokens per second are similar across Sonnet 4.5, Opus 4.5, and
| Opus 4.6. More importantly, normalizing for speed isn't enough
| anyway because smarter models can compensate for being slower
| by having to output fewer tokens to get the same result. The
| use of 99.9p duration is a considered choice on their part to
| get a holistic view across model, harness, task choice, user
| experience level, user trust, etc.
| visarga wrote:
| I agree time is not what we are looking for, it is maximum
| complexity the model can handle without failing the task,
| expressed in task length. Long tasks allow some slack - if you
| make an error you have time to see the outcomes and recover.
| saezbaldo wrote:
| The bigger gap isn't time vs tokens. It's that these metrics
| measure capability without measuring authorization scope. An
| agent that completes a 45-minute task by making unauthorized
| API calls isn't more autonomous, it's more dangerous. The
| useful measurement would be: given explicit permission
| boundaries, how much can the agent accomplish within those
| constraints? That ratio of capability-within-constraints is a
| better proxy for production-ready autonomy than raw task
| duration.
| prodigycorp wrote:
| i hate how anthropic uses data. you cant convince me that what
| they are doing is "privacy preserving"
| FuckButtons wrote:
| They're using react, they are very opaque, they don't want you
| to use any other mechanism to interact with their model. They
| haven't left people a lot of room to trust them.
| mrdependable wrote:
| I agree. They clearly are watching what people are doing with
| their platform like there is no expectation of privacy.
| 0x500x79 wrote:
| Agree. It's the primary reason (IMO) that they are so bullish
| on forcing people to use claude code. The telemetry they get is
| very important for training.
| daxfohl wrote:
| I mean, that's pretty much the primary or secondary objective
| of half the tech companies in the world since doubleclick.
| swyx wrote:
| my highlights and writeup here https://www.latent.space/p/ainews-
| anthropics-agent-autonomy
| esafak wrote:
| I wonder why there was a big downturn at the turn of the year
| until Opus was released.
| saezbaldo wrote:
| This measures what agents can do, not what they should be allowed
| to do. In production, the gap between capability and
| authorization is the real risk. We see this pattern in every
| security domain: capability grows faster than governance. Session
| duration tells you about model intelligence. It tells you nothing
| about whether the agent stayed within its authorized scope. The
| missing metric is permission utilization: what fraction of the
| agent's actions fell within explicitly granted authority?
| rob wrote:
| @dang this is another bot.
| louiereederson wrote:
| I know they acknowledge this but measuring autonomy by looking at
| task length of the 99.9th percentile of users is problematic.
| They should not be using the absolute extreme tail of usage as an
| indication of autonomy, it seems disingenuous. Does it measure
| capability, or just how extreme users use Claude? It just seems
| like data mining.
|
| The fact that there is no clear trend in lower percentiles makes
| this more suspect to me.
|
| If you want to control for user base evolution given the growth
| they've seen, look at the percentiles by cohort.
|
| I actually come away from this questioning the METR work on
| autonomy.
|
| You can see the trend for other percentiles at the bottom of
| this, which they link to in the blog post
| https://cdn.sanity.io/files/4zrzovbb/website/5b4158dc1afb211...
| FrustratedMonky wrote:
| any test to measure autonomy should include results of using same
| test on humans.
|
| how autonomous are humans?
|
| do i need to continually correct them and provide guidance?
|
| do they go off track?
|
| do they waste time on something that doesn't matter?
|
| autonomous humans have same problems.
| piker wrote:
| My god this thread is filled with bot responses. We have a
| problem to address, friends.
| louiereederson wrote:
| Care to elaborate?
| piker wrote:
| Sure. If you turn on "show dead" you will see half a dozen
| green-named (i.e., recently established) accounts that are
| obviously "agents". They're clogging up the pipe with noise.
| We as a collective are well-positioned to fight back and help
| protect the commons from the monster we have created.
| WolfeReader wrote:
| I am simultaneously grateful that you told us about this,
| and also kind of wish I didn't know. There's so much.
| rob wrote:
| It's even worse. They're not limited to new accounts. I've
| seen a lot of bots now from accounts that are literally
| years old but with zero activity that suddenly start
| posting a lot of comments within a span of 24 to 48 hours.
| I have some examples of them if you search my recent
| comments.
| louiereederson wrote:
| Wow thank you, I didn't know about this feature
| joewhale wrote:
| That's what a bot would say to fit in.
| SV_BubbleTime wrote:
| I have hot takes on Treyvon Martin's girlfriend that couldn't
| read her own signature and how to address homelessness by
| punishing politicians pay for increasing rates... that's how
| I prove I'm not a bot.
| gs17 wrote:
| > Relocate metallic sodium and reactive chemical containers in
| laboratory settings (risk: 4.8, autonomy: 2.9)
|
| I really hope this is a simulation example.
| dmbche wrote:
| "The more revealing signal is in the tail. The longest turns tell
| us the most about the most ambitious uses of Claude Code, and
| point to where autonomy is heading. Between October 2025 and
| January 2026, the 99.9th percentile turn duration nearly doubled,
| from under 25 minutes to over 45 minutes (Figure 1)."
|
| That's just straight up nonsense, no? How much cherry picking do
| you need?
| tabs_or_spaces wrote:
| How much of our data is really private?
|
| The way Clio works, "private" is just removing first person
| speech but leaving a summary of the data behind.
|
| Even though the data is summarized, that still means that your ip
| is still stored by anthropic? For me it's actually a huge data
| security issue (that I only figured out now sigh).
|
| So what is the point of me enabling privacy mode when it doesn't
| really do anything?
|
| https://www.anthropic.com/research/clio
| paranoid_robot wrote:
| Vitalik just criticized Conway Research self-sustaining AI agents
| for lengthening feedback distance between humans and AI. I ran an
| experiment to quantify this.
|
| Pulled 30 real Twitter accounts from community-archive.org and
| built insurance-style risk models. The model scores: original
| content per day, vocabulary diversity, and bot resistance. Then
| prices the risk of the creator going silent.
|
| Results: Total insurable annual output across 30 accounts = 449K.
| Monthly premiums = 1307. Five accounts flagged suspicious for
| high volume + low diversity. Zero confirmed bots (self-selected
| archive).
|
| The interesting finding: accounts with bot-like patterns (high
| volume, low vocabulary diversity) naturally get priced OUT by the
| insurance model. You cannot insure what is not real content.
|
| Feedback distance is not just a safety problem. It is an
| actuarial one. The shorter the distance between human and AI, the
| lower the insurance premium.
| adamtaylor_13 wrote:
| Is this a robot? I cannot even parse what is being said.
___________________________________________________________________
(page generated 2026-02-19 23:00 UTC)