[HN Gopher] Show HN: AutoThink - Boosts local LLM performance wi...
___________________________________________________________________
Show HN: AutoThink - Boosts local LLM performance with adaptive
reasoning
I built AutoThink, a technique that makes local LLMs reason more
efficiently by adaptively allocating computational resources based
on query complexity. The core idea: instead of giving every query
the same "thinking time," classify queries as HIGH or LOW
complexity and allocate thinking tokens accordingly. Complex
reasoning gets 70-90% of tokens, simple queries get 20-40%. I also
implemented steering vectors derived from Pivotal Token Search
(originally from Microsoft's Phi-4 paper) that guide the model's
reasoning patterns during generation. These vectors encourage
behaviors like numerical accuracy, self-correction, and thorough
exploration. Results on DeepSeek-R1-Distill-Qwen-1.5B: - GPQA-
Diamond: 31.06% vs 21.72% baseline (+43% relative improvement) -
MMLU-Pro: 26.38% vs 25.58% baseline - Uses fewer tokens than
baseline approaches Works with any local reasoning model -
DeepSeek, Qwen, custom fine-tuned models. No API dependencies. The
technique builds on two things I developed: an adaptive
classification framework that can learn new complexity categories
without retraining, and an open source implementation of Pivotal
Token Search. Technical paper:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5253327 Code
and examples:
https://github.com/codelion/optillm/tree/main/optillm/autoth...
PTS implementation: https://github.com/codelion/pts I'm curious
about your thoughts on adaptive resource allocation for AI
reasoning. Have you tried similar approaches with your local
models?
Author : codelion
Score : 380 points
Date : 2025-05-28 02:39 UTC (20 hours ago)
| codelion wrote:
| The motivation for AutoThink came from watching how current
| reasoning models waste computation - they spend the same amount
| of "thinking time" on "what's 2+2?" as they do on complex
| mathematical proofs. This seemed obviously inefficient.
|
| The breakthrough was combining two techniques I'd been working on
| separately: adaptive classification (which can learn new
| categories without retraining) and an open source implementation
| of Pivotal Token Search from Microsoft's Phi-4 paper. When I put
| them together with dynamic token budgeting, the performance gains
| were much better than expected.
|
| What surprised me most was that the technique actually uses fewer
| tokens on average while improving performance. The adaptive
| allocation means simple queries finish faster, offsetting the
| extra computation on complex ones.
|
| A few technical notes:
|
| - The steering vectors are small (typically <1MB per pattern) and
| add minimal memory overhead
|
| - Classification adds about 10ms latency, which is negligible
|
| - Target layer selection matters - I found middle layers (15-20)
| work best for most models
|
| I'd love feedback on:
|
| - Have you tried similar adaptive approaches with your models?
|
| - What other reasoning patterns would be useful to steer toward?
|
| - Ideas for automatically detecting the optimal target layer?
|
| Thanks for checking it out! Happy to answer any questions about
| the implementation or results.
| behnamoh wrote:
| > they spend the same amount of "thinking time" on "what's
| 2+2?" as they do on complex mathematical proofs.
|
| Not anymore. Have you seen Gemini 2.5 Pro? Ask it simple
| questions and it almost doesn't "think". Ask it a coding
| question and it'll write a long reasoning article. I think the
| same goes for o3.
| codelion wrote:
| Yes, we started with the idea of trying to replicate similar
| control on thinking processes for open reasoning models. They
| also announced the Deep Think approach at IO which goes even
| further and combines parallel CoTs at inference.
| sigmoid10 wrote:
| The original o1 also didn't do this. Neither did the actual
| DeepSeek R1. You could even get it to answer immediately
| without any reasoning tokens. These highly distilled versions
| just lost most of their common sense for this.
| shing3232 wrote:
| Well, it does overthink quite a bit. if It can reduce
| overthink,it s gonna be useful
| victorbjorklund wrote:
| Overthink is subjectibe. It really depends on how much
| you value the answer.
|
| "how long break distance does a train need if going in
| 100 km/hour?"
|
| Just need a quick reply and you dont care so much (maybe
| showerthought)? Or is life and death depending on the
| answer?
|
| The same question can need different amount of thinking.
| normie3000 wrote:
| > is life and death depending on the answer?
|
| In this situation I suspect you'd still want the answer
| quickly.
| diggan wrote:
| Huge assumption, there is a wide range of various
| parameters that goes into how accurate you need an
| response to be, depending on context. As sure as there
| exists questions that you need 100% accurate response
| regardless of response times, I'm sure there exists
| questions on the other extreme.
| GTP wrote:
| In this situation you would have someone with actual
| knowledge of the mechanics involved do the computation
| using the actual data (e.g., what's the mass of the
| train? Which kind of breaks does it have?) instead of
| asking an LLM and trusting it to give the correct answer
| without checking.
| TeMPOraL wrote:
| Assuming you could find an expert like that in time, and
| that they will then be able to understand and solve the
| problem fast enough to still be helpful.
|
| If you need the answer within a couple hours, you can
| probably get it for an expert; if you need to get an
| actionable answer within minutes, based on some back-of-
| the-envelope calculations, then a SOTA LLM is a _much_
| safer bet than flagging whoever seems the smartest in the
| room and asking them for help.
| CjHuber wrote:
| What I really don't like is that I can't manually decide how
| much thinking it Gemini should allocate to a prompt. You're
| right sometimes it doesn't think but for me this also happens
| on complex query where I WOULD want it to think. Even things
| like "super think about this" etc don't help, it just refuses
| to
| thegeomaster wrote:
| Gemini 2.5 Pro is getting thinking budgets when it GAs in
| June (at least that's the promise).
| vladf wrote:
| This is available for Flash
| CharlesW wrote:
| > _I think the same goes for o3._
|
| Definitely, in my experience. Elsewhere in the thread, OP
| says that open models/systems _don 't_ do this, in which case
| this seems like important work toward making open
| alternatives competitive.
| olddustytrail wrote:
| Is that not just caching? If you have the same query just
| return the same response.
|
| You could even put a simpler AI in front to decide if it was
| effectively the same query.
| mclau157 wrote:
| Has Gemini or OpenAI put out any articles on this or is this
| just something you noticed?
| Abishek_Muthian wrote:
| Congratulations! Any work to optimise efficiency w.r.t LLMs is
| much appreciated.
|
| So far I've taken only lazy approach to optimising local LLMs
| by sending small queries to my M4 Mac Mini running MLX models
| and sending larger queries to my Nvidia 4090; it's remarkable
| how efficient M4 is compared to Nvidia and I think Apple is in
| the right direction with MLX.
|
| I would read about AutoThink and try to integrate it with my
| workflow.
| Lerc wrote:
| I have thought it might be worth seeding responses with the
| output of non-reasoning models, so after the user prompt,
| inject a block of "a non-reasoning model thought this:... stuff
| ....Was that what the user wanted?" For the instances where the
| non reasoning version was sufficient it might help the
| reasoning model get to the point earlier.
| codelion wrote:
| This is an interesting idea, I hadn't thought of it. It is
| worth experimenting I am not aware of anyone else trying it
| yet.
| waffletower wrote:
| Claude Sonnet 3.5 (not even the latest iterations: 3.7 or 4)
| clearly adapts processing time to query complexity --
| processing time is dynamic.
| nssnsjsjsjs wrote:
| This is an obvious optimisation. Surprised this isn't been done
| already. Good job writing it up and showing how it can be done.
| transfire wrote:
| That's awesome!
|
| Now have it mark blocks of text on or off, so it can ignore
| irrelevant, or worse erroneous material -- no need to include it
| in the context window.
| pkoird wrote:
| Back to TF IDF we go.
| knuppar wrote:
| One could argue TF-IDF is a case of an attention layer... but
| not quadratic in inference/training and kinda just a
| quotient. Yeah maybe we should go back
| codelion wrote:
| This sounds like an interesting idea, can you elaborate more
| may be with a concrete example. I am wondering if this can be
| implemented easily as a plugin in optillm.
| shah_akshat wrote:
| Surprised this didn't exist. Great work @codelion
| danielhanchen wrote:
| Super cool and the results look pretty solid as well! Will give
| it a try!
| keeganpoppen wrote:
| i have definitely observed a similar pattern in the Big Label
| Foundation Models... so, i'm glad to see it in this realm too <3
| bufferoverflow wrote:
| But how do you classify a question as high vs low complexity?
| Some seemingly simple questions can turn out to be very very
| complex. For example, integer solution to x3 +
| y3 + z3 = 42
|
| took over a hundred years of compute time to find.
|
| Or another seemingly simple equation with positive integers x,y,z
| x/(y+z)+y/(z+x)+z/(x+y) = 4
|
| requires elliptic curve knowledge, and the solution is huge
| x = 1544768021087461664419513150199198374856643256695654317000266
| 34898253202035277999 y = 368751317941299998271978115
| 65225474825492979968971970996283137471637224634055579
| z = 4373612677928697257861252602371390152816537558161613618621437
| 993378423467772036
|
| (Solution is discussed here: https://www.quora.com/How-do-you-
| find-the-positive-integer-s...)
| codelion wrote:
| Query complexity in this context is based on how many tokens it
| took for the model to respond to a query correctly based on a
| ground truth dataset like GSM8k. The adaptive classifier learns
| over this dataset and then we use it at inference for
| classification.
| bufferoverflow wrote:
| So it can be very very wrong.
|
| You're trading correctness for speed.
| codelion wrote:
| Yes, the goal here is to avoid overthinking and be as
| efficient as possible in terms of the minimal tokens
| required to solve a query. Often, queries that require too
| many tokens are unlikely to lead to correct answers anyways
| otherwise they would show up when we are learning the
| classifier.
| VagabundoP wrote:
| If you ask it to rethink the problem again because you've
| found a flaw, does it bump up the complexity and actually
| think about it. Like a person might give you a quick
| answer to something and then questioning the answer would
| cause them to think deeper about it.
| codelion wrote:
| The short answer is in general yes it helps improve the
| accuracy, there is a whole line of work on self
| consistency and critique that supports it. Many of those
| approaches are already implemented in optillm.
| baobabKoodaa wrote:
| Yes, if you only care about correctness, you always use the
| maximum possible inference compute. Everything that does
| not do that is trading correctness for speed.
| wat10000 wrote:
| If compute is limited, then dedicating more resources to
| the questions that are more likely to need it will increase
| correctness overall, even if it may decrease correctness
| for some individual responses.
| xigency wrote:
| > You're trading correctness for speed.
|
| That's AI in a nutshell.
| MrManatee wrote:
| I think there exists a separate skill for classifying problems
| by difficulty, apart from being able to solve them. This skill
| can be developed from both directions by learning which
| problems have been solved and which haven't been.
|
| If someone asked me to find solutions to these example
| equations, there are three complications that I would
| immediately notice:
|
| 1. We are looking for solutions over integers. 2. There are
| three variables. 3. The degree of the equation is 3.
|
| Having all three is a deadly combination. If we were looking
| for solutions over reals or complex numbers? Solvable. Less
| than three variables? Solvable. Degree less than 3? Solvable.
| With all three complications, it's still not _necessarily_
| hard, but now it might be. We might even be looking at an
| unsolved problem.
|
| I haven't studied enough number theory to actually solve either
| of these problems, but I have studied enough to know where to
| look. And because I know where to look, it only takes me a few
| seconds to recognize the "this might be very difficult" vibe
| that both of these have. Maybe LLMs can learn to pick up on
| similar cues to classify problems as difficult or not so
| difficult without having needing to solve them. (Or, maybe they
| have already learned?)
| vintermann wrote:
| If host models for others, then sure, I'm happy to save some
| computation time for really simple queries. Sure the cost is that
| the model will be effectively dismissive of questions it judges
| to be "easy", but I'm not the one carrying that cost I suppose.
|
| However, for a local model, answering my own queries? That's the
| last thing I want. I already spent way too much money on that
| GPU, might as well get use out of it.
| MagicMoonlight wrote:
| You didn't invent this. Models like o3 already do it, that's why
| the amount of thinking time varies.
| rohansood15 wrote:
| He's not claiming he did. It says right there that it's an
| open-source implementation to run with local models.
| shwouchk wrote:
| Very interesting, thanks for sharing!
|
| FWIW gemini explicitly told me that it ranks question difficulty
| from 1 to 100 and depending on the bin allocates more or less
| resources to answering it
| NitpickLawyer wrote:
| > gemini explicitly told me
|
| Do you mean someone from the gemini team? If you "asked" the
| LLM then it's likely a "hallucinated" answer. They say all sort
| of things about "themselves" only because they were trained to
| do so. They likely have 0 knowledge about their true
| architecture.
| throwaway314155 wrote:
| Gemini has no access to its internal processes outside of whats
| in its system prompt - and even then LLM's are known to
| fabricate information about their inception.
| casenmgreen wrote:
| It seems to me inadvisable to say "think" and "reason", because
| those words have particular meanings, and those particular
| meanings are not in use by LLMs.
|
| They are a computing method, where we can choose to use more or
| less run time (and so processor time), to generate results.
| falcor84 wrote:
| The ship has sailed, just like "computers" once referred to a
| human profession and now referred to machines.
| dgb23 wrote:
| My worldview is materialist and deterministic in principle. But
| day to day I'm an existentialist with a touch of spiritualism.
|
| To me, a fairly pragmatic way of characterizing these tools day
| to day is to anthropomorphize them. One benefit of this
| heuristic: they simulate conversation and it's much easier to
| use them with a conversational flow. Another one is to create
| an approximation of a character, which makes it easier to build
| a useful intuition for what they can and cannot do.
|
| Obviously these kinds of heuristics do break down. But it's
| obvious enough when they do so one can switch into a more
| precise and analytical mode of thinking.
| dymk wrote:
| When you "ping" and IP address, are you bouncing sound waves
| off of the metal hull of the other computer? No, but the word
| is used anyways, as it's a useful metaphor for what's really
| going on.
| mentalgear wrote:
| It's great how small models help small teams and individual
| researchers everywhere now compete with big AI labs by allowing
| them to demonstrate new innovative approaches on small
| experiments.
|
| Also, as small language models (SML) become more competent, it's
| amazing what they can do on-device !
| chrisweekly wrote:
| > "small language models (SML)"
|
| that should be SLM, right?
| Dowwie wrote:
| Hey, this is really interesting. What are the features you used
| to measure the reasoning complexity? In other words, how does one
| evaluate a query during classification?
| codelion wrote:
| We use an adaptive classifier to learn how many tokens the
| model takes to respond correctly on a known dataset. I used the
| https://huggingface.co/adaptive-classifier/llm-router for
| experiments it is based on distilbert.
| lostmsu wrote:
| Official https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-
| Qwen-... says GPQA-Diamond is 33.8
| codelion wrote:
| Hey, yes the reported results do not restrict any time limit or
| token limit for the benchmarks. We run our baseline with the
| same config 0.6 temp and max_token 32k but we set a timeout
| after 600 secs. Otherwise it would take forever to benchmark
| with the resources we had. I have a note in the actual paper on
| that in the implementation details section.
| lostmsu wrote:
| GPQA-Diamond is 200 questions. Any GPU since 2019 with 12GB
| of VRAM should be able to run tens if not hundreds of queries
| for a 1.5B model in parallel.
| codelion wrote:
| If we try to benchmark GPQA-Diamond with DeepSeek-R1 in the
| suggested configuration of 0.6 temp and 32k max_tokens and
| say if every instance takes the maximum tokens it will
| require 6.4 M tokens. Which without batching on a single
| H100 at 80 tok/s will take 23 hrs to run. To run with 32k
| context length on a single H100 a 1.5B model will require
| ~15-20 GB VRAM so you cannot run 10s or 100s of queries in
| parallel.
|
| MMLU-PRO is 12,000 instances. To avoid this we set a 600
| seconds timeout for each instance to run.
| NiloCK wrote:
| I, too, built a POC autothink shortly after the Claude 3.7
| release that included the `extended thinking` toggle. It's
| literally also called autothink:
|
| https://github.com/NiloCK/autothink
|
| https://www.paritybits.me/think-toggles-are-dumb/
|
| My own version took a first pass with an LLM whose job was to
| assign a 0-100 complexity rating, and then there was more or less
| a linear scaling of the allocated thinking budget.
|
| The OP effort here is obviously higher grade, and I'm really
| tickled to see quantitative results. Well done.
| GENIXUS wrote:
| I'm very new to the world of LLMs and AI, but this project really
| caught my attention.
|
| From what I understood, AutoThink helps the AI "think more
| wisely" by adjusting how much effort it spends based on how hard
| the question is. That makes a lot of intuitive sense -- like how
| people don't spend 10 minutes figuring out what 2+2 is, but do
| take time with tricky problems.
|
| Even though I don't know the technical parts (like token
| budgeting or steering vectors), it's fascinating to see how these
| methods can make the AI both faster and smarter at the same time.
|
| Thanks for sharing -- I'm definitely going to follow this kind of
| work more closely from now on.
| CMay wrote:
| In terms of reasoning models like QwQ or Qwen 3 I didn't waste
| too much time trying to improve their results aside from coming
| up with various ways to constrain their reasoning token output
| with prompts.
|
| Even though Gemma 3 27B QAT is not a reasoning model, it's so
| good at instruction following and being used in LLM chains/routes
| that it can be used for classifying/language optimization steps
| before instructing it how to reason about the prompt in the next
| step. You can even have it output intermediate answers
| interspersed between multiple think tags in the same response. In
| many ways for these models I just define thinking as any tokens
| that are helping the model arrive at the conclusion, but are not
| fully formed parts of the answer.
|
| Instructing it to use certain words (tokens) and types of
| phrasing preferentially is something that is known to improve
| results in general, not just in LLMs and I've seen improved
| results by encouraging certain types of language to be used.
| AutoThink using the highest performing tokens out of a dataset
| _could_ be a nice way to optimize towards that in a more general
| way.
|
| It seems like there's a risk of using so many pivotal tokens that
| it almost overfits responses to benchmark questions, though. So,
| while I have personally seen careful word/token selection improve
| result quality and also see it as a potential low cost high
| return optimization, I'd still want to see how AutoThink
| generalizes.
| SamScout wrote:
| Great food for thought! We will discuss this approach as we find
| our evolving AI-crawler should ideally be able to recognize when
| a site we visit needs more vs. less queries.
|
| For context, we're samaritanscout.org a search engine that is
| attempting to provide a comprehensive view into all local
| volunteering opportunities posted on a range of nonprofit
| websites.
___________________________________________________________________
(page generated 2025-05-28 23:01 UTC)