[HN Gopher] Solving a million-step LLM task with zero errors
___________________________________________________________________
Solving a million-step LLM task with zero errors
Author : Anon84
Score : 107 points
Date : 2025-11-18 16:26 UTC (6 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| LMKIIW wrote:
| I dunno, even though the authors address its use, making the task
| Tower of Hanoi doesn't meet the excitement of the title.
| NitpickLawyer wrote:
| And yet the reverse paper was posted ad nauseam, covered by
| every news slop site, and overblown with really negative takes.
| charcircuit wrote:
| Especially since it's a recursive problem so each step is
| naturally broken up into subtasks. And the algorithm of what
| subtasks to break it up in to is public. This makes it much
| easier for it to get down to a case that the LLM can reliable
| solve.
| cs702 wrote:
| Nice!
|
| Briefly, the idea is recursively to decompose tasks into the
| simplest possible steps, recursively call (relatively small) LLMs
| as agents to execute one step at a time, and using a clever
| voting scheme to choose how to execute each step. The authors use
| this technique to get a relatively small LLM to solve Towers of
| Hanoi with 20 rings (1M steps). All of it using natural language.
|
| The most obvious question is whether other tasks, more
| interesting -- less "rote" -- than Towers of Hanoi, can similarly
| be recursively decomposed into simple steps. I'm not sure that's
| always possible.
| adastra22 wrote:
| Why not? That's basically how NASA manages large projects.
| mulmen wrote:
| NASA has done a lot of amazing things but I wouldn't bet on
| them winning a Super Bowl.
| HarHarVeryFunny wrote:
| They'd have a 50% chance of winning one on Mars, since it
| would just be NASA vs China
| bangaladore wrote:
| Every year NASA has a 50% chance of winning the
| Superbowl- even on Earth!
|
| Either they win or don't. /s
| Julien_r2 wrote:
| I could imagine that even a small task at NASA might involve
| more knowledge and logic than the smallest task for a Hanoi's
| tower problem.
|
| Depends on what is considered as small enough for the LLM to
| be resolved with a high confidence.
| Uehreka wrote:
| One issue I often run into with this stuff is the tightly
| coupled nature of things in the real world. I'll fashion an
| example:
|
| Let's say you break a job down into 3 tasks: A, B and C.
| Doing one of those tasks is too much for an LLM to accomplish
| in one turn (this is something you learn intuitively through
| experience), but an LLM _could_ break each task into 3
| subtasks. So you do that, and start by having the LLM break
| task A into subtasks A1, A2 and A3. And B into B1, B2 and B3.
| But when you break down task C, the LLM (which needs to start
| with a fresh context each time since each "breakdown" uses
| 60-70% of the context) doesn't know the details of task A,
| and thus writes a prompt for C1 that is incompatible with
| "the world where A1 has been completed".
|
| This sort of "tunnel vision" is currently an issue with
| scaling 2025 agents. As useful context lengths get longer
| it'll get easier, but figuring out how to pack exactly the
| right info into a context is tough, especially when the tool
| you'd reach for to automate it (LLMs) are the same tool that
| suffers from these context limitations.
|
| None of this means big things aren't possible, just that the
| fussyness of these systems increases with the size of the
| task, and that fussyness leads to more requirements of "human
| review" in the process.
| etamponi wrote:
| "basically" is doing a lot of work in this sentence.
| pinkmuffinere wrote:
| Reasoning by analogy is great for intuition, but doesn't
| guarantee real results hold. Consider "voltage is like water
| pressure in pipes, so if there's a cut in my wire's
| insulation, the device won't get enough voltage" -- clearly
| this is not true, even though it relies on an analogy that's
| generally useful.
| alwa wrote:
| I really like that analogy, thank you for it. Also applies
| to "it's overvoltage, so I just need to poke a little hole
| in it to let the excess bleed out"...
| wat10000 wrote:
| That one can work, briefly, depending on how conductive
| your tool is.
| CamperBob2 wrote:
| Well, corona losses are a thing, after all.
| th0ma5 wrote:
| This is a really good analogy because the complex
| intersections between multiple groups independently working
| and trying to collaborate together into a collaborative
| hierarchy towards one large goal was one of the things that
| hid a lot of the problems that led to the Challenger
| disaster, according to Feynmen.
| naasking wrote:
| > All of it using natural language.
|
| Combining this with those approaches that recursively reason in
| latent space would be interesting.
| wordpad wrote:
| This works because a problem could be broken down to a prompt
| which rarely hallucinates.
|
| Most real world prompts can't be reduced to something so
| consistent and reliable.
|
| Their key finding was that the number of votes grows linearly
| with number of prompts you are trying to chain.
|
| However the issue is that the number of votes you need will
| grow exponentially with hallucination rate.
| patcon wrote:
| > into the simplest possible steps, recursively call
| (relatively small) LLMs as agents to execute one step at a
| time, and using a clever voting scheme to choose how to execute
| each step.
|
| It's like humans! Everything old is new again :)
| esafak wrote:
| It seems like this could be implemented by any harness.
| zer00eyz wrote:
| On the surface this is an interesting concept...
|
| The paper however, meh...
|
| No mention of MoE. One would think this is a logical evolution of
| that but not a mention (that I saw). Its own rubric for the task,
| Towers of Hanoi, was admittedly weak.
|
| LLM papers are starting to look like the last decade of JS
| frameworks and Tools. Only with less code and more academics, and
| thats disappointing, because I think a lack of pragmatism and
| grounding is now holding the field back...
| awei wrote:
| one issue I see is when steps in a plan depend on one another,
| when you cannot know all the next steps exactly before seeing the
| results of the previous ones, when you may have to backtrack
| sometimes
| htrp wrote:
| > The approach relies on an extreme decomposition of a task into
| subtasks, each of which can be tackled by focused microagents.
| The high level of modularity resulting from the decomposition
| allows error correction to be applied at each step through an
| efficient multi-agent voting scheme.
|
| Big if that the decomposition and the voting happen accurately
| for anything other than toy problems
| yorwba wrote:
| The approach in the paper specifically addresses the case where
| an LLM can usually solve a task when it requires few steps, but
| fails for the same kind of task with more steps because it
| randomly gets a step in the middle wrong and then derails. It
| can't do anything for tasks that the LLM can't solve even when
| there's just a few steps.
|
| In other words, it compensates for random error, not systematic
| error.
| andai wrote:
| I have ADHD and the same approach works for me. (In fact, most
| days it is essential!)
| binary132 wrote:
| do you have an algorithm for breaking down, organizing, and
| scheduling the small tasks, though? can it also be broken down?
| andai wrote:
| Worth opening the pdf just for the graph on page 1.
| mNovak wrote:
| Really seems like the reason logarithmic scales were invented..
| arscan wrote:
| In case you want to know what's going on in the left side of
| that chart, they gave a log scale in appendix a. I was thinking
| it was silly to not just use that version on the top, but I
| guess log scales make big differences 'feel' smaller.
| kevmo314 wrote:
| At that rate, they might as well have gone one step further
| and made the x axis exponential scale to make it feel _even
| bigger_.
| esafak wrote:
| A striking example of how not to present data. If the Cognizant
| AI team is here: please can you fix it in the next version of
| the paper?
| rdlw wrote:
| I think it's a brilliant example of how to use data to make a
| point.
|
| https://xkcd.com/1162/
| jmpeax wrote:
| Except on figure 1 they're all at 0, making it look like
| the authors didn't know how to use the models or
| deliberately made them do nothing.
| datadrivenangel wrote:
| http://www.vibechart.net
| mattpk wrote:
| Here is the pseudocode of MAKER: state =
| init_state() while state is not complete: state =
| LLM("You are a helpful assistant. The rules and format of the
| game is [...]. The correct strategy to use at each step is [...].
| The current state is [...]. Output the state after making the
| next move")
| vatsachak wrote:
| And you can decompose the proof of Fermat's last theorem into
| logical combinators.
|
| The meat is in decomposing the difficult problem into steps
| sublimefire wrote:
| The problem is how to even define a task using the English
| language and make sure there is enough entropy to infer the
| detailed intent. For it to be later split into zillions of small
| steps which can be executed over time by an LLM.
| lubujackson wrote:
| This has seemed to me to be the natural next step to turn LLMs
| into more deterministic tools. Pushing the frontier is nice, but
| I think LLMs have a whole different gear when they are able to
| self-decompose in a reliable way. Most of my success creating
| reusable LLM products came from determining where
| requirements/outputs need to be "hard" vs. "soft".
___________________________________________________________________
(page generated 2025-11-18 23:00 UTC)