[HN Gopher] When models manipulate manifolds: The geometry of a ...
___________________________________________________________________
When models manipulate manifolds: The geometry of a counting task
Author : vinhnx
Score : 88 points
Date : 2025-10-29 05:50 UTC (5 days ago)
(HTM) web link (transformer-circuits.pub)
(TXT) w3m dump (transformer-circuits.pub)
| Rygian wrote:
| > The task we study is linebreaking in fixed-width text.
|
| I wonder why they focused specifically on a task that is already
| solved algorithmically. The paper does not seem to address this,
| and the references do not include any mentions of non-LLM
| approaches to the line-breaking problem.
| omnicognate wrote:
| There's also a lot of analogising of this to visual/spatial
| reasoning, even to the point of talking about "visual
| illusions", when its clearly a counting task as the title says.
|
| It makes it tedious to figure out what they actually did (which
| sounds interesting) when it's couched in such terms and
| presented in such an LLMified style.
| dist-epoch wrote:
| it's not strictly a counting task, the LLM sees same-sized-
| tokens, but a token corresponds to a variable number of
| characters (which is not directly fed into the model)
|
| like the difference between Unicode code-points and UTF-8
| bytes, you can't just count UTF-8 bytes to know how many
| code-points you have
| omnicognate wrote:
| There's an aspect of figuring out what to count, but that
| doesn't make this task visual/spatial in any sense I can
| make out.
| Legend2440 wrote:
| They study it _because_ it already has a known solution.
|
| The point is to see how LLMs implement algorithms internally,
| starting with this simple easily understood algorithm.
| Rygian wrote:
| That makes sense; however it does not seem like they check
| the LLM outputs against the known solution. Maybe I missed
| that in the article.
| catgary wrote:
| I think this is an interesting direction, but I think that
| step 2 of this would be to formulate some conjectures about
| the geometry of other LLMs, or testable hypotheses about how
| information flows wrt character counting. Even checking some
| intermediate training weights of Haiku would be interesting,
| so they'd still be working off of the same architecture.
|
| The biology metaphor they make is interesting, because I
| think a biologist would be the first to tell you that you
| need more than one datapoint.
| lccerina wrote:
| Utter disrespect for using the term "biology" relating to LLM. No
| one would call the analysis of a mechanical engine "car biology".
| It's an artificial system, call it system analysis.
| lewtun wrote:
| The analogy stems from the notion that neural nets are "grown"
| rather than "engineered". Chris Olah has an old, but good post
| with some specific examples: https://colah.github.io/notes/bio-
| analogies/
| UltraSane wrote:
| It makes sense if you define "biology" as "incredibly
| complicated system not designed by humans that we kind of poke
| at to try to understand it."
| djoldman wrote:
| A superior LLM for line length optimization:
|
| https://www.youtube.com/watch?v=Y65FRxE7uMc
___________________________________________________________________
(page generated 2025-11-03 23:02 UTC)