[HN Gopher] Deep Learning Is Applied Topology
___________________________________________________________________
Deep Learning Is Applied Topology
Author : theahura
Score : 313 points
Date : 2025-05-20 13:54 UTC (9 hours ago)
(HTM) web link (theahura.substack.com)
(TXT) w3m dump (theahura.substack.com)
| esafak wrote:
| If it was topology we wouldn't bother to warp the manifold so we
| can do similarity search. No, it's _geometry_ , with a metric.
| Just as in real life, we want to be able to compare things.
|
| Topological transformation of the manifold happens during
| training too. That makes me wonder: how _does_ the topology
| evolve during training? I imagine it violently changing at first
| before stabilizing, followed by geometric refinement. Here are
| some relevant papers:
|
| * _Topology and geometry of data manifold in deep learning_
| (https://arxiv.org/abs/2204.08624)
|
| * _Topology of Deep Neural Networks_
| (https://jmlr.org/papers/v21/20-345.html)
|
| * _Persistent Topological Features in Large Language Models_
| (https://arxiv.org/abs/2410.11042)
|
| * _Deep learning as Ricci flow_
| (https://www.nature.com/articles/s41598-024-74045-9)
| profchemai wrote:
| Agree, if anything it's Applied Linear Algebra...but that
| sounds less exotic.
| lostmsu wrote:
| Well, we know it is non-linear. More like differential
| equations.
| theahura wrote:
| > Topological transformation of the manifold happens during
| training too. That makes me wonder: how does the topology
| evolve during training?
|
| If you've ever played with GANs or VAEs, you can actually
| answer this question! And the answer is more or less 'yes'. You
| can look at GANs at various checkpoints during training and see
| how different points in the high dimensional space move around
| (using tools like UMAP / TSNE).
|
| > I imagine it violently changing at first before stabilizing,
| followed by geometric refinement
|
| Also correct, though the violent changing at the beginning is
| also influenced the learning rate and the choice of optimizer.
| esafak wrote:
| And crucially, the initialization algorithm.
| umutisik wrote:
| Data doesn't actually live on a manifold. It's an approximation
| used for thinking about data. Near total majority, if not 100%,
| of the useful things done in deep learning have come from not
| thinking about topology in any way. Deep learning is not applied
| anything, it's an empirical field advanced mostly by trial and
| error and, sure, a few intuitions coming from theory (that was
| not topology).
| thuuuomas wrote:
| I cannot understand this prideful resentment of theory common
| among self-described practitioners.
|
| Even if existing theory is inadequate, would an operating
| theory not be beneficial?
|
| Or is the mystique combined with guess&check drudgery job
| security?
| hiddencost wrote:
| Maybe a little less with the ad hominems? The OP is providing
| an accurate description of an extremely immature field.
| cnity wrote:
| Many mathematicians are (rightly, IMO) allergic to
| assertions that certain branches are not useful (explicit
| in OP) and especially so if they are dismissive of attempts
| to understand complicated real world phenomema (implicit in
| OP, if you ask me).
| jebarker wrote:
| There are strong incentives to leave theory as technical debt
| and keep charging forward. I don't think it's resentment of
| theory, everyone would love a theory if one were available
| but very few are willing to forgoe the near term rewards to
| pursue theory. Also it's really hard.
| canjobear wrote:
| If there were theory that led to directly useful results
| (like, telling you the right hyperparameters to use for your
| data in a simple way, or giving you a new kind of
| regularization that you can drop in to dramatically improve
| learning) then deep learning practitioners would love it. As
| it currently stands, such theories don't really exist.
| fiddlerwoaroof wrote:
| Useful theories only come to exist because someone started
| by saying they must exist and then spent years or lifetimes
| discovering them.
| theahura wrote:
| This is way too rigorous. You can absolutely have theories
| that lead to useful results even if they aren't _as_
| predictive as you describe. Theory of evolution for an
| obvious counterpoint.
| lumost wrote:
| There are many reasons to believe a theory may not be
| forthcoming, or that if it is available may not be useful.
|
| For instance, we do not have consensus on what a theory
| should accomplish - should it provide convergence
| bounds/capability bounds? Should it predict optimal parameter
| counts/shapes? Should it allow more efficient calculation of
| optimal weights? Does it need to do these tasks in linear
| time?
|
| Even materials science in metals is still cycling through
| theoretical models after _thousands_ of years of making steel
| and other alloys.
| danielmarkbruce wrote:
| Who is proud? What you are seeing in some cases is eye
| rolling. And it's fair eye rolling.
|
| There is an enormous amount of theory used in the various
| parts of building models, there just isn't an overarching
| theory at the very most convenient level of abstraction.
|
| It almost has to be this way. If there was some neat theory,
| people would use it and build even more complex things on top
| of it in an experimental way and then so on.
| motoboi wrote:
| Your comment sits in the nice gradient between not seeing at
| all the obvious relationships between deep learning and
| topology and thinking that deep learning is applied topology.
|
| See? Everything lives in the manifold.
|
| Now for a great visualization about the Manifold Hypothesis I
| cannot recommend more this video:
| https://www.youtube.com/watch?v=pdNYw6qwuNc
|
| That helps to visualize how the activation functions, bias and
| weights (linear transformations) serve to stretch the high
| dimensional space so that data go into extremes and become easy
| to put in a high dimension, low dimensional object (the
| manifold) where is trivial to classify or separate.
|
| Gaining an intuition about this process will make some deep
| learning practices so much easy to understand.
| behnamoh wrote:
| > a few intuitions coming from theory (that was not topology).
|
| I think these 'intuitions' are an after-the-fact thing, meaning
| AFTER deep learning comes up with a method, researchers in
| other fields of science notice the similarities between the
| deep learning approach and their (possibly decades old)
| methods. Here's an example where the author discovers that GPT
| is really the same computational problems he has solved in
| physics before:
|
| https://ondrejcertik.com/blog/2023/03/fastgpt-faster-than-py...
| ogogmad wrote:
| I beg to differ. It's complete hyperbole to suggest that the
| article said "it's the same problem as something in physics",
| given this statement: It seems that the
| bottleneck algorithm in GPT-2 inference is matrix-matrix
| multiplication. For physicists like us, matrix-matrix
| multiplication is very familiar, *unlike other aspects of AI
| and ML* [emphasis mine]. Finding this familiar ground
| inspired us to approach GPT-2 like any other numerical
| computing problem.
|
| Note: Matrix-matrix multiplication is basic mathematics, and
| not remotely interesting as physics. It's not physically
| interesting.
| bee_rider wrote:
| Agreed.
|
| Although, to try to see it from the author's perspective,
| it is pulling tools out of the same (extremely well
| developed and studied in it's own right) toolbox as
| computational physics does. It is a little funny although
| not too surprising that a computational physics guy would
| look at some linear algebra code and immediately see the
| similarity.
|
| Edit: actually, thinking a little more, it is basically
| absurd to believe that somebody has had a career in
| computational physics without knowing they are relying
| heavily on the HPC/scientific computing/numerical linear
| algebra toolbox. So, I think they are just using that to
| help with the narrative for the blog post.
| constantcrying wrote:
| You are exactly right, after deep learning researchers had
| invented Adam for SGD, numerical analysts finally discovered
| Gradient descent. And after the first neural net was
| discovered, finally the matrix was invented in the novel
| field of linear algebra.
| sota_pop wrote:
| I disagree with this wholeheartedly. Sure, there is lots of
| trial and error, but it's more an amalgamation of theory from
| many areas of mathematics including but not limited to:
| topology, geometry, game theory, calculus, and statistics. The
| very foundations (i.e. back-propagation) is just the chain rule
| applied to the weights. The difference is that deep learning
| has become such an accessible (sic profitable) field that many
| practitioners have the luxury of learning the subject without
| having to learn the origins of the formalisms. Ultimately
| allowing them to utilize or "reinvent" theories and techniques
| often without knowing they have been around in other fields for
| much longer.
| kwertzzz wrote:
| Can you give an example where theories and techniques from
| other fields are reinvented? I would be genuinely interested
| for concrete examples. Such "reinventions" happen quite often
| in science, so to some degree this would be expected.
| srean wrote:
| Bethe ansatz is one. It took a toure de force by Yedidia to
| recognize that loopy belief propagation is computing the
| stationary point of Bethe's approximation to Free Energy.
|
| Many statistical thermodynamics ideas were reinvented in
| ML.
|
| Same is true for mirror descent. It was independently
| discovered by Warmuth and his students as Bregman
| divergence proximal minimization, or as a special case
| would have it, exponential gradient algorithms.
|
| One can keep going.
| ogogmad wrote:
| The connections of deep learning to stat-mech and
| thermodynamics are really cool.
|
| It's led me to wonder about the origin of the probability
| distributions in stat-mech. Physical randomness is mostly
| a fiction (outside maybe quantum mechanics) so
| probability theory must be a convenient fiction. But
| objectively speaking, where then do the probabilities in
| stat-mech come from? So far, I've noticed that the
| (generalised) Boltzmann distribution serves as the bridge
| between probability theory and thermodynamics: It lets us
| take non-probabilistic physics and invent probabilities
| in a useful way.
| srean wrote:
| In Boltzmann's formulation of stat-mech it comes from the
| assumption that when a system is in "equilibrium", then
| all the micro-states that are consistent with the macro-
| state are equally occupied. That's the basis of the
| theory. A prime mover is thermal agitation.
|
| It can be circular if one defines equilibrium to be that
| situation when all the micro-states are equally occupied.
| One way out is to define equilibrium in temporal terms -
| when the macro-states are not changing with time.
| mitthrowaway2 wrote:
| The Bayesian reframing of that would be that when all you
| have measured is the macrostate, and you have no further
| information by which to assign a higher probability to
| any compatible microstate than any other, you follow the
| principle of indifference and assign a uniform
| distribution.
| srean wrote:
| Yes indeed, thanks for pointing this out. There are
| strong relationships between max-ent and Bayesian
| formulations.
|
| For example one can use a non-uniform prior over the
| micro-states. If that prior happens to be in the Darmois-
| Koopman family that implicitly means that there are some
| non explicitly stated constraints that bind the micro-
| state statistics.
| nickpsecurity wrote:
| One might add 8-16-bit training and quantization. Also,
| computing semi-unreliable values with error correction.
| Such tricks have been used in embedded, software
| development on MCU's for some time.
| whatever1 wrote:
| I mean the entire domain of systems control is being
| reinvented by deep RL. System identification, stability,
| robustness etc
| srean wrote:
| Good one. Slightly different focus but they really are
| the same topic. Historically, Control Theory has focused
| on stability and smooth dynamics while RL has
| traditionally focused on convergence of learning
| algorithms in discrete spaces.
| saberience wrote:
| None of the major aspects of deep learning came from
| manifolds though.
|
| It is primarily linear algebra, calculus, probability theory
| and statistics, secondarily you could add something like
| information theory for ideas like entropy, loss functions
| etc.
|
| But really, if "manifolds" had never been
| invented/conceptualized, we would still have deep learning
| now, it really made zero impact on the actual practical
| technology we are all using every day now.
| qbit42 wrote:
| Loss landscapes can be viewed as manifolds. Adagrad/ADAM
| adjust SGD to better fit the local geometry and are widely
| used in practice.
| niemandhier wrote:
| It's alchemy.
|
| Deep learning in its current form relates to a hypothetical
| underlying theory as alchemy does to chemistry.
|
| In a few hundred years the Inuktitut speaking high schoolers of
| the civilisation that comes after us will learn that this
| strange word "deep learning" is a left over from the lingua
| franca of yore.
| adamnemecek wrote:
| Not really, most of the current approaches are some
| approximations of the partition function.
| fmap wrote:
| The reason deep learning is alchemy is that none of these
| deep theories have predictive ability.
|
| Essentially all practical models are discovered by trial
| and error and then "explained" after the fact. In many
| papers you read a few paragraphs of derivation followed by
| a simpler formulation that "works better in practice".
| E.g., diffusion models: here's how to invert the forward
| diffusion process, but actually we don't use this, because
| gradient descent on the inverse log likelihood works
| better. For bonus points the paper might come up with an
| impressive name for the simple thing.
|
| In most other fields you would not get away with this. Your
| reviewers would point this out and you'd have to
| reformulate the paper as an experience report, perhaps with
| a section about "preliminary progress towards theoretical
| understanding". If your theory doesn't match what you do in
| practice - and indeed many random approaches will kind of
| work (!) - then it's not a good theory.
| theahura wrote:
| It's true that there is no directly predictive model of
| deep learning, and it's also true that there is some
| trial and error, but it is wrong to say that therefore
| there is no operating theory at all. I recommend reading
| Ilyas 30 papers (here's my review of that set:
| https://open.substack.com/pub/theahura/p/ilyas-30-papers-
| to-...) to see how shared intuitions and common threads
| are clearly developed over the last decade+
| Koshkin wrote:
| > _Data doesn 't actually live on a manifold._
|
| Often, they do (and then they are called "sheaves").
| wenc wrote:
| Many types of data don't. Disconnected spaces like integer
| spaces don't sit on a manifold (they are lattices). Spiky
| noisy fragmented data don't sit on a (smooth) manifold.
|
| In fact not all ML models treat data as manifolds. Nearest
| neighbors, decision trees don't require the manifold
| assumption and actually work better without it.
| qbit42 wrote:
| Any reasonable statistical explanation of deep learning
| requires there to be some sort of low dimensional latent
| structure in the data. Otherwise, we would not have enough
| training data to learn good models, given how high the
| ambient dimensions are for most problems.
| wenc wrote:
| Deep learning specifically yes. It needs a manifold
| assumption. But not data in general which was what I was
| responding to.
| theahura wrote:
| It turns out a lot of disconnected spaces can be
| approximated by smooth ones that have really sharp
| boundaries, which more or less seems to be how neural
| networks will approximate something like discrete tokens
| wenc wrote:
| Can be approximated yes. Approximated well? No, but you
| can get away with it sometimes with saturation functions
| like softmax. But badly. It's like trying to solve an
| integer program as a linear program. You end up with a
| relaxation that is not integral and not the real answer.
|
| An integer lattice can only be a manifold in a trivial
| sense (dimension 0). But not for any positive dimensions.
| theahura wrote:
| I say this as someone who has been in deep learning for over a
| decade now: this is pretty wrong, both on the merits (data
| obviously lives on a manifold) and on its applications to deep
| learning (cf chris olah's blog as an example from 2014, which
| is linked in my post --
| https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/).
| Embedding spaces are called 'spaces' for a reason. GANs, VAEs,
| contrastive losses -- all of these are about constructing
| vector manifolds that you can 'walk' to produce different kinds
| of data.
| almostgotcaught wrote:
| You're citing a guy that never went to college (has no math
| or physics degree), has never published a paper, etc. I guess
| that actually tracks pretty well with how strong the whole
| "it's deep theory" claim is.
| smsx wrote:
| Chris Olah has never published a paper? ... https://scholar
| .google.com/citations?user=6dskOSUAAAAJ&hl=en...
| theahura wrote:
| Chris Olah? One of the founders of Anthropic and the head
| of their interpretability team?
| umutisik wrote:
| If data did live on a manifold contained, e.g. images in
| R^{n^2}, then it wouldn't have thickness or branching, which
| it does. It's an imperfect approximation to help think about
| it. The use of mathematical language is not the same as an
| application of mathematics (and the use of the word 'space'
| there is not about topology).
| esafak wrote:
| It does if you relax your definition to accommodate
| approximation error, cf. e.g., _Intrinsic Dimensionality
| Explains the Effectiveness of Language Model Fine-Tuning_
| (https://aclanthology.org/2021.acl-long.568.pdf)
| csimon80 wrote:
| "All models are wrong, but some are useful" -George Box
| woopwoop wrote:
| I don't agree with your first sentence, but I agree with the
| rest of this post.
| Regic wrote:
| I feel like the fact that ML has no good explanation why it
| works this well gives a lot of people room to invent their
| head-canon, usually from their field of expertise. I've seen
| this from exceptionally intelligent individuals too. If you
| only have a hammer...
| nomel wrote:
| I think it would be more unusual, and concerning, if an
| intelligent individual didn't attempt to apply their
| expertise for a head-canon of something unknown.
|
| Coming up with an idea for how something works, by applying
| your expertise, is the fundamental _foundation_ of
| intelligence, learning, and was behind every single
| advancement of human understanding.
|
| People thinking is always a good thing. Thinking about the
| unknown is better. Thinking with others is best, and sharing
| those thoughts isn't somehow bad, even if they're not
| complete.
| HarHarVeryFunny wrote:
| When you say ML, I assume you really mean LLMs?
|
| Even with LLMs, there's no real mystery about _why_ they work
| so well - they produce human-like input continuations (aka
| "answers") because they are trained to predict continuations
| of human-generated training data. Maybe we should be a bit
| surprised that the continuation signal is there in the first
| place, but given that it evidentially is, it's no mystery
| that LLMs are able to use it - just testimony to the power of
| the Transformer as a predictive architecture, and of course
| to gradient descent as a cold unthinking way of finding an
| error minimum.
|
| Perhaps you meant _how_ LLMs work, rather than _why_ they
| work, but I 'm not sure there's any real mystery there either
| - the transformer itself is all about key-based attention,
| and we now know that training a transformer seems to
| consistently cause it to leverage attention to learn
| "induction heads" (using pairs of adjacent attention heads)
| that are the main data finding/copying primitive they use to
| operate.
|
| Of course knowing how an LLM works in broad strokes isn't the
| same as knowing specifically how it is working in any given
| case, how is it transforming a specific input layer by layer
| to create the given output, but that seems a bit like saying
| that because I can't describe - precisely - why you had
| pancakes for breakfast, that we don't know how the brains
| works.
| baxtr wrote:
| Just a side comment to your observation: the principle is
| called reductionism and has been tried on many fields.
|
| Physics is just applied mathematics
|
| Chemistry is just applied physics
|
| Biology is just applied chemistry
|
| It doesn't work very well.
| yubblegum wrote:
| > Near total majority, if not 100%, of the useful things done
| in deep learning have come from not thinking about topology in
| any way.
|
| Of course. Now, to actually deeply understand what is happening
| with these constructs, we will use topology. Topoligical
| insights will without doubt then inform the next generations of
| this technology.
| solomatov wrote:
| May I ask you to give examples of insights from topology
| which improved existing models, or at least improved our
| understanding of them? arxiv papers are preferred.
| constantcrying wrote:
| >it's an empirical field advanced mostly by trial and error
| and, sure, a few intuitions coming from theory (that was not
| topology).
|
| Neural Networks consist almost exclusively of two parts,
| numerical linear algebra and numerical optimization.
|
| Even if you reject the abstract topological description.
| Numerical linear algebra and optimization couldn't be any more
| directly applicable.
| sota_pop wrote:
| I've always enjoyed this framing of the subject, the idea of
| mapping anything as hyperplanes existing in a solution space is
| one of the ideas that really blew my hair back during my academic
| studies. I would nitpick at your "dots in a circle example - with
| the stoner reference joke" I could be mistaken, but common
| practice isn't to "move to a higher dimension", but use a kernel
| (i.e. parameterize the points into the polar |r,theta> basis).
| All things considered, nice article.
| theahura wrote:
| I'm pulling directly from Chris Olah's blog post with that
| example. But I will say that in practice, its always surprising
| how increasing the dimensionality of a neural network magically
| solves all sorts of problems. You could use a kernel if you
| don't have more computation available, but given more
| computation adding a dimension is strictly more flexible (and
| is capable of separating a much wider range of datasets)
| benl wrote:
| Your explanation of finding a surface to separate good
| reasoning traces from bad reasoning traces in a high
| dimensional space worked as a great framing of the problem.
| It seems though that the surface will be fractal - the
| distance between a good trace and a bad trace could be
| arbitrarily small. If so then the work required to find and
| compute better and better surfaces will grow arbitrarily
| large. I wonder if there is a rigorous way to determine if
| the surface is fractal or not.
| polotics wrote:
| Ok, how do transformers fit into this understanding of deep
| learning?
| motoboi wrote:
| Transformers don't feel differentiable (because of the
| attention mechanism), but they actually are (as being back-
| propagation based forces it to be).
|
| The attention mechanism is not a stretching of the manifold,
| but is trained to be able to measure distances in the manifold
| surface, which is stretched and deformed (or transformed?) in
| the feed-forward layers.
| theahura wrote:
| Transformers learn embedding representations of tokens, which
| are easily mapped into a space. Similar tokens are mapped to
| similar places on the space. The fully connected layer at the
| end of each transformer block defines a transformation of a set
| of points in a space to another point in that space, not unlike
| the example of adding colors together to get a new color
| jebarker wrote:
| Transformers (with self-attention being the key operation) are
| kernel smoothers which fits easily into this view of the world.
| See here: http://bactra.org/notebooks/nn-attention-and-
| transformers.ht...
| parpfish wrote:
| This is also how I've often thought about deep learning --
| focusing on the geometry of the _data_ at each layer rather than
| the weights and biases is far more revealing.
|
| I've always been hopeful that some algebraic topology master
| would dig into this question and it'd provide some better design
| principles for neural nets. which activation functions? how much
| to fan in/out? how many layers?
| profchemai wrote:
| Once I read "This has been enough to get us to AGI.", credibility
| took a nose dive.
|
| In general it's a nice idea, but the blogpost is very fluffy,
| especially once it connects it to reasoning, there is serious
| technical work in this area (i.g.
| https://arxiv.org/abs/1402.1869) that has expanded this idea and
| made it more concrete.
| mirekrusin wrote:
| Just because manifold looks a bit like burrito if you squint
| doesn't mean it is a burrito.
| ComplexSystems wrote:
| What if you don't have to squint very much?
| mirekrusin wrote:
| Same amount of squinting needed as for claim that deep
| learning is just a bunch of matrices. Or 0s and 1s. Cool. Now
| what?
| fedeb95 wrote:
| Interesting read. This seems hard to prove:
|
| "Everything lives on a manifold"
| ComplexSystems wrote:
| I really liked this article, though I don't know why the author
| is calling the idea of finding a separating surface between two
| classes of points "topology." For instance, they write
|
| "If you are trying to learn a translation task -- say, English to
| Spanish, or Images to Text -- your model will learn a topology
| where bread is close to pan, or where that picture of a cat is
| close to the word cat."
|
| This is everything that topology is _not_ about: a notion of
| points being "close" or "far." If we have some topological space
| in which two points are "close," we can stretch the space so as
| to get _the same topological space_ , but with the two points now
| "far". That's the whole point of the joke that the coffee cup and
| the donut are the same thing.
|
| Instead, the entire thing seems to be a real-world application of
| something like algebraic geometry. We want to look for something
| like an algebraic variety the points are near. It's all about
| geometry and all about metrics between points. That's what it
| seems like to me, anyway.
| srean wrote:
| > This is everything that topology is not about
|
| 100 percent true.
|
| I can only hope that in an article that is about two things, i)
| topology and ii) deep learning, the evident confusions are
| contained within one of them -- topology, only.
| theahura wrote:
| fair, I was using 'topology' more colloquially in that
| sentence. Should have said 'surface'.
| srean wrote:
| Ah! That clears it up.
|
| You then mean Deep Learning has a lot in common with
| differential geometry and manifolds in general. That I will
| definitely agree with. DG and manifolds have far richer and
| informative structure than topology.
| Koshkin wrote:
| Speaking of: https://www2.math.upenn.edu/~ghrist/notes.html
| soulofmischief wrote:
| Thanks for sharing. I also tend to view learning in terms of
| manifolds. It's a powerful representation.
|
| > I'm personally pretty convinced that, in a high enough
| dimensional space, this is indistinguishable from reasoning
|
| I actually have journaled extensively about this and even written
| some on Hacker News about it with respect to what I've been
| calling probabilistic reasoning manifolds:
|
| > This manifold is constructed via learning a decontextualized
| pattern space on a given set of inputs. Given the inherent
| probabilistic nature of sampling, true reasoning is expressed in
| terms of probabilities, not axioms. It may be possible to
| discover axioms by locating fixed points or attractors on the
| manifold, but ultimately you're looking at a probabilistic
| manifold constructed from your input set.
|
| > But I don't think you can untie this "reasoning" from your
| input data. It's possible you will find "meta-reasoning", or
| similar structures found in any sufficiently advanced reasoning
| manifold, but these highly decontextualized structures might be
| entirely useless without proper recontextualization,
| necessitating that a reasoning manifold is trained on input whose
| patterns follow learnable underlying rules, if the manifold is to
| be useful for processing input of that kind.
|
| > Decontextualization is learning, decomposing aspects of an
| input into context-agnostic relationships. But
| recontextualization is the other half of that, knowing how to
| take highly abstract, sometimes inexpressible, context-agnostic
| relationships and transform them into useful analysis in novel
| domains
|
| Full comment: https://news.ycombinator.com/item?id=42871894
| mjburgess wrote:
| Are you talking about reasoning in general, reasoning qua that
| mental process which operates on (representations of)
| propositions?
|
| In which case, I cannot understand " true reasoning is
| expressed in terms of probabilities, not axioms "
|
| One of the features of reasoning is that it does not operate in
| this way. It's highly implausible animals would have been
| endowed with no ability to operate non-probabilistically on
| propositions represented by them, since this is essential for
| correct reasoning -- and a relatively trivial capability to
| provide.
|
| Eg., "if the spider is in boxA, then it is not everywhere else"
| and so on
| jvanderbot wrote:
| I suspect, as a layperson who watches people make decisions
| all the time, that somewhere in our mind is a "certainty
| checker".
|
| We don't do logic itself, we just create logic from certainty
| as part of verbal reasoning. It's our messy internal
| inference of likelihoods that causes us to pause and think,
| or dash forward with confidence, and convincing others is the
| only place we need things like "theorems".
|
| This is the only way I can square things like intuition,
| writing to formalize thoughts, verbal argument, etc, with the
| fact that people are just so _mushy_ all the time.
| mjburgess wrote:
| People are only mushy in their verbalised reasoning,
| because its the nature of such reasoning to handle hard
| cases. Animal cognition, at its basic levels, is incredibly
| refined and makes necessary use of logic, flawlessly,
| frequently.
|
| This naive cynicism about our mental capacities is a
| product of this credulity about statistical AI. If one
| beings with an earnest study of animal intelligence, in
| order to describe it, it disappears. It's exactly and only
| a project of the child playing with his lego, certain that
| great engineering projects have little use for any more
| than stacking bricks.
| jvanderbot wrote:
| Well, we disagree fundamentally. And, I applaud the heavy
| handed use of condescension.
|
| Logical propositions ("2+2=4 regardless of my certainty
| about it") seem a long way from necessary _or_ sufficient
| to survival for animals. A fuzzy heatmap of "where is
| prey going" or "How many prey over there" is much closer
| to necessary and sufficient. The fact that measurements
| or senses can update those estimates is a long way from a
| logical deduction.
|
| Something more like probability factor graph can do it,
| without the pernicious use of "concepts" or too much need
| for implication, which is sticky and overly rigorous.
|
| That's all I have to say, and I doubt we'll find middle
| ground.
| mjburgess wrote:
| You can enumerate, all you wish, all the fuzzy judgements
| we need to make. This confirms a capacity for uncertain
| reasoning. It says nothing about the trivial and
| innumerable ways concepts compose both in content
| (imagine that A and-also B) and in logical relation (eg.,
| imagine that not A and B).
|
| The point of my "condescension" was to point out that
| people of your position are arguing from ignorance, with
| confirmation bias -- ie., having no study of animal
| intelligence, and only ever repeating what they know
| about their own study of irrelevant systems.
|
| Your reply evidences this exactly. Zero engagement with
| any facts on the ground about actual animal intelligence.
| Are you really actually trying to account for animal
| intelligence, or as I have claimed twice now, are you
| only really wishing to maintain your ignorance of it,
| dismiss any analysis of it, and instead "confirm" that
| whatever you are aware of "must, presumably, apply".
|
| Imagine being faced with such bad faith over and over and
| over again. It is like arguing with people who insist the
| world is flat, and when challenged, point to euclidean
| geometry and the flatness of the pavement under their
| feet. If I begin by anticipating such behaviour, you can
| see why.
| soulofmischief wrote:
| Propositions are just predictions, they all come with some
| level of uncertainty even if we ignore that uncertainty for
| practical purposes.
|
| Any validation of a theory is inherently statistical, as you
| must sample your environment with some level of precision
| across spacetime, and that level of precision correlates to
| the known accuracy of hypotheses. In other words, we can
| create axiomatic systems of logic, but ultimately any attempt
| to compare them to reality involves empirical sampling.
|
| Unlike classical physics, our current understanding of
| quantum physics essentially allows for anything to be
| "possible" at large enough spacetime scales, even if it is
| never actually "probable". For example, quantum tunneling,
| where a quantum system might suddenly overcome an energy
| barrier despite lacking the required energy.
|
| Every day when I walk outside my door and step onto the
| ground, I am operating on a belief that gravity will work the
| same way every time, that I won't suddenly pass through the
| Earth's crust or float into the sky. We often take such
| things for granted, as axiomatic, but ultimately all of our
| reasoning is based on statistical correlations. There _is_
| the ever-minute possibility that gravity suddenly stops
| working as expected.
|
| > if the spider is in boxA, then it is not everywhere else
|
| We can't even physically prove that. There's always some
| level of uncertainty which introduces probability into your
| reasoning. It's just convenient for us to say, "it's
| exceedingly unlikely in the entire age of the universe that a
| macroscopic spider will tunnel from Box A to Box B", and
| apply non-probabilistic heuristics.
|
| It doesn't remove the probability, we just don't bother to
| consider it when making decisions because the energy required
| for accounting for such improbabilities outweighs the energy
| saved by not accounting for them.
|
| As mentioned in my comment, there's also the possibility that
| universal axioms may be recoverable as fixed points in a
| reasoning manifold, or in some other transformation. If you
| view these probabilities as attractors on some surface, fixed
| points may represent "axioms" that are true or false under
| any contextual transformation.
| mjburgess wrote:
| This response doesn't fill me with confidence. You aren't
| really engaging with any of the actual issues your position
| entails.
|
| A proposition is not a prediction. A prediction is either
| an estimate of the value of some quantity ("the dumb ML
| meaning of prediction") or a proposition which describes a
| future scenario. We can trivially enumerate propositions
| that do not describe future scenarios, eg., 2 + 2 = 4.
|
| Uncertainty is a property of belief attitudes towards
| propositions, it isn't a feature of their semantic content.
| A person doesnt mean anything different by "2 + 2 = 4" if
| they are 80 or 90% sure of it.
|
| > We can't even physically prove that.
|
| Irrelevant. Our minds are not constrained by physical
| possibility, necessarily so, as we know very little about
| what is physically possible. I can imagine abitary number
| of cases, arising out of logical manipulation of
| propositons, that are not physically possible. (Eg.,
| "Superman can lift any building. The empire state building
| is so-and-so a kind of building. Imagine(Superman lifting
| the empire state building)").
|
| The infinite variety of our imagination is a trivial
| consequence of non-probabilistic operations on
| propositions, it's incomprehensibly implausible as a
| consequence of merely probabilistic ones.
|
| That nature seems to have endowed minds with discrete
| operations, that these are empirical in operation across
| very wide classes of reasoning, including imagination, that
| these seem trivial for nature to provide (etc.) render the
| notion that they don't exist highly highly implausible.
|
| There is nothing lacking explanation here. The relevant
| mental processes we have to hand are fairly obvious and
| fairly easy to explain.
|
| Its an obvious act of credulity to try and find some way to
| make the latest trinkets of the recent rich some sort of
| miracle. All of these projects of "incredible abstraction"
| follow around these hype cycles, turning lead into gold: if
| x "is really" y, and y "is really" z, and ..., then x is
| amazin! This piles towers of every more general hollowed-
| out words on top of each other until the most trivial thing
| sounds like a wonder.
| naasking wrote:
| > It's highly implausible animals would have been endowed
| with no ability to operate non-probabilistically on
| propositions represented by them, since this is essential for
| correct reasoning
|
| Why would animals need to evolve 100% correct reasoning if
| probabilistically correct reasoning suffices? If
| probabilistic reasoning is cheaper in terms of energy then
| correct reasoning is a disadvantage.
| mjburgess wrote:
| It doesnt suffice. It's also vastly energetically cheaper
| just to have (algorithmic) negation. Compressing (A, not A)
| into a probability function is extremely incomprehensibly
| expensive.
| naasking wrote:
| > It's also vastly energetically cheaper just to have
| (algorithmic) negation.
|
| Even if true, that's an argument that it's cheaper to
| _have_ something, not that it 's cheaper to _develop it_
| through natural selection. Training time and energy for
| LLMs shows how energy intensive training to get to the
| point of grokking /circuit generalization.
| mjburgess wrote:
| It is a matter of empirical fact that we can reason with
| logical relationships. Thus taking an LLM and it's
| training as a model of conginition is empriically false.
|
| It should be obviously doubly so, since as a model -- as
| you point out -- it makes trivial aspects of our
| cognition impossibly expensive to acqurie.
| maxiepoo wrote:
| Isn't it more differential geometry?
| srean wrote:
| The title, as it stands, is trite and wrong. More about that a
| little later. The article on the other hand is a pleasant read.
|
| Topology is whatever little structure that remains in geometry
| after you throwaway distances, angles, orientations and all sorts
| of non tearing stretchings. It's that bare minimum that still
| remains valid after such violent deformations.
|
| While notion of topology is definitely useful in machine
| learning, -- scale, distance, angles etc., all usually provide
| lots of essential information about the data.
|
| If you want to distinguish between a tabby cat and a tiger it
| would be an act of stupidity to ignore scale.
|
| Topology is useful especially when you cannot trust lengths,
| distances angles and arbitrary deformations. That happens, but to
| claim deep learning is applied topology is absurd, almost stupid.
| theahura wrote:
| > Topology is useful especially when you cannot trust lengths,
| distances angles and arbitrary deformations
|
| But...you can't. The input data lives on a manifold that you
| cannot 'trust'. It doesn't mean anything apriori that an image
| of a coca-cola can and an image of a stopsign live close to
| each other in pixel space. The neural network applies all of
| those violent transformations you are talking about
| srean wrote:
| > But...you can't.
|
| Only in a desperate sales pitch or a desparate research
| grants. There are of course some situations were certain
| measurements are untrustworthy, but to claim that is the
| common case is very snake oily.
|
| When certain measurements become untrustworthy, that it does
| so only because of some unknown smooth transformation, is not
| very frequent (this is what purely topological methods will
| deal with). Random noise will also do that for you.
|
| Not disputing the fact that sometimes metrics cannot be
| trusted entirely, but to go to a _topological_ approach seems
| extreme. One should use as much of the relevant non-
| topological information as possible.
|
| As the hackneyed example goes a topological methods would not
| be able to distinguish between a cup and a donut. For that
| you would need to trust non-topological features such as
| distances and angles. Deep learning methods can indeed
| differentiate between cop-nip and coffee mugs.
|
| BTW I am completely on-board with the idea that data often
| looks as if it has been sampled from an unknown, potentially
| smooth, possibly non-Euclidean manifold and then corrupted by
| noise. In such cases recovering that manifold from noisy data
| is a very worthy cause.
|
| In fact that is what most of your blogpost is about. But
| that's differential geometry and manifolds, they have
| structure far richer than a topology. For example they may
| have tangent planes, a Reimann metric or a symplectic form
| etc. A topological method would throw all of that away and
| focus on topology.
| kentuckyrobby wrote:
| I don't think that was their point, I think their point was
| that neural networks 'create' their optimization space by
| using lengths, distances, and angles. You can't reframe it
| from a topological standpoint, otherwise optimization spaces
| of some similar neural networks on similar problems would
| topologically comparable, which is not true.
| theahura wrote:
| Well, sorta. There is some evidence to suggest that neural
| networks learn 'universal' features (cf Anthropic's
| circuits thread). But I'll openly admit to being out of my
| depth here, and maybe I just don't understand OPs point
| throwawaymaths wrote:
| once you get into the nitty gritty, a lot of things that
| wouldn't matter if it were pure topology, do, like number of
| layers all the way to quantization/fp resolution
| quantadev wrote:
| The word "topology" has a legitimate dictionary definition,
| that has none of the requirements that you're asserting. I
| think what you're missing is that it has two definitions.
| srean wrote:
| In blog posts about specialised and technical topics it is
| expected that in-domain technical keywords that have long
| established definitions and meanings be used in the same
| technical sense. Otherwise it can become quite confusing.
| Gravity means gravity when we are talking Newtonian
| mechanics. Similarly, in math and ML _' topology'_ has a
| specific meaning.
| quantadev wrote:
| The word "topology" is quite commonly used in all kinds of
| books, papers, and technical materials any time they're
| discussing geometric characteristics of surfaces. The term
| is probably used 1000000 times more commonly in this more
| generic way than it's ever used in the strict pedantic way
| you're asserting that it must.
| srean wrote:
| Surfaces certainly have a topology (potentially more than
| one), surfaces are examples of one kind of a topological
| space, in fact the next interesting one after a curve. So
| I will not be surprised at all with co-occurrences of
| 'surface' and 'topology'. But surfaces and topologies
| mean different things.
|
| Dogs have fur. Dogs are an example of a furry animal. But
| dogs and furs are not the same thing although they may
| appear in the same text often.
|
| Topology is a traditional as well as an active branch of
| applied and pure mathematics, well, Physics too.
|
| It has tons of text books printed on it, has several
| active journals and conferences dealing with it. https://
| www.amazon.com/s?k=Topology&sprefix=topology+%2Caps%...
|
| Surprise, surprise ...not ...has an extensive Wikipedia
| page.
|
| https://en.m.wikipedia.org/wiki/Topology
|
| Math magazines for high schoolers have articles on it.
| Colleges offer multiple courses on it. Some of those
| courses would be mandatory for a degree in even undergrad
| mathematics.
|
| If one wants to do graduate studies then one can do a
| Masters or a PhD in Topology, well in one of it's many
| branches.
|
| It's also not a new kid on the block. It goes back to ...
| analysis situs ... further back to Leibniz, although it
| began to crystalize formally after Poincare.
|
| If someone wants to use the phrase _' differential
| calculus'_ to mean something else in their love letters
| and sweet nothings, that's absolutely fine :) but in
| Maths (and Machine Learning, well, with quality of peer
| reviewing this might soon be iffy) it has a well
| established and unambiguous meaning.
|
| Note because of its shared beginning at the feet of
| Leibniz, comparing it with calculus is not an unfair
| comparison.
| quantadev wrote:
| The most common uses of "topology", whenever used to
| convey a geometry-related idea, is in the more general
| sense meaning "surfaces". Only one out of a million times
| is anyone ever referring to the specific mathematical
| field of the same name to which you refer.
| srean wrote:
| LOL the very first line of your own link
|
| (that now you seem to have deleted after my comment
| https://en.m.wikipedia.org/wiki/Topology_(disambiguation)
| )
|
| _" Topology is a branch of mathematics concerned with
| geometric properties preserved under continuous
| deformation (stretching without tearing or gluing)"_
|
| That is indeed the established meaning of topology, more
| so in mathematics and the blog post was on _applied
| mathematics_. That it may mean something else in other
| contexts is irrelevant.
|
| I rest my case.
|
| > The most common uses of "topology", whenever used to
| convey a geometry-related idea, is in the more general
| sense meaning "surfaces"
|
| _Erm, citation please._
|
| I included a search on Amazon on topology https://www.ama
| zon.com/s?k=Topology&sprefix=topology+%2Caps%... (without
| even adding the keyword maths. None of the results seem
| to be about _surfaces_. Shouldn 't there have been a few
| ? Wouldn't Amazon search results reflect the general
| sense meaning ?).
|
| If it were true, wouldn't the Wikipedia pages have talked
| about that general sense meaning first ?
|
| Alternatively, I would say, take a breath. Is this hill
| really the one worth dying on ? There are better ones.
| Have a good day and if work permits, get yourself a juicy
| topology book, it can be interesting, if presented well.
| quantadev wrote:
| The question was never "Is topology a field of
| mathematics". The question was, is that term most often
| used to refer to surfaces in general, and the answer to
| that is still 'yes'.
| srean wrote:
| https://www.amazon.com/s?k=Topology&sprefix=topology+%2Ca
| ps%...
|
| Could you drop Amazon a message. They should really
| change the search results if that's what topology means
| in general. I am sure they would be delighted to receive
| the bug report.
|
| For laughs, I asked ChatGPT to use 'topology' in place of
| 'surface'. Here's what it wrote: The
| topology of the lake was so calm it reflected the
| mountains perfectly. She wiped the kitchen
| topology clean after cooking dinner. After
| years of silence, the truth began to topology.
| The spacecraft landed safely on the topology of Mars.
| A thin layer of dust had settled on the topology of the
| old table. He barely scratched the topology
| of the topic in his presentation. As the
| submarine ascended, it broke through the topology of the
| ocean. The topology of the road was slick
| with ice. Despite her calm topology, she was
| extremely nervous inside. The paint bubbled
| and peeled off the topology due to the heat.
|
| Our disagreement aside, I think these are hilarious. We
| should agree on that.
|
| The best was The surface of a doughnut
| resembles that of a coffee mug due to their similar
| structure.
| TimorousBestie wrote:
| > Alternatively, I would say, take a breath. Is this hill
| really the one worth dying on ?
|
| This applies both ways. Maybe you could relax with the
| facetiousness? It doesn't help your argument and makes
| you look like an ass.
| quantadev wrote:
| Doesn't apply both ways. He's denying the word "topology"
| is most often used when discussing surfaces and shapes,
| rather than meaning the name of the field of mathematics
| that has the same name.
| cvoss wrote:
| The phrase "applied X" invokes the technical, scientific, or
| academic meaning of X. So for example, "applied chemistry"
| does not refer to one's experience on a dating app.
| quantadev wrote:
| The word "topology" is _much_ more commonly used as a
| general synonym for "surfaces" than in any other way.
| terabytest wrote:
| I'm confused by the author's diagram claiming that AGI/ASI are
| points on the same manifold as next token prediction, chat
| models, and CoT models. While the latter three are provably part
| of the same manifold, what justifies placing AGI/ASI there too?
|
| What if the models capable of CoT aren't and will never be,
| regardless of topological manipulation, capable of processes that
| could be considered AGI? For example, human intelligence (the
| closest thing we know to AGI) requires extremely complex sensory
| and internal feedback loops and continuous processing unlike
| autoregressive models' discrete processing.
|
| As a layman, this matches my intuition that LLMs are not at all
| in the same family of systems as the ones capable of generating
| intelligence or consciousness.
| theahura wrote:
| Possible. AGI/ASI are poorly defined. I tend to think we're
| already at AGI, obviously many disagree.
|
| > For example, human intelligence (the closest thing we know to
| AGI) requires extremely complex sensory and internal feedback
| loops and continuous processing unlike autoregressive models'
| discrete processing.
|
| I've done a fair bit of connectomics research and I think that
| this framing elides the ways in which neural networks and
| biological networks are actually quite similar. For example, in
| mice olfactory systems there is something akin to a 'feature
| vector' that appears based on which neurons light up. Specific
| sets of neurons lighting up means 'chocolate' or 'lemon' or
| whatever. More generally, it seems like neuronal
| representations are somewhat similar to embedding
| representations, and you could imagine constructing an
| embedding space based on what neurons light up where.
| Everything on top of the embeddings is 'just' processing.
| fusionadvocate wrote:
| I believe we already have the technology required for AGI. It
| perhaps is analogous to a lunar manned station or a 2 mile
| tall skyscrapper. We have the technology required to build
| it, but we don't for various reasons.
| _alternator_ wrote:
| The question is not so much whether this is true--we can
| certainly represent any data as points on a manifold. Rather,
| it's the extent to which this point of view is useful. In my
| experience, it's not the most powerful perspective.
|
| In short, direct manifold learning is not really tractable as an
| algorithmic approach. The most powerful set of tools and
| theoretical basis for AI has sprung from statistical optimization
| theory (SGD, information-theoretical loss minimization, etc.).
| The fact that data is on a manifold is a tautological footnote to
| this approach.
| crgi wrote:
| Interesting read. Reminded me of the Trinity 3D manifold
| visualization tool which (among other things) let's you explore
| the hyperspace of neural networks: https://github.com/trinity-
| xai/Trinity
| vayllon wrote:
| Another type of topology you'll encounter in deep neural networks
| (DNNs) is network topology. This refers to the structure of the
| network -- how the nodes are connected and how data flows between
| them. We already have several well-known examples, such as auto-
| encoders, convolutional neural networks (CNNs), and generative
| adversarial networks (GANs), all of which are bio-inspired.
|
| However, we still have much to learn about the topology of the
| brain and its functional connectivity. In the coming years, we
| are likely to discover new architectures -- both internal within
| individual layers/nodes and in the ways specialized networks
| connect and interact with each other.
|
| Additionally, the brain doesn't rely on a single network, but
| rather on several ones -- often referred to as the "Big 7" --
| that operate in parallel and are deeply interconnected. Some of
| these include the Default Mode Network (DMN), the Central
| Executive Network (CEN) or the Limbic Network, among others. In
| fact, a single neuron can be part of multiple networks, each
| serving different functions.
|
| We have not yet been able to fully replicate this complexity in
| artificial systems, and there is still much to be learned and
| inspired by from this "network topologies".
|
| So, "Topology is all you need" :-)
| mbowcut2 wrote:
| To a topologist, everything is topology.
| kookamamie wrote:
| To a man with only a hammer, everything looks like a nail.
| Graviscalar wrote:
| I was one of the people that was super excited after reading the
| Chris Olah blogpost from 2014, and over the past decade I've seen
| the insight go exactly nowhere. It's neat but it hasn't driven
| any interesting results, though Ayasdi did some interesting stuff
| with TDA and Gunnar Carlson has been playing around with neural
| nets recently.
| rubitxxx8 wrote:
| What would you have expected to happen?
|
| Advances and insights sometimes lie dormant for decades or more
| before someone else picks them up and does something new.
| Graviscalar wrote:
| I would expect model/algorithm improvements from using
| topological concepts to analyze the manifolds in question or
| concrete results in model interpretability. Gunnar has
| studied some toy examples, but they were barely a step up
| from the ones Olah constructed for the sake of explanation
| and they haven't borne any further fruit.
|
| You can say any advance or insight is just lying dormant, it
| doesn't mean anything unless you can specifically articulate
| why it still has potential. I haven't made any claims on the
| future of the intersection of deep learning and topology, I
| was pointing out that it's been anything but dormant given
| the interest in it but it hasn't lead anywhere.
| theahura wrote:
| I think it's incorrect that the insight has gone nowhere. See,
| for eg, contrastive loss / clip, or vqgan image generation.
| Arguably also diffusion models.
|
| More generally, in my experience as an AI researcher,
| understandings of the geometry of data leads directly to
| changes in model architecture. Though people disparage that as
| "trial and error" it is far more directed than people on the
| outside give credit for.
| Graviscalar wrote:
| The geometric intuition is solid, but actually applying
| topology has been less fruitful in spite of a lot of people
| trying their best, as Chris Olah himself has said elsewhere
| in this thread.
| jxramos wrote:
| Ayasdi immediately came to mind too seeing this post. I haven't
| thought of them in a long time, looks like they got bought out
| in 2019, prepandemic too which was probably best since mid
| pandemic had a lot of poor valuations
|
| https://www.symphonyai.com/news/financial-services/ayasdi-jo...
| adsharma wrote:
| Isn't Deep Learning more like Graph Theory? I shared yesterday
| that Google published a paper called CRISP
| (https://arxiv.org/pdf/2505.11471) that carefully avoids any
| reference to the word "Graph".
|
| So then the question becomes what's the difference between Graph
| Theory and Applied Topology? Graphs operate on discrete
| structures and topology is about a continuous space. Otherwise
| they're very closely related.
|
| But the higher order bit is that AI/ML and Deep Learning in
| particular could do a better job of learning from and
| acknowledging prior art from related fields. Reusing older
| terminology instead of inventing new.
| kookamamie wrote:
| > This has been enough to get us to AGI.1
|
| Hard disagree.
| colah3 wrote:
| Since this post is based on my 2014 blog post
| (https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ),
| I thought I might comment.
|
| I tried really hard to use topology as a way to understand neural
| networks, for example in these follow ups:
|
| - https://colah.github.io/posts/2014-10-Visualizing-MNIST/
|
| - https://colah.github.io/posts/2015-01-Visualizing-Representa...
|
| There are places I've found the topological perspective useful,
| but after a decade of grappling with trying to understand what
| goes on inside neural networks, I just haven't gotten that much
| traction out of it.
|
| I've had a lot more success with:
|
| * The linear representation hypothesis - The idea that "concepts"
| (features) correspond to directions in neural networks.
|
| * The idea of circuits - networks of such connected concepts.
|
| Some selected related writing:
|
| - https://distill.pub/2020/circuits/zoom-in/
|
| - https://transformer-circuits.pub/2022/mech-interp-essay/inde...
|
| - https://transformer-circuits.pub/2025/attribution-graphs/bio...
| montebicyclelo wrote:
| Related to ways of understanding neural networks, I've seen
| these views expressed a lot, which to me seem like
| misconceptions:
|
| - LLMs are basically just slightly better `n-gram` models
|
| - The idea of "just" predicting the next token, as if next-
| token-prediction implies a model must be dumb
|
| (I wonder if this [1] popular response to Karpathy's RNN [2]
| post is partly to blame for people equating language neural
| nets with n-gram models. The stochastic parrot paper [3] also
| somewhat equates LLMs and n-gram models, e.g. "although she
| primarily had n-gram models in mind, the conclusions remain apt
| and relevant". I guess there was a time where they were more
| equivalent, before the nets got really really good)
|
| [1] https://nbviewer.org/gist/yoavg/d76121dfde2618422139
|
| [2] https://karpathy.github.io/2015/05/21/rnn-effectiveness/
|
| [3] https://dl.acm.org/doi/pdf/10.1145/3442188.3445922
| colah3 wrote:
| I guess I'll plug my hobby horse:
|
| The whole discourse of "stochastic parrots" and "do models
| understand" and so on is deeply unhealthy because it should
| be scientific questions about mechanism, and people don't
| have a vocabulary for discussing the range of mechanisms
| which might exist inside a neural network. So instead we have
| lots of arguments where people project meaning onto very
| fuzzy ideas and the argument doesn't ground out to
| scientific, empirical claims.
|
| Our recent paper reverse engineers the computation neural
| networks use to answer in a number of interesting cases
| (https://transformer-circuits.pub/2025/attribution-
| graphs/bio... ). We find computation that one might
| informally describe as "multi-step inference", "planning",
| and so on. I think it's maybe clarifying for this, because it
| grounds out to very specific empirical claims about mechanism
| (which we test by intervention experiments).
|
| Of course, one can disagree with the informal language we
| use. I'm happy for people to use whatever language they want!
| I think in an ideal world, we'd move more towards talking
| about concrete mechanism, and we need to develop ways to talk
| about these informally.
|
| There was previous discussion of our paper here:
| https://news.ycombinator.com/item?id=43505748
| mdp2021 wrote:
| Absolutely, the first task should be to understand how and
| why black boxes with emergent properties actually work, in
| order to further knowledge - but importantly, in order to
| improve them and build on the acquired knowledge to surpass
| them. That implies, curbing <<parrot[ing]>> and inadequate
| <<understand[ing]>>.
|
| I.e. those higher concepts are kept in mind as a goal. It
| is healthy: it keeps the aim alive.
| HarHarVeryFunny wrote:
| 1) Isn't it unavoidable that a transformer - a sequential
| multi-layer architecture - is doing multi-step inference ?!
|
| 2) There are two aspects to a rhyming poem:
|
| a) It is a poem, so must have a fairly high degree of
| thematic coherence
|
| b) It rhymes, so must have end-of-line rhyming words
|
| It seems that to learn to predict (hence generate) a
| rhyming poem, both of these requirements (theme/story
| continuation+rhyming) would need to be predicted
| ("planned") at least by the beginning of the line, since
| they are inter-related.
|
| In contrast, a genre like freestyle rap may also rhyme, but
| flow is what matters and thematic coherence and rhyming may
| suffer as a result. In learning to predict (hence generate)
| freestyle, an LLM might therefore be expected to learn that
| genre-specific improv is what to expect, and that rhyming
| is of secondary importance, so one might expect less rhyme-
| based prediction ("planning") at the start of each bar
| (line).
| agentcoops wrote:
| 1000%. It's really hard to express this to non-engineers who
| never wasted years of their life trying to work with n-grams
| and NLTK (even topic models) to make sense of textual data...
| Projects I dreamed of circa 2012 are now completely trivial.
| If you do have that comparison ready-at-hand, the problem of
| understanding what this mind-blowing leap means, to which end
| I find writing like the OP helpful, is so fascinating and
| something completely different than complaining that it's a
| "black box."
|
| I've expressed this on here before, but it feels like the
| everyday reception of LLMs has been so damaged by the general
| public having just gotten a basic grasp on the existence of
| machine learning.
| winwang wrote:
| hey chris, I found your posts quite inspiring back then, with
| very poetic ideas. cool to see you follow up here!
| riemannzeta wrote:
| I think it's interesting that in physics, different global
| symmetries (topological manifolds) can satisfy the same metric
| structure (local geometry). For example, the same metric tensor
| solution to Einstein's field equation can exist on
| topologically distinct manifolds. Conversely, looking at
| solutions to the Ising Model, we can say that the same lattice
| topology can have many different solutions, and when the system
| is near a critical point, the lattice topology doesn't even
| matter.
|
| It's only an analogy, but it does suggest at least that the
| interesting details of the dynamics aren't embedded in the
| topology of the system. It's more complicated than that.
| colah3 wrote:
| If you like symmetry, you might enjoy how symmetry falls out
| of circuit analysis of conv nets here:
|
| https://distill.pub/2020/circuits/equivariance/
| riemannzeta wrote:
| Thanks for this additional link, which really underscores
| for me at least how you're right about patterns in circuits
| being a better abstraction layer for capturing interesting
| patterns than topological manifolds.
|
| I wasn't familiar with the term "equivariance" but I "woke
| up" to this sort of approach to understanding deep neural
| networks when I read this paper, which shows how restricted
| boltzman machines have an exact mapping to the
| renormalization group approach used to study phase
| transitions in condensed matter and high energy physics:
|
| https://arxiv.org/abs/1410.3831
|
| At high enough energy, everything is symmetric. As energy
| begins to drain from the system, eventually every symmetry
| is broken. All fine structure emerges from the breaking of
| some symmetries.
|
| I'd love to get more in the weeds on this work. I'm in my
| own local equilibrium of sorts doing much more mundane
| stuff.
| theahura wrote:
| Thanks for the follow up. I've been following your circuits
| thread for several years now. I find the linear representation
| hypothesis very compelling, and I have a draft of a review for
| Toy Models of Superposition sitting in my notes. Circuits I
| find less compelling, since the analysis there feels very tied
| to the transformer architecture in specific, but what do I
| know.
|
| Re linear representation hypothesis, surely it depends on the
| architecture? GANs, VAEs, CLIP, etc. seem to explicitly model
| manifolds. And even simple models will, due to optimization
| pressure, collapse similar-enough features into the same linear
| direction. I suppose it's hard to reconcile the manifold
| hypothesis with the empirical evidence that simple models will
| place similar-ish features in _orthogonal_ directions, but
| surely that has more to do with the loss that is being
| optimized? In Toy Models of Superposition, you 're using a MSE
| which effectively makes the model learn an autoencoder
| regression / compression task. Makes sense then that the
| interference patterns between co-occurring features would
| matter. But in a different setting, say a contrastive loss
| objective, I suspect you wouldn't see that same interference
| minimization behavior.
| colah3 wrote:
| > Circuits I find less compelling, since the analysis there
| feels very tied to the transformer architecture in specific,
| but what do I know.
|
| I don't think circuits is specific to transformers? Our work
| in the Transformer Circuits thread often is, but the original
| circuits work was done on convolutional vision models
| (https://distill.pub/2020/circuits/ )
|
| > Re linear representation hypothesis, surely it depends on
| the architecture? GANs, VAEs, CLIP, etc. seem to explicitly
| model manifolds
|
| (1) There are actually quite a few examples of seemingly
| linear representations in GANs, VAEs, etc (see discussion in
| Toy Models for examples).
|
| (2) Linear representations aren't necessarily in tension with
| the manifold hypothesis.
|
| (3) GANs/VAEs/etc modeling things as a latent gaussian space
| is actually way more natural if you allow superposition
| (which requires linear representations) since central limit
| theorem allows superposition to produce Gaussian-like
| distributions.
| theahura wrote:
| > the original circuits work was done on convolutional
| vision models
|
| O neat, I haven't read that far back. Will add it to the
| reading list.
|
| To flesh this out a bit, part of why I find circuits less
| compelling is because it seems intuitive to me that neural
| networks more or less smoothly blend 'process' and 'state'.
| As an intuition pump, a vector x matrix matmul in an MLP
| can be viewed as changing the basis of an input vector (ie
| the weights act as a process) or as a way to select
| specific pieces of information from a set of embedding rows
| (ie the weights act as state).
|
| There are architectures that try to separate these out with
| varying degrees of success -- LSTMs and ResNets seem to
| have a more clear throughline of 'state' with various
| 'operations' that are applied to that state in sequence.
| But that seems really architecture-dependent.
|
| I will openly admit though that I am very willing to be
| convinced by the circuits paradigm. I have a background in
| molecular bio and there's something very 'protein pathways'
| about it.
|
| > Linear representations aren't necessarily in tension with
| the manifold hypothesis.
|
| True! I suppose I was thinking about a 'strong' form of
| linear representations, which is something like: features
| are represented by linear combinations of neurons _that
| display the same repulsion-geometries as observed in Toy
| Models_ , but that's not what you're saying / that's me
| jumping a step too far.
|
| > GANs/VAEs/etc modeling things as a latent gaussian space
| is actually way more natural if you allow superposition
|
| Superposition is one of those things that has always been
| so intuitive to me that I can't imagine it _not_ being a
| part of neural network learning.
|
| But I want to make sure I'm getting my terminology right --
| why does superposition necessarily require the linear
| representation hypothesis? Or, to be more specific, does
| [individual neurons being used in combination with other
| neurons to represent more features than neurons]
| necessarily require [features are linear compositions of
| neurons]?
| colah3 wrote:
| > True! I suppose I was thinking about a 'strong' form of
| linear representations, which is something like: features
| are represented by linear combinations of neurons that
| display the same repulsion-geometries as observed in Toy
| Models, but that's not what you're saying / that's me
| jumping a step too far.
|
| Note this happens in "uniform superposition". In reality,
| we're almost certainly in very non-uniform superposition.
|
| One key term to look for is "feature manifolds" or
| "multi-diemsnional features". Some discussion here:
| https://transformer-circuits.pub/2024/july-
| update/index.html...
|
| (Note that the term "strong linear representation" is
| becoming a term of art in the literature referring to the
| idea that all features are linear, rather than just most
| or some.)
|
| > I want to make sure I'm getting my terminology right --
| why does superposition necessarily require the linear
| representation hypothesis? Or, to be more specific, does
| [individual neurons being used in combination with other
| neurons to represent more features than neurons]
| necessarily require [features are linear compositions of
| neurons]?
|
| When you say "individual neurons being used in
| combination with other neurons to represent more features
| than neurons", that's a way one might _informally_ talk
| about superposition, but doesn't quite capture the
| technical nuance. So it's hard to know the full scope of
| what you intend. All kinds of crazy things are possible
| if you allow non-linear features, and it's not
| necessarily clear what a feature would mean.
|
| Superposition, in the narrow technical sense of
| exploiting compressed sensing / high-dimensional spaces,
| requires linear representations and sparsity.
| theahura wrote:
| > One key term to look for is "feature manifolds" or
| "multi-diemsnional features"
|
| I should probably read the updates more. Not enough time
| in the day. But yea the way you're describing feature
| manifolds and multidimensional features, especially the
| importance of linearity-in-properties and not necessarily
| linearity-in-dimensions, makes a lot of sense and is
| basically how I default think about these things.
|
| > but doesn't quite capture the technical nuance. So it's
| hard to know the full scope of what you intend.
|
| Fair, I'm only passingly familiar with compressed sensing
| so I'm not sure I could offer a more technical definition
| without, like, a much longer conversation! But it's good
| to know in the future that in a technical sense linear
| representations and superposition are dependent.
|
| > all features are linear, rather than just most or some
|
| Potentially a tangent, but compared to what? I suppose
| the natural answer is "non linear features" but has there
| been anything to suggest that neural networks represent
| concepts in this way? I'd be rather surprised if they did
| within a single layer. (Across layers, sure, but that
| actually starts to pull me more towards circuits)
| iNic wrote:
| My guess is that the linear representation hypothesis is only
| approximately right in the sense that my expectation is that it
| is more like a Lie Group. Locally flat, but the concept breaks
| at some point. Note that I am a mathematician who knows very
| little about machine learning apart from taking a few classes
| at uni
| godelski wrote:
| Loved these posts and they inspired a lot of my research and
| directions during my PhDs.
|
| For anyone interested in these may I also suggest learning
| about normalizing flows? (They are the broader class to flow
| matching) They are learnable networks that learn coordinate
| changes. So the connection to geometry/topology is much more
| obvious. Of course the down side of flows is you're stuck with
| a constant dimension (well... sorta) but I still think they can
| help you understand a lot more of what's going on because you
| are working in a more interpretable environment
| dang wrote:
| That earlier post had a few small HN discussions (for those
| interested):
|
| _Neural Networks, Manifolds, and Topology (2014)_ -
| https://news.ycombinator.com/item?id=19132702 - Feb 2019 (25
| comments)
|
| _Neural Networks, Manifolds, and Topology (2014)_ -
| https://news.ycombinator.com/item?id=9814114 - July 2015 (7
| comments)
|
| _Neural Networks, Manifolds, and Topology_ -
| https://news.ycombinator.com/item?id=7557964 - April 2014 (29
| comments)
| nis0s wrote:
| > One way to think about neural networks, especially really large
| neural networks, is that they are topology generators. That is,
| they will take in a set of data and figure out a topology where
| the data has certain properties. Those properties are in turn
| defined by the loss function.
|
| Latent spaces may or may not have useful topology, so this idea
| is inherently wrong, and builds the wrong type of intuition.
| Different neural nets will result in different feature space
| understanding of the same data, so I think it's incorrect to
| believe you're determining intrinsic geometric properties from a
| given neural net. I don't think people should throw around words
| carelessly because all that does is increase misunderstanding of
| concepts.
|
| In general, manifolds can help discern useful characteristics
| about the feature space, and may have useful topological
| structures, but trying to impose an idea of "topology" on this is
| a stretch. Moreover, the kind of basics examples used in this
| blog post don't help prove the author's point. Maybe I am
| misunderstanding this author's description of what they mean, but
| this idea of manifold learning is nothing new.
| khoangothe wrote:
| Cool post! Thanks
| deepburner wrote:
| This whole article is just a nothingburger. Saying something is
| applied topology is only one step more advanced than saying
| something is maths - duh. These mathematical abstractions are
| incredibly general and and you can pretty much draw up anything
| in terms of anything, the challenging part is being able to turn
| around and use the model/abstraction to say things about the
| thing you're abstracting. I don't think scholars have been very
| successful in that regard, less so this article.
|
| Yeah deep learning is applied topology, it's also applied
| geometry, and probably applied algebra and I wouldn't be
| surprised if it was also applied number theory.
| ada1981 wrote:
| For the last few years, I've been "seeing maps" whenever I think
| about LLMs. It's always felt like the most natural way to
| understand what is going on.
|
| It's also for this reason that I think new knowledge is
| discoverable from with in LLMs.
|
| I imagine having a topographic map of some island that has only
| been explored partially by humans. But if I know the surrounding
| topography, I can make pretty accurate guesses about the areas I
| haven't been. And I think the same thing can be applied to
| certain areas of human knowledge, especially when represented as
| text or symbolically.
| revskill wrote:
| So ml is about drawing the line.
| Koshkin wrote:
| On the TDA in general, I found this series of articles by Brandon
| Brown helpful:
|
| https://github.com/outlace/outlace.github.io
| CliffStoll wrote:
| Applied topology. Might a Klein bottle actually be a useful?
| colah3 wrote:
| It's a bit different than what's discussed here, but color-
| contrast detectors in neural networks can be thought of as
| forming a Klein bottle:
| https://distill.pub/2020/circuits/equivariance/#hue-rotation...
|
| (This is, in some sense, for similar reason to Gunnar Carlson
| et al finding a Klein bottle when looking at high-contrast
| image patches, except one level more abstract, since it's about
| features rather than data points.)
| garganzol wrote:
| I want to share one more related observation: by definition,
| topology math refers to geometrical objects and transformations.
| But there exists another, more computer-esque definition of
| topology that defines relations between abstract objects.
|
| For example, let's take a look at graph data structure. A graph
| has a set of stored objects (vertices) and a set of stored
| relations between the vertices (edges). In this way, graph
| defines a topology in discrete form.
|
| Let's take a look at network data structure which is closely
| related to the graph. It is very much the same idea, but it
| additionally has a value stored in every edge. A network has a
| set of objects (vertices) and a set of relations between the
| objects (edges), while edges also hold edge values. So it is also
| a form of topology because the network defines the relations
| between the abstract objects.
|
| In this light, you can view a graph as a neural network with {0,
| 1} weights. The graph edge is either present or absent, hence {0,
| 1} values only. The network structure, however, can hold any
| assigned value in every edge, so every connection between objects
| (neurons) can be characterized not only by its presence, but also
| by edge-assigned values (weights). Now we get the full model of a
| neural network. And yes, it is built upon topology in its
| discrete form.
___________________________________________________________________
(page generated 2025-05-20 23:00 UTC)