[HN Gopher] General Theory of Neural Networks
___________________________________________________________________
General Theory of Neural Networks
Author : rdlecler1
Score : 202 points
Date : 2024-07-11 14:34 UTC (4 days ago)
(HTM) web link (robleclerc.substack.com)
(TXT) w3m dump (robleclerc.substack.com)
| rdlecler1 wrote:
| Despite vast implementation constraints spanning diverse
| biological systems, a clear pattern emerges the repeated and
| recursive evolution of Universal Activation Networks (UANs).
| These networks consist of nodes (Universal Activators) that
| integrate weighted inputs from other units or environmental
| interactions and activate at a threshold, resulting in an action
| or an intentional broadcast. Minimally, Universal Activator
| Networks include gene regulatory networks, cell networks, neural
| networks, cooperative social networks, and sufficiently advanced
| artificial neural networks.
|
| Evolvability and generative open-endedness define Universal
| Activation Networks, setting them apart from other dynamic
| networks, complex systems or replicators. Evolvability implies
| robustness and plasticity in both structure and function,
| differentiable performance, inheritable replication, and
| selective mechanisms. They evolve, they learn, they adapt, they
| get better and their open-enedness lies in their capacity to form
| higher-order networks subject to a new level of selection.
| RaftPeople wrote:
| Thoughts:
|
| > _2-UANs operate according to either computational principles
| or magic._
|
| Given that quantum effects do exist, does this mean that the
| result of quantum activity is still just another physical input
| into the UAN and does not change the analysis of what the UAN
| computes? It seems difficult to think that what a UAN computes
| is not impacted by those lower level details (meaning
| specifically quantum effects, I'm not thinking of just
| alternate implementations).
|
| > _4-A UANs critical topology, and its implied gating logic,
| dictate its function, not the implementation details._
|
| Dynamic/short term networks in brain:
|
| Neurons in the brain are dynamically inhibited+excited due to
| various factors including brain waves, which seems like they
| are dynamically shifting between different networks on the fly.
| I assume when you say topology, you're not really thinking in
| terms of static physical topology, but more of the current
| logical topology that may be layered on top of the physical?
|
| Accounting for Analog:
|
| A neurons function is heavily influenced by current analog
| state, how is that accounted for in the formula for the UAN?
|
| For example, activation at the same synapse can either trigger
| an excitatory post synaptic action potential or an inhibitory
| post synaptic action potential depending on the concentration
| of permeant ions inside and outside the cell at that moment.
|
| I'm assuming a couple possible responses might be:
|
| 1-Even though our brain has analog activity that influence the
| operation of cells, there is still an equivalent UAN that does
| not make use of analog.
|
| or
|
| 2-Analog activity is just a lower level UAN (e.g. atom/molecule
| level)
|
| I don't think either of those are strong responses. The first
| triggers the question: "How do you know and how do you find
| that UAN?". The second one seems to push the problem down to
| just needing to simulate physics within +/- some error.
| kaibee wrote:
| > Given that quantum effects do exist, does this mean that
| the result of quantum activity is still just another physical
| input into the UAN
|
| Yeah, it could be a spurious input though. My understanding
| is that quantum mechanics doesn't really matter at biological
| scale, and that kinda makes sense right? Like, if this whole
| claim about biology being reducible to the topology of the
| components of the network is true, then the first thing you'd
| do is try to evolve components that are robust to quantum
| noise or leverage it for some result (ie: one can imagine
| some binding site constructed in such a way that it requires
| a rare event that none-the-less actually has a very specific
| probability of occurring).
|
| > and does not change the analysis of what the UAN computes?
| It seems difficult to think that what a UAN computes is not
| impacted by those lower level details (meaning specifically
| quantum effects, I'm not thinking of just alternate
| implementations).
|
| What the UAN computes is impacted by those lower level
| details, but it is abstractable given enough simulation data.
|
| ie, imagine if you had a perfect molecular scan of a modern
| CPU that detailed the position of every atom. While it would
| be neat to simulate it physically, for the purpose of
| analysis, you'd likely want to at least abstract it to the
| transistor level. The 'critical topology' is I guess, the
| highest possible level of abstraction before a CPU tester can
| tell your simulation from an atom-level simulation.
|
| Now for CPUs, we designed that model first and then built the
| CPU. In biology, it evolved on the physical level, but still
| maps to a 'critical topology'.
| AndrewKemendo wrote:
| This is another example of Markov Chains in the wild - so that's
| what he's seeing
|
| The general nn is a discrete implementation of that
|
| https://en.m.wikipedia.org/wiki/Markov_chain
| rdlecler1 wrote:
| No, too inclusive.
| AIorNot wrote:
| What's wild to me is that Donald Hoffman is also proposing a
| similar foundation for his metaphysical theory of consciousness,
| ie that it is a fundamental property and that it exists outside
| of spacetime and leads via a markov chain of conscious agents (in
| a Network as described above)
|
| Ie everything that exists may be the result of some kind of Uber
| Network existing outside of space and time
|
| It's a wild theory but the fact that these networks keep popping
| up and recurring at level upon level when agency and intelligence
| is needed is crazy
|
| https://youtu.be/yqOVu263OSk?si=SH_LvAZSMwhWqp5Q
| mistermann wrote:
| I don't think one even needs "supernatural" explanations.
|
| 1. Consider the base hardware of each agent:
|
| http://neuropathologyblog.blogspot.com/2017/06/shannon-curra...
|
| 2. Consider that (according to science anyways) there is no
| central broadcaster of reality (it is at least plausible)
|
| 3. Consider each agent (often/usually) "knows" all of reality,
| or at least any point you query them about (for sure: all
| agents claim to know the unknowable, regularly; I have yet to
| encounter one who can stop a "powerful" invocation of #3 (or
| even try: the option seems literally unavailable), _though
| minor ones can be overridden fairly trivially_ (I can think of
| two contrasting paths of interesting consideration based on
| this detail, one of them being extremely optimistic, _and
| trivially plausible_ ))
|
| Simplified: what is known to be, _is_ (locally).
|
| 4. Consider the possibility (or assume as a premise of a
| thought experiment) that reality and the universe are not
| _exactly_ the very same thing ( "it exists outside of
| spacetime"), though it may appear that they are (see #3)
|
| Is it not fairly straightforward what is going on?
|
| A big part of the problem is that #3 is ~inevitably[1] invoked
| if such things are analyzed, screwing up the analysis, thus
| rendering the theory _necessarily_ "false" (it "is"
| false...though, it will typically not be asserted as such
| explicitly, and direct questions will be ignored/dodged).
|
| [1] which is... _weird_ (the inevitable part...like, it is as
| if consciousness is ~hardwired to disallow _certain_ inspection
| (highly predictable evasive actions are invoked in response),
| something which can _easily be tested /demonstrated_).
| cscurmudgeon wrote:
| Why is #3 obvious? How can agents know all of reality? May be
| a subset?
| mistermann wrote:
| knowledge = knowledge (true belief)
|
| "knowledge" = belief (possibly true but not necessarily,
| but _sincerely_ perceived as "true")
|
| (I'm considering this from an abstract / autistic / "That's
| _pedantic!_ [so stop doing it] " perspective, so I include
| quotation marks to note the technical distinction...in
| phenomenological analysis, perhaps they'd be left out, to
| better illustrate the local experience of reality, the true
| "is-ness" _as it is_. In normative discussions ( "anything
| that good hackers would find interesting"), these things
| are generally rather taboo.)
|
| There's lots of nuance I'm leaving out, but that's the
| general idea.
|
| A popular though terminating description for the phenomenon
| is "that's just people expressing their opinion, _everyone_
| does it, that 's what _everything_ boils down to " (which
| can make it not only not obvious, but damn near _invisible_
| )...but consider the semantic differences of that with and
| without the inclusion of the word "just". (Also: watch out
| for #3, it's recursively self-referential, and has
| substantial cloaking / shape-shifting abilities. It is
| _almost_ always and everywhere.)
|
| An alternate perspective: consider what an uneducated
| person "sees" in "reality" (aka: what "is", and "is not")
| as they go about their day, compared to highly educated (as
| opposed to _knowledgeable_ ) people _from very distinct
| disciplines_.
| AIorNot wrote:
| Can you explain #3 and 4# more clearly?
|
| in #2 you are claiming there is no objective reality or no
| 'broadcaster' of reality
|
| We must assume some things as being objective such as a
| rational universe in order to make any claims at all.
|
| -if you are saying in #3 that humans as conscious agents make
| subjective claims about reality but that those claims are in
| fact 'the reality' for that agent or person, that is a
| subjective claim. (I'm not saying that that subjective
| reality isn't true for that person)
|
| Also, Hoffman doesn't make a 'supernatural' claim per se, his
| claim is simply that reality as 'we all see it' is NOT the
| whole story, and that it is in fact only the projection of a
| vast, infinitely complex network of conscious agents that
| creates what we perceive as the material universe and time.
| He starts with the idea that consciousness as a property is
| fundamental, existing outside of space and time and that if
| you apply reasoning and mathematics that networks of agents
| acting as UANs in a sense project that material universe into
| being, with that assumption, ie that it extrapolates to our
| entire universe.
|
| I'm not sure I'm (or anyone for that matter) is really
| qualified to answer that claim..it's so big that it does
| verge on mysticism. that's why I said its such a wild idea,
| but I found the article above another interesting piece of
| evidence for Hoffman, because it talks about a general theory
| underlying such networks:
|
| whose "repeated and recursive evolution of Universal
| Activation Networks (UANs). These networks consist of nodes
| (Universal Activators) that integrate weighted inputs from
| other units or environmental interactions and activate at a
| threshold, resulting in an action or an intentional
| broadcast"
|
| ie this is very similar to Hoffmans system of Conscious
| Agents -which is an extreme theory of such networks that I
| described above
|
| https://evolutionnews.org/2023/10/eccentric-theories-of-
| cons...
| mistermann wrote:
| These are good questions, I am on mobile at the moment so
| won't be able to make a response that does them justice for
| 2 days or so.
|
| I'd think my other post provides some relevant content
| though?
|
| In the meantime it may help...an important axiom in my
| model/theory is that the universe exists independent of us,
| but reality is downstream of us. I think Donald's theory is
| based on Idealism maybe, where he disagrees and thinks
| reality is downstream of us, and the universe is downstream
| of reality? But that raises some very tricky paradoxes,
| more so than the one main paradox/problem that all models
| have (I think? Maybe not, maybe I just lack adequate
| imagination! And it doesn't make him necessarily wrong, but
| it puts it into the same category as God(s) imho: anything
| is possible, including the "impossible". Which is fine,
| _but please acknowledge it explicitly, Donald_.)
|
| I'm not terribly hung up on which model one subscribes to
| (or _has been_ subscribed to) in general, but I am
| extremely hung up on logical inconsistencies and paradoxes
| within them, that are not explicitly acknowledged _in a
| non-dismissive manner_...this is fundamentally important to
| my model, as mine has an opinionated ~ethical component
| (Utopianism), and an extremely strong dislike for
| "imposters" in this regard.
| optimalsolver wrote:
| >it exists outside of spacetime
|
| So I guess this theory won't be subject to empirical testing
| any time soon?
| jungturk wrote:
| Perhaps "outside" means "resident on the boundary of", a la
| holograms and Maldecena's AdS/CFT work, and so still within
| reach of experiment?
| codethief wrote:
| How is AdS/CFT (as it pertains to describing spacetime &
| the Standard Model) within reach of experiment? We're not
| even in an AdS universe.
| hughesjj wrote:
| I was going to write something up but honestly the top
| two answers in this physics stack exchange do it more
| competently and comprehensively than I'm able
|
| https://physics.stackexchange.com/questions/122540/why-
| are-a...
|
| My takeaway/summary is 'AdS/CFT let's you temporarily
| change your space/model to make the math easier and then
| map it back into the original model. Although we don't
| currently have such a model specified for our particular
| observed universe, it still allows us to study
| functionality equivalent behavior and make some
| determinations over what is/isn't/could be possible'
| codethief wrote:
| From the link you posted:
|
| > Ok, so if I understand correctly, AdS spaces aren't
| meant to directly model the universe, but rather are used
| as a tool to make certain calculations in CFT easier?
|
| GP's comment was modelling the universe as AdS/CFT,
| though, which is why my question addressed precisely that
| part:
|
| > How is AdS/CFT (as it pertains to describing spacetime
| & the Standard Model)
| jungturk wrote:
| Sorry, my point was to (almost facetiously) interject the
| topological concept of "boundary/bulk" correspondence (as
| also exists with holograms & AdS/CFT) rather than
| suggesting the correctness of AdS/CFT.
|
| The idea that "outside" becomes available via a change in
| perspective from boundary to bulk.
| austinjp wrote:
| For those who, like me, are not familiar with AdS/CFT, it's
| "anti-de Sitter/conformal field theory correspondence"
| which is "a conjectured relationship between two kinds of
| physical theories", allowing quantum field theories to be
| made more mathematically tractable.
|
| https://en.m.wikipedia.org/wiki/AdS/CFT_correspondence
| downboots wrote:
| A universe where everything can be empirically tested would
| be conveniently and suspiciously human-centered
| janto wrote:
| And neccessary.
| hughesjj wrote:
| Well, likely necessary to determine a unique and
| comprehensive, consistent model (or equivalence class
| thereof)
|
| There's no guarantee such a model can be determined via
| an observer in the universe the observer is observing.
|
| Functionality though, yeah that's the goal to either find
| such a model or show that one can't exist, or can't be
| proven one way or the other.
| naasking wrote:
| I don't see how that follows. Almost nothing about what
| we've discovered is human-centric, we've had to build
| machines to sense beyond our human perceptual limits. In
| what ways are gravitational waves detected by LIGO human-
| centric?
| janto wrote:
| In that humans built LIGO to bring forth phenomena for
| them. There is no science without an observer. The
| universe might exist without us, but without an observer
| there is no-one to describe it.
| naasking wrote:
| I don't see how this has any bearing on the original
| point. Why would universe where everything can be
| empirically tested be _conveniently and suspiciously_
| human-centered?
| janto wrote:
| That was my reply to you, not the original point. If it
| helps:
|
| I think the GP implied with the statement that we should
| subscribe to some non-empirical theories ("outside
| spacetime"), because the endeavour is not supposed to be
| human-centered.
|
| 1. You said that the observations are not human-centered
| since we need to make machines, so the GP does not need
| to be suspicious and should stick to empiricism.
|
| 2. Both you and the GP seem to assume that the endeavour
| is not supposed to be human-centered
|
| 3. I say it is human-centered because we make the
| machines for us as observers at the center, so your
| argument doesn't quite work
|
| 4. I think our science is entangled with us as observers
| on _many_ levels, and it should be. So the GP 's
| statement should actually be pushed through suspicion
| towards neccessity.
| naasking wrote:
| > You said that the observations are not human-centered
| since we need to make machines
|
| I didn't say LIGO was not human centered, but that the
| gravitational waves which LIGO detected are not human-
| centered.
|
| I just don't think your take is equivalent to the OP's
| claim. Yes, our instruments are human-centered because
| humans are the observers, and so phenomena outside of our
| perceptual range has to be projected into our perceptual
| range. That doesn't imply that the underlying phenomena
| are human-centered, or that the theories formed from
| those observations are semantically human-centered
| (syntactically they are because humans have to be able to
| read them).
|
| I frankly don't even understand the OP's claim: how does
| the proposition "this is a universe where everything can
| be empirically tested" logically entail "this universe is
| human centered". I can agree with being suspicious of the
| claim that we can empirically test everything, I just
| don't get how that entails human centeredness.
| janto wrote:
| Well, my take is not equivalent to OP, in that I actually
| don't agree :) I am a human-centered empiricist.
|
| I think OP's logic flows the other way: skeptical of
| human-centeredness position (like you) and from that
| skeptical of going purely empirical (not like you/me).
|
| I'd say LIGO described phenomena as gravitational waves
| through the interaction of the experiment. They didn't
| detect anything because that would go beyond empiricism
| into assuming the existence of a thing beyond the
| interaction
| pyinstallwoes wrote:
| So the gnostics were right? Demiurge spatial-temporal firewall
| of reality nodes
| quetzthecoatl wrote:
| weren't the sophia/gnosis, emnations and eons were from greek
| philosophy? also any philosophy/hottakes that stress on
| duality (what's seen here and what's out there that is
| causing what's seen here - such as manichean, advaita etc).
| kovezd wrote:
| Category theory is the mathematical formulation/foundation of
| this "Uber Network".
|
| Graphs are the most basic unit of meaning.
| rdlecler1 wrote:
| No, graphs are too inclusive.
| superidiot1932 wrote:
| Can you expand on that?
| rdlecler1 wrote:
| More generally there's graph neural networks, for
| instance, but not you're including many dynamic networks
| that are not open-ended or evolvable. The idea is to
| identify common dynamics and add constraints on the types
| of networks that are included to find general principles
| within that class. Kisen the constraints, you make the
| class too broad and can't identify common principles.
| raidicy wrote:
| I am a hobby student of category theory. Is there any
| breadcrumbs to your comment?
| rdlecler1 wrote:
| I don't know if this exists outside of spacetime, but I have a
| suspicion that UACs didn't begin with gene regulatory networks,
| but are more fundamental part of a computational universe
| hypothesis.
| winter_blue wrote:
| This is a pretty cool theory that resonates well with me. What
| are some good places I can read more about this (and related
| theories)?
| rdlecler1 wrote:
| This sits in a larger field of complexity theory and complex
| adaptive systems. There was also some interesting work on
| "Artificial Life" although that research program seems to
| have fallen out of favor. My introduction in 1995 was the
| book Chaos and then Stuart Kauffman's At Home in the
| Universe. Wolframs New Kind of Science was also interesting.
| CuriouslyC wrote:
| This is just Berkeley's idealism with a bunch of
| pseudoscientific hand waiving.
|
| Consciousness isn't outside of space and time, it creates it.
| humansareok1 wrote:
| We've already invalidated hidden variable theories in Physics
| so I find it hard to believe consciousness has some separate
| class of hidden effects still undiscovered and allowable in our
| universe.
| naasking wrote:
| > We've already invalidated hidden variable theories in
| Physics
|
| No we haven't.
| humansareok1 wrote:
| The only theories not ruled out by Bell are non local. You
| have to accept such a mountain of nonsense for any non
| local theory to be valid that I don't think anyone takes
| them seriously.
| naasking wrote:
| Aside from non-local theories which have been known since
| the 1950s are are regularly used in quantum chemistry,
| superdeterministic theories have seen virtually no
| development. Claims that they either of these approaches
| entail a mountain of nonsense are based on no evidence.
| codethief wrote:
| > We've already invalidated hidden variable theories in
| Physics
|
| Not quite, see e.g.
| https://en.wikipedia.org/wiki/De_Broglie%E2%80%93Bohm_theory
| 29athrowaway wrote:
| In the biology there are families of neurons, each one with
| different morphologies.
| rdlecler1 wrote:
| Are those just implementation details?
| 29athrowaway wrote:
| The scientist ambition is a grand unifying theory of
| minimalistic, reductionist and elegant principles that
| explain everything. Some even argue that we are already
| there.
|
| But the truth is: when it comes to neurons, all those
| theories are effectively inferior to what evolution has
| achieved. They can explain some of what is going on, but they
| cannot reproduce the results of the biological counterparts.
|
| The artificial results either require orders of magnitude
| more power, or examples, or has to be hardwired or trained in
| advance, or requires a billion dollars facility to
| manufacture the hardware involved.
|
| Biological neurons get trained as they do inference, require
| fewer examples, use less power and the agent can get drunk
| and high and lose millions of neurons and synaptic
| connections and their brain will either keep working as
| usual, or everything will get rewired after a while.
|
| We don't understand as much as we claim to do yet, if we did,
| we would have the same results at least.
| rdlecler1 wrote:
| Those neurons are being trained the day we were born.
| Reality corresponds to about 11 million bits per second.
| What I suspect's happening is that we train higher and
| higher levels of abstraction and we get to a point where
| new knowledge is involves training a new permutation of a
| few high level neurons.
| dboreham wrote:
| Before we are born, most likely too.
| smokel wrote:
| People seem to be obsessed with finding fundamental properties in
| neural networks, but why not simply marvel at the more basic
| incredible operations of addition and multiplication, and stop
| there?
| falcor84 wrote:
| Evolutionary pressure is such that, generally speaking,
| individuals who "stop there" are less successful than ones who
| always crave more. We are all descendants of those who were
| "obsessed" with: mating, hoarding, conquering and yes, finding
| patterns and fundamental properties.
| smokel wrote:
| My point exactly, but I obviously failed to communicate that
| :)
|
| Multiplication and addition are more fundamental than neural
| networks.
| Jerrrrrrry wrote:
| >Multiplication and addition are more fundamental than
| neural networks.
|
| Time and complexity are not related, just acquaintances.
| sixo wrote:
| God this grandiose prose style is insufferable. Calm down.
|
| Anyway, this doesn't even try to make the case that that equation
| is universal, only that "learning" is a general phenomena of
| living systems, which can be modeled probably in many different
| ways.
| ai4ever wrote:
| architecture astronauts let loose on unified field theories..
| talking warm and fuzzy - big bold ideas.
|
| let them, i say, until, the tide shifts to something else
| tomorrow, and a new generation of big-picture thought leaders
| take over dumping their insufferable text on the populace.
| proof_by_vibes wrote:
| The excitement of new horizons is necessary for innovation, and
| a substack article is a safe way to express that excitement.
| It's clearly understood by the choice of medium that this is
| meant to be speculation, so there aren't any significant risks
| in engaging with the text on its own terms.
| cfgauss2718 wrote:
| Agreed, I can't help but feel there is some overcompensation
| driving the style of writing. It was difficult to finish.
| grape_surgeon wrote:
| Yeah my bs meter went off in seconds. So much fluff
| downboots wrote:
| Can you share the source code? (Half joking)
| mistermann wrote:
| You should write a blog post on this (not joking at all).
| rdlecler1 wrote:
| You're right. Writing is hard--especially when you're cutting
| across disciplines. I wasn't happy with the writing, but I
| stand by the claims.
| sharp11 wrote:
| Personally, I find the writing to be just fine. It is clear
| and cogent. I don't have enough background to follow all the
| details, but I certainly hope you are not discouraged from
| pursuing big ideas by negative comments on style!
| cfgauss2718 wrote:
| There are some interesting parallels to ideas in this article and
| IIT. The focus on parsimony in networks, and pruning connections
| that are redundant to reveal the minimum topology (and the
| underlying computation)is reminiscent of parts of IIT: I'm
| thinking of the computation of the maximally irreducible concept
| structure via searching for a network partition which minimizes
| the integrated cause-effect information in the system. Such
| redundant connections are necessarily severed by the partition.
| zaven wrote:
| What's IIT?
| diego898 wrote:
| I'm guessing Integrated Information Theory
|
| https://en.wikipedia.org/wiki/Integrated_information_theory
| t_serpico wrote:
| "Topology is all that matters" --> bold statement, especially
| when you read the paper. The original authors were much more
| reserved in terms of their conclusions.
| griffzhowl wrote:
| Yes, on its face it looks like he's saying that you can throw
| out the weights of any network and still expect the same or
| similar behaviour, which is obviously false. It's also
| contradicted in that very section where he reports from the
| cited paper that randomized parameters reproduced the desired
| behaviour in about 1 in 200 cases. All these cases have the
| same network topology so while that might be higher than
| expected probability for retaining function with randomized
| paramteres (over 2-3 orders of magnitude), it's also a clear
| demonstration that more than topology is significant
| rdlecler1 wrote:
| The topology needs to be information bearing. Weights of
| 0.0001 are likely spurious and if other weights are so
| relatively big they can effectively make the other fan in
| weights spurious as well.
| rdlecler1 wrote:
| The original papers were published in scientific journals. More
| assertive claims aren't kosher.
| flufluflufluffy wrote:
| We must always remember that all models are wrong, though some
| are useful.
| LarsDu88 wrote:
| There are a whole lot more activation functions used nowadays in
| NNs
|
| https://dublog.net/blog/all-the-activations/
|
| The author is extrapolating way too much. The simplest model of X
| is similar to the simplest model of Y, therefore the common
| element is deep and insightful, rather than mathematical modelers
| simply being rationally parsimonious.
| rdlecler1 wrote:
| Activation functions are implementation details. See appendix
| for the general formula.
| LarsDu88 wrote:
| Ok, I get what you mean now. You can build a model by
| plugging in any activation function into the two slots in the
| equation at the bottom.
|
| There's a typo in the activation function next to "otherwise"
| in the "Ant Pheromone Signaling" row.
| cventus wrote:
| Nice list and history of common activation units used today.
|
| Small note though, the heaviside function used in the the
| perceptron is non-linear (it can tell you which side of a plane
| the input point lies), and a multi-layer perceptron could
| classify the red and blue dots in your example. But it cannot
| be used with back-propagation because its derivative is zero
| everywhere, except at f(0), where it's non-differentiable.
| LarsDu88 wrote:
| Thanks for the clarification. I'll update the post!
| LarsDu88 wrote:
| I think I should clarify... A multilayer perceptron can
| classify the red and blue dots if it uses a non-linear
| activation function for some or most of its layers correct?
|
| If its perceptrons all the way down, it will fundamentally
| reduce down to a linear function or single linear layer and
| will not be able to classify the dots.
|
| So there's the downside of not being able to linearly
| separate certain datasets, and the inability to scale weights
| or thresholds by differences in expected and observed data
| (e.g. using backpropagation)
| cventus wrote:
| You're right that if the activation function is linear,
| like the identity function, then it doesn't matter how many
| layers you have. But with the step function two layers is
| enough.
|
| We can manually derive a network that can classify the
| sample data using the step function:
| import numpy as np # Input: [x1, x2]
| # First layer: 4 nodes W1 = np.array([ [
| 1, 0, -0.2], [-1, 0, 0.8], [ 0, 1,
| -0.2], [ 0, -1, 0.8] ]) # Second
| layer: 1 node W2 = np.array([[1, 1, 1, 1, -4]])
| # The step activation function def step(x):
| return x >= 0 # Forward pass def f(x1,
| x2): one = np.ones((1, 1)) v =
| np.array([x1, x2]).reshape((2, 1)) v = step(W1 @
| np.r_[v, one]) v = step(W2 @ np.r_[v, one])
| return v[0, 0] >>> np.array([ ...
| [f(0, 1 ), f(0.5, 1 ), f(1, 1 )], ... [f(0,
| 0.5), f(0.5, 0.5), f(1, 0.5)], ... [f(0, 0 ),
| f(0.5, 0 ), f(1, 0 )] ... ])
| array([[False, False, False], [False, True,
| False], [False, False, False]])
|
| The four nodes in the first layer define four lines,
| tangents to the square 0.2 < x1 < 0.8 and 0.2 < x2 < 0.8,
| and the step function effectively checks which side of the
| line the point lies. The second layer just counts the
| number of "successful" line checks and yields True if all
| four pass. If the square is too rough of a shape then we
| can add more lines to the first layer to approximate any
| convex shape.
|
| If the regions are concave then we can split them up into
| convex parts and add nodes to the second layer, one for
| each convex region. A third layer could then check if any
| of the convex region neurons activate. While in theory two
| layers with a non-linear activation function is enough to
| approximate this function, its structure would be harder to
| interpret.
|
| But how do you find the right parameters without back
| propagation? The reason we don't use the step function is
| because its derivative is zero.
| hnax wrote:
| I switched off at paragraph two:
|
| "Prokaryotes emerged 3.5 billion years ago, their gene networks
| acting like rudimentary brains. These networks controlled
| chemical reactions and cellular processes, laying the foundation
| for complexity."
|
| ... for which there is no evidence at all. Psuedo-science, aka
| Fantasy.
| rdlecler1 wrote:
| I could have bogged the essay down with qualifiers to address
| all the potential straw man objections, but that didn't seem
| productive. It's easy to take an uncharitable view on this, but
| I do explain more about GRNs later in the essay. I worked with
| them for 8 years, and yes, they do act like the rudimentary
| brains of the cell, and that's the reason this system is
| selected again and again by evolution.
| xiaodai wrote:
| can't rule out it was generated by ChatGPT
| macilacilove wrote:
| If there is a 'god equation' it will almost certainly include
| a+b=c because we use it all the time to describe "diverse
| biological systems with vast implementation constraints".
|
| This article is lacking originality and insight to such degree
| that I susupect it is patentable.
| pbd wrote:
| i thought bernoulli's theorem is already the god equation :) .
| No equation is more fundamental this one from thermodynamics.
| inciampati wrote:
| I love your hot take, but you forgot the nonlinear
| transformation which lets the "god equation" represent
| literally everything.
|
| The post makes a nice point but it's not really surprising that
| everything can be modeled by an equation capable of universal
| approximation.
|
| What I don't get is how genetic systems relate to this. They
| don't hook into it cleanly and the author just jumps right past
| them even though they're the most fundamental (biological)
| system of all those described.
| rdlecler1 wrote:
| Genetic systems code gene regulatory networks. I spent most
| of the essay on them.
| lumost wrote:
| The existence of a universal function approximator or function
| representation is not particularly unique to neural networks.
| Fourier transforms can represent any function as a (potentially)
| infinite vector on an orthonormal basis.
|
| What would be particularly interesting is if there were a proof
| that some universal approximators were more parameter efficient
| than others. The simplicity of the neural representation would
| suggest that it may be a particularly useful - if inscrutable
| approximator.
| rdlecler1 wrote:
| I'm not arguing that this approximator is necessary (not
| sufficient) for this class of networks. I've proposed some
| conjectures on what we might expect to see, but there are
| certainly other salient ingredients and common principles that
| we haven't discovered, and I think it's important to hunt for
| them.
| lumost wrote:
| Oh absolutely, the article gave me quite a bit to think
| about. It wasn't until I sat down and tried swapping a
| fourier transform/representation into the conjectures that I
| was able to think critically on the topic.
|
| I suspect that the pruning operation is useful to consider
| mathematically. A fourier transform is a universal
| approximator - but only has useful approximation power when
| the basis vectors have eigenvalues which are significant for
| the problem at hand (PCA). If NN's replace that condition
| with a topological sense of utility. Then that is a major win
| (if formalized).
| Imnimo wrote:
| How does the attention operator in transformers, in which input
| data is multiplied by input data (as opposed other neural network
| operations in which input data is multiplied by model weights)
| fit into the notion of a universal activator?
| rdlecler1 wrote:
| This is a great question, and I don't yet have an answer. I'm
| going to butcher this description, so please be charitable, but
| functionally, the attention mechanism reduces the dimensions
| and uses the coincidence between the Q and K linear layers to
| narrow down to a subset of the input, and then the softmax
| amplifies the signal.
|
| One unsatisfying argument might be that this might fall into
| implementation details for this particular class. Another
| prediction might be that an attention mechanism is an essential
| element of these networks that appears in other networks of
| this class. Another is that this is a decent approximation, but
| has limitations, and we'll figure out how the brain does it and
| replace it with that.
___________________________________________________________________
(page generated 2024-07-15 23:02 UTC)