[HN Gopher] Turing Machines Are Recurrent Neural Networks (1996)
___________________________________________________________________
Turing Machines Are Recurrent Neural Networks (1996)
Author : todsacerdoti
Score : 52 points
Date : 2022-12-05 18:24 UTC (4 hours ago)
(HTM) web link (users.ics.aalto.fi)
(TXT) w3m dump (users.ics.aalto.fi)
| lisper wrote:
| This is a very misleading title. The result here is not that TM's
| _are_ recurrent neural networks, it is that it is possible to
| construct an (infinite) recurrent neural network that emulates a
| TM. But the fact that a TM can be built out of perceptrons is
| neither surprising nor interesting. It 's pretty obvious that you
| can build a NAND gate out of perceptrons, and so _of course_ you
| can build a TM out them.
|
| In fact, it's probably the case that you can build a NAND gate
| (and hence a TM) out of _any_ non-linear transfer function. I 'd
| be surprised if this is not a known result one way or the other.
| whatshisface wrote:
| I don't remember the name of the theorem, but you can
| approximate any nonlinear multivariable function arbitrarily
| with a multi-layer perceptron with any non-polynomial nonlinear
| function, applied after the linear weights and bias. It has to
| be non-polynomial because the set of all polynomials is closed
| under linear combinations, adding a constant, and composition,
| so if the nonlinear function was (say) x^3 you would only get
| polynomials out of the model.
|
| I'm not sure why that's a problem because polynomial
| approximations are still useful.
| jmalicki wrote:
| https://en.wikipedia.org/wiki/Universal_approximation_theore.
| ..
| tsimionescu wrote:
| Note that there are two caveats.
|
| For one, only continuous functions can be represented.
|
| Much more importantly, the theorem doesn't prove that it's
| possible to learn the necessary weights to approximate any
| function, just that such weights much exist.
|
| With our current methods, only a subset of all possible NNs
| are actually trainable, so we can only automate the
| construcion approximations for certain continuous functions
| (generally those that are differentiable, but there may be
| exceptions, I'm not as sure).
| kkylin wrote:
| Are you talking about something like this?
|
| https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr.
| ..
| dang wrote:
| Related:
|
| _Turing Machines Are Recurrent Neural Networks (1996)_ -
| https://news.ycombinator.com/item?id=10930559 - Jan 2016 (12
| comments)
| adlpz wrote:
| As a side note, it's fascinating to read the comments on that
| thread when talking about RNNs and Deep Learning. So much has
| changed in the last 6 years and feels so weird to read
| dismissive comments about the capabilities of these systems
| seeing what people are getting out of ChatGPT.
| BlueTemplar wrote:
| Not really ?
|
| << Inflated >> expectations doesn't mean NO expectations...
|
| People are still throwing << AI >> around as a buzzword like
| it's something distinct from a computer program, in fact the
| situation got worse because non-neural network programs are
| somehow dismissed as << not AI >> now.
|
| Autonomous cars are still nowhere to be seen, even more so
| for << General << AI >> >>.
|
| The singularity isn't much on track either looking at
| Kurzweil's predictions : we should have had molecular
| manufacturers by now, and nanobots connecting our brains to
| the Internet, extending our lives, and brain scanning people
| to recreate their avatars when they are dead, don't seem like
| they are going to happen by the 2030s either. (2045 is still
| far enough away that I wouldn't _completely_ bet against a
| singularity by then.)
|
| (And Kurzweil doesn't get to blame it on misunderstanding the
| exponential function : how people have a too linear view of
| the future and tend to overestimate changes in the near
| future, but underestimate them in the long term !)
| alexheikel wrote:
| New technology need to be created for real AI to exist. As
| we go, we are creating Hal, we aim to pass Turing using ai
| and hi (human intelligence) http://gethal.com
| whatshisface wrote:
| ChatGPT hasn't overcome any of the fundamental issues, it's
| just a huge improvement on the things that the original GPTs
| were good at. Being able to stay coherent for a trained-in
| length that gets longer with larger models is different from
| the length-unlimited coherence that human beings can manage,
| spanning lifetimes of thought and multiple lifetimes of
| discourse.
| dragonwriter wrote:
| > ChatGPT hasn't overcome any of the fundamental issues,
| it's just a huge improvement on the things that the
| original GPTs were good at. Being able to stay coherent for
| a trained-in length that gets longer with larger models is
| different from being internally coherent for 90 years by
| nature like people are.
|
| People are very often not internally coherent over periods
| much shorter than 90 years.
| whatshisface wrote:
| The hazily defined ideal of general intelligence - which
| everyone imagines that they are, but most of us do not
| live up to all the time, and nobody lives up to every
| waking moment (or at least before we've had our morning
| cup of coffee) isn't within the reach of present day
| transformer architectures because the length of text they
| can stay coherent over is limited by how far back they
| can look through their own output. Human beings can form
| new memories from their outputs that stay with them for
| the rest of their lives.
| Xcelerate wrote:
| At what point have our neural networks crossed over to
| demonstrating algorithmic behavior and we no longer consider them
| fancy interpolating functions? Is there a way to quantify this?
| whatshisface wrote:
| A model with no training data would know nothing, so in a sense
| they're always going to be something like a fancy form of
| interpolation/extrapolation.
| Xcelerate wrote:
| There's a difference between interpolation and induction.
| You're not going to interpolate a hash function.
| whatshisface wrote:
| There's no way a neural network could ever learn a hash
| function directly (unless it had every possible input and
| output in its table), and if there was an indirect way to
| train it, you'd discover that it was interpolating between
| (say) possible hash functions by working in a larger space,
| for example if it was trained to generate and test C
| programs that computed hashes.
| jvm___ wrote:
| Does the new chat AI bot pass the turing test?
___________________________________________________________________
(page generated 2022-12-05 23:00 UTC)