[HN Gopher] Teaching a Bayesian spam to filter play chess (2005)
___________________________________________________________________
Teaching a Bayesian spam to filter play chess (2005)
Author : ignoranceprior
Score : 34 points
Date : 2021-01-30 16:41 UTC (6 hours ago)
(HTM) web link (dbacl.sourceforge.net)
(TXT) w3m dump (dbacl.sourceforge.net)
| zackmorris wrote:
| I realized 20 years ago after learning about genetic algorithms
| (GAs) that all AI and machine learning approaches can be used
| interchangeably to solve any problem, and that the fundamental
| approach to problem-solving is evolution itself. The differences
| only arise when we examine various models from a compute-bound
| perspective because we ruled out the O(n^2+) embarrassingly
| simple/parallel algorithms that actually work.
|
| Unfortunately genetic algorithms can't run on video cards
| efficiently (because shaders are SIMD instead of MIMD), so IMHO
| most of the "progress" in computing power the last 20 years, even
| all the way up to stuff as advanced as TensorFlow on GPUs, has
| mostly been hand waving.
|
| The problem is that shaders run the same code with different data
| (that's the SIMD part), but GAs run different code on each
| CPU/thread/actor instance (which is MIMD or even MISD).
|
| But, maybe there are ways. This looks promising:
|
| https://people.scs.carleton.ca/~dmckenne/5704/Papers/14.pdf
|
| This is the canonical article about the heart of the problem that
| I re-share periodically:
|
| http://aggregate.org/MOG/
|
| And unbelievably, it never occurred to me why MISD is so
| important and overlooked, but this is about the closest thing to
| a eureka moment that I've had in a long time:
|
| https://en.wikipedia.org/wiki/MISD
|
| A "theory" about why progress in AI is so sluggish is that we
| can't scale Docker containers to 1000+ on our own computers with
| video cards. Until we can do that, progress is effectively at a
| standstill. And that visceral reaction of "oh that's a terrible
| idea for XYZ reason" represents the real work that needs to be
| done to advance computing power. It also captures the sense of
| why computers today feel like they are running many thousands of
| times slower than they should be for the number of transistors
| they have.
|
| TL;DR: I'm saying that with this computer, you could write a
| short shell script that uses a Bayesian spam filter (or anything
| else) and it would compete with the best algorithms we have by
| sheer brute force. But more importantly, we could iterate on our
| solutions faster (because they're so much simpler than neural
| nets so more people could be involved) and quickly discover
| emergent behavior that guides us towards a formula for artificial
| general intelligence (AGI). There's no mystery here, to me, it's
| just plug and chug and thinking about the heart of the problem
| from a different perspective.
| lisper wrote:
| > the fundamental approach to problem-solving is evolution
| itself
|
| Not quite. The fundamental approach to problem solving is
| search, and evolution is a kind of search. It's not a
| particularly efficient search algorithm, but it has the
| interesting property that it is able to invent new search
| algorithms that are more efficient than it is. We have a pretty
| good handle on search. What we do not yet have is the foggiest
| clue about this sort of "meta-search", the search for new kinds
| of searching mechanisms. We've invented/discovered one: the
| Turing machine, and everything we've done since then has been a
| riff off that one idea. Even so called "neural networks" are
| really just a fairly uninteresting algorithm that happens to
| bear a superficial resemblance to our brains and happens to
| produce some interesting results. But, as you very correctly
| observe, they are nowhere near the Right Answer.
|
| UPDATE: OK, quantum computers probably count as a mechanism
| distinct from TMs. But I'll give long odds against QC turning
| out to be the key to understanding how our brains work, Roger
| Penrose notwithstanding.
| zackmorris wrote:
| Ya I was just thinking that the simplest learning algorithm
| would actually be something simpler like "solving systems of
| equations as a matrix" (I'm feeling lucky Google result):
|
| https://www.mathsisfun.com/algebra/systems-linear-
| equations-...
|
| So you enter your inputs and outputs in a big table and solve
| by brute force to get an algorithm. It might not be an
| interesting one, or one useful beyond that one problem, but
| it's something.
|
| There are also some graphical techniques for truth tables
| (boolean-value matrices):
|
| https://www.allaboutcircuits.com/textbook/digital/chpt-8/log.
| ..
|
| And this is kinda sorta loosely related to classification
| algorithms like k-means clustering, at least that's how my
| brain associates them:
|
| https://en.wikipedia.org/wiki/K-means_clustering
|
| Anyway, I view neural networks as a way of solving a huge
| matrix with gradient descent (hill climbing) as the
| optimization technique. It fundamentally will always struggle
| with the local minimum (maximum) problem just like in
| calculus.
|
| Whereas genetic algorithms bust out of that mental trap via
| random sampling. I'm sure there's a proof out there somewhere
| that says they can solve any problem, given enough time,
| which is why I view them as the basis of learning. As I write
| this with my neural net hmmm..
|
| To me, the only way forward is at the meta level, quantum
| computing or similar (as you very astutely pointed out).
|
| An analogy for this might be that the vast majority of us do
| our daily work on the imperative side of programming (Turing
| machines) without realizing that they can be made equivalent
| to the vastly simpler functional side of programming (Lambda
| calculus) that solved most of the problems we face today 50+
| years ago.
|
| The state of the art when I first starting following this in
| the late 90s was optimizing neural nets evolved via genetic
| algorithms. I never heard anything else about it, but keep in
| mind that shortly after that, HP bought Compaq, private
| industry research spending cratered, native mobile app
| programming buried the innovation of declarative and data-
| driven programming that the web popularized, single page apps
| made web development untenable, video cards ended CPU
| innovation, and wealth inequality directed us all into
| nursing CRUD apps back from the brink of death instead of
| making contributions to open source projects that could
| fundamentally advance the human condition.
|
| The problems are so widespread and entrenched now that if we
| can even break past a few of them, then I see rapid and
| transformative change potentially happening very quickly.
| It's like how the forces that be work to make sure that
| nobody gets UBI, because we can't have the plebeians enjoying
| the same return on investment that the wealthy take for
| granted. No, everyone must be kept poor and struggling or
| else they might just find the time to automate themselves out
| of a job.
|
| Edit: sorry I just realized that I contradicted you on
| search. I was trying to make the point that if you could fit
| the whole search space into a matrix, then we could just
| solve it. Since that's not possible for most real-world
| problems, then yes I agree with you that search is actually
| the fundamental problem-solving algorithm.
| jakear wrote:
| > The fundamental approach to problem solving is search, and
| evolution is a kind of search
|
| Ehh, problem-solving _is_ search (searching for an answer),
| so it this statement doesn't really make sense except as a
| tautology. I agree that evolution is a core means by which
| problems are solved, and perhaps _the_ core means.
|
| Humans are incredibly good searchers because of the evolution
| we've undergone, to the extent we've started evolving our own
| searchers to enhance our problem solving ability. I'm not
| sure if we can reasonably say that those searchers have
| started to evolve their own searchers, but it's safe to say
| it will happen soon.
|
| The big question to me is what beings evolved us (the
| universe/whatever the embedding we call reality really is),
| and what problem were we meant to solve? This is basically
| unknowable as far as I can tell, and the foundation of all
| "religion".
| lisper wrote:
| Not all problem solving is search and not all search is
| problem solving. Some problems can be solved
| algorithmically with no search (e.g. computing a square
| root) and search can obviously be done in service of
| something other than problem-solving (unless you take the
| meaning of "problem" to be so broad as to apply to just
| about any activity).
|
| I also intended the word "search" to be understood in a
| narrow technical sense:
|
| https://en.wikipedia.org/wiki/Search_algorithm
|
| [EDIT] Just to clarify, because I just realized that these
| two statements might seem contradictory:
|
| "The fundamental approach to problem solving is search"
|
| "Not all problem solving is search"
|
| All problem solving _can_ be cast as search, but some
| problems have heuristics that allow the correct branch of
| the search space to be chosen the first at every decision
| point with no backtracking required. The "search" becomes
| degenerate because no backtracking is ever required. The
| word "search" is generally reserved to apply only to
| algorithms that don't have such strong heuristics and do
| require backtracking.
| jakear wrote:
| I fear this is just a debate over semantics, which is
| always an unhappy debate. In my opinion "finding the
| square root" isn't a "problem", perhaps precisely because
| it doesn't involve search. Something like "how can I
| determine the side length of this square I needed 67 sq
| feet of tile to cover" is a problem: I could use your
| algorithm on paper, I could guess based on prior
| experience, I could measure, I could estimate, I could
| use a calculator, etc. Which of those I pick is a search
| based on the constraints at hand (how much time do I
| have, what resources do I have, how precise an answer do
| I need, etc), and the act of considering many approaches
| to the problem and proceeding with some based on the
| constraints at hand is evolution.
|
| Again, this is all semantics imo, so not really a debate
| so much as sharing world views. Which perhaps is what
| debate is. Hm.
| lisper wrote:
| > "finding the square root" isn't a "problem", perhaps
| precisely because it doesn't involve search
|
| Well, it _can_ be solved by search (as I 've defined it).
| It just happens that it doesn't _have_ to be.
| zackmorris wrote:
| Oh man, reading what you wrote out, it just occurred to
| me that learning is actually caching.
|
| We already have a multitude of machines that can solve
| any problem: the global economy, corporations, capitalism
| (Darwinian evolution casted as an economic model),
| organizations, our brains, etc.
|
| So take an existing model that works, convert it to code
| made up of the business logic and tests that we write
| every day, and start replacing the manual portions with
| algorithms (automate them). The "work" of learning to
| solve a problem is the inverse of the solution being
| taught. But once you know the solution, cache it and use
| it.
|
| I'm curious what the smallest fully automated model would
| look like. We can imagine a corporation where everyone
| has been replaced by a virtual agent running in code. Or
| a car where the driver is replaced by chips or (gasp) the
| cloud.
|
| But how about a program running on a source code repo
| that can incorporate new code as long as all of its
| current unit tests pass. At first, people around the
| world would write the code. But eventually, more and more
| of the subrepos would be cached copies of other working
| solutions. Basically just keep doing that until it passes
| the Turing test (which I realize is just passe by today's
| standards, look at online political debate with troll
| bots). We know that the compressed solution should be
| smaller than the 6 billion base pairs of DNA. It just
| doesn't seem like that hard of a problem. Except I guess
| it is:
|
| https://github.com/opencog/opencog
| zackmorris wrote:
| In my own life, I've come to view the search for meaning as
| the fundamental goal of consciousness. My good friend calls
| this the "reconnection" and I've often wondered if
| connectedness could be used as the primary incentive for a
| general machine learning algorithm.
| [deleted]
| KingOfCoders wrote:
| Ahead of its time.
| uniformlyrandom wrote:
| That was fun to read, thank you. There are obvious limitation to
| the approach, as stated by the author, but it was an interesting
| experiment.
| djmips wrote:
| Title needs an edit.
| mgsouth wrote:
| To wit, "to filter" -> "filter to".
| ignoranceprior wrote:
| Yeah, sorry, I can't edit it.
___________________________________________________________________
(page generated 2021-01-30 23:01 UTC)