[HN Gopher] Human-level few-shot concept induction through minim...
___________________________________________________________________
Human-level few-shot concept induction through minimax entropy
learning
Author : geox
Score : 83 points
Date : 2024-04-19 23:54 UTC (23 hours ago)
(HTM) web link (www.science.org)
(TXT) w3m dump (www.science.org)
| janpmz wrote:
| If I understood this correctly:
|
| Minimizing entropy: Learn filters that can increase the
| likelihood of the observed data.
|
| Maximizing entropy: Learning how to combine the filters.
| 3abiton wrote:
| Is this reasoning from the paper?
| versteegen wrote:
| In original FRAME, "learning filters" means selecting filters
| from a (very small) fixed set. In this paper they add* an
| additional "bilevel optimization", where they both learn th_j,
| the parameters of the filters, and select the filters
| (indicator variables z_j).
|
| Either way, that learning is the entropy minimisation part.
|
| Maximizing entropy is not really about learning anything, it's
| about not overfitting. (Although you are correct in that you
| have to find the Langrange multipliers to correctly combine the
| filters.) Basically, the desired probability distribution will
| be constrained only by certain statistics of (the filters you
| learnt) on the data, and nothing else. For example, you want a
| model that says "if panel 1 has an N-sided polygon, and panel 2
| has an N+1-sided polygon, panel 3 has an N+2-sided polygon",
| and says nothing about anything else.
|
| * I'm a bit peeved they didn't cite me for extending FRAME by
| learning the features like that. That was my PhD topic. But
| let's face it, they probably don't know about my work, or
| quickly forgot it because the papers were too ugly, and
| admittedly they've done it in a cleaner way. I had to wrap
| kludgy iterative feature addition around the feature
| optimisation step to make it more efficient for image
| modelling.
| shiandow wrote:
| It's perhaps useful to note that their notation is not
| accidental.
|
| When they state: P = 1/Z exp -li Hi
|
| They're directly referring to the Gibbs distribution that pops up
| in thermodynamics. The Hi are the different terms of the
| Hamiltonian and the Z is the corresponding partition function.
|
| From that point of view they're taking a couple of 'response
| functions' treating them as if they are part of some kind of
| energy and find the statistical distribution that such a system
| would have at different temperatures depending on the average
| value of each term. The next part is then to find the value for
| each term that best predicts the observed distribution.
|
| It's a bit like observing how a system behaves for different
| volumes and then using that information to determine the
| temerature, pressure and volume of an observed system. (perhaps
| volume isn't the best example, but something like molarity and
| chemical potential is hard to visualize)
| versteegen wrote:
| Yes, but...
|
| > some kind of energy
|
| The negenergy (negative energy) here is just the (unnormalised,
| hence the 1/Z term) log-probability mass or density. Like the
| logit outputs of classifier.
|
| You shouldn't read anything more into the 'energy'. There is no
| temperature parameter in these minimax models (although adding
| one to smooth the model during sampling or learning can be
| useful, to interpolate between pure noise and the learnt
| model).
| davedx wrote:
| This strikes me as potentially a high impact avenue of research.
| Inductive reasoning is core to human learning.
| Hugsun wrote:
| I can't wait to see the next innovation, that is orthogonal to
| the scaling laws, that will cause the next leap in LLM
| performance.
| cs702 wrote:
| Interesting. There's a lot of preprocessing involved in getting
| these toy models to work, but the approach seems promising.
|
| Is there code available, so others can examine it and replicate
| the results? I couldn't find a link.
| versteegen wrote:
| It's surreal to see this on the front page since I worked on
| minimax entropy learning for texture modelling (following these
| authors), then abandoned that line of research because I wanted
| to work on concept induction and reasoning instead. I'm amazed
| these authors are still drawing a direct line through 27 years of
| their papers.
|
| To explain what this is about (not having read the whole paper):
| it's using 'classical', small sample size statistics to infer
| relations between very small sets of data such as individual
| questions of an IQ test. Obviously that's going to give the best
| possible answer, because that's what classical statistics does.
| (Well, if you could solve exactly, but with this model you
| actually can't.) Then it's just a question of having a good prior
| over possible concepts. In this paper the concepts are hand-
| defined parametrized features (aka 'filters' in the paper, from
| the texture modelling lineage), e.g. "families of arithmetic
| expressions of length from two to six" for the number problems.
| There will be a _lot_ of details involved in that.
|
| Alternatively, something like a deep neural network trained on
| vast numbers of generated examples could come to learn what a
| suitable prior should be by learning the patterns used to
| generate them. Yet another option is to use a universal prior as
| in Solomonoff induction, based on representation/coding length.
|
| My conclusion: the theory is good and I'm happy to see this but
| don't think that you can directly apply this to learning
| 'concepts' in anything other than IQ tests. The hard part of
| generalising this is firstly finding the prior over the concepts,
| and secondly the difficult optimisation step: using MCMC as part
| of the optimisation is both extremely slow and stochastic in a
| bad way (unlike SGD) and the reason I abandoned this approach.
| p1esk wrote:
| _deep neural network trained on vast numbers of generated
| examples could come to learn what a suitable prior should be by
| learning the patterns used to generate them._
|
| This sounds promising. Do you see any potential issues with
| this approach?
| uoaei wrote:
| This is how NN theory describes the process of training,
| i.e., that's the mechanism by which all neural networks come
| to represent the problem they are being applied to solve. GP
| is using a fancy way to say "train NNs on massive datasets"
| which is par for the course.
| eli_gottlieb wrote:
| So they minimize the forward KL divergence from the generative
| model to the data distribution, and then maximize the entropy of
| the generative model? Seems like it would be simpler to just
| minimize the reverse/exclusive KL divergence, giving you a cross-
| entropy term to minimize and then an entropy of the generative
| model to maximize.
|
| I guess the obstacle to doing that is not being able to evaluate
| the true data log-density, only sample from it, so the cross-
| entropy term can't be estimated?
___________________________________________________________________
(page generated 2024-04-20 23:02 UTC)