[HN Gopher] Neural networks need data to learn, even if it's fake
___________________________________________________________________
Neural networks need data to learn, even if it's fake
Author : nsoonhui
Score : 35 points
Date : 2023-06-17 05:08 UTC (1 days ago)
(HTM) web link (www.quantamagazine.org)
(TXT) w3m dump (www.quantamagazine.org)
| williamtrask wrote:
| Fake isn't the right word here. The synthetic images contain real
| structure useful for learning.
| ilyt wrote:
| Hallucinated then
| sim7c00 wrote:
| agree. fake sounds prone to erroneous data, while some systems
| are fairly easy to simulate, generating real data from a
| simulation. for more complex scenarios this is trickier, and a
| faulty simulation moght leave subtle issues in the networks
| functioning. if u dont have real.data to verify it with u will
| be in the dark about real accuracy. depending on the networks
| purpose, this moght be really dangerous.
| lettergram wrote:
| Lol we actually showed synthetic data often produced better
| models depending what you're trying to do
|
| https://www.capitalone.com/tech/software-engineering/why-you...
|
| Part of the reason was models generating synthetic data hone in
| on more key differentiable features
| orangepurple wrote:
| CMU tried to build a self-learning expert system called NELL but
| it failed. Never-Ending Language Learning system (NELL) is a
| semantic machine learning system developed by a research team at
| Carnegie Mellon University, and supported by grants from DARPA,
| Google, NSF, and CNPq with portions of the system running on a
| supercomputing cluster provided by Yahoo!
|
| In his 2019 book "Human Compatible", Stuart Russell commented
| that 'Unfortunately NELL has confidence in only 3 percent of its
| beliefs and relies on human experts to clean out false or
| meaningless beliefs on a regular basis--such as its beliefs that
| "Nepal is a country also known as United States" and "value is an
| agricultural product that is usually cut into basis."'
| mabbo wrote:
| What's weird about this to me is the flow of information,
| knowledge from one place to another.
|
| We're using software that has encoded the rules we want to use to
| generate data. The ML is using the data to infer the rules. It
| seems like we need a better way to give our ML algorithms the
| _rules_ that we already know rather than this processing-
| intensive process.
|
| But perhaps that's the cost of using neural networks- they can
| learn any function, but they can only learn from data, examples.
| clircle wrote:
| TBH, ordinary least squares can also "learn" any function[1],
| so maybe you are not framing the trade-offs correctly.
| Timon3 wrote:
| I think "giving the rules we already know" is only a good idea
| if we are 100% sure that we know 100% of the rules necessary
| with 100% accuracy. The fuzzyness of NNs is what in many cases
| allows them to work as well as they do, and training on close-
| to-perfect data is going to preserve this aspect better than
| close-to-perfect rules could.
| contravariant wrote:
| With (imho closely related) bayesian statistics you can add
| those rules into your model, the problem is that it quickly
| becomes computationally intractable to do exactly (in a
| calculate this N dimensional integral with N>>1000 kind of
| way).
|
| So in the end you still end up fitting some simplified vaguely
| similar model/function to your data and/or using sampling to
| keep things tractable.
| bckr wrote:
| What you're describing would be an Expert System, part of
| GOFAI.
|
| Neural nets work by generalizing. This is important because
| GOFAI systems are brittle. We are never able to encode all the
| relevant rules for real world situations by hand.
|
| Also, note that the synthetic data is generated from rules
| about how the world looks, not rules about what the model
| should do.
|
| There may still be some direction for zapping the rules-of-the-
| world right into the circuitry of, say, a subnet of the model,
| without doing backprop. Something like a one-shot
| initialization of the network to approximate the given
| distribution.
| moffkalast wrote:
| I think what he's saying is that currently we can get one of
| these expert systems to create 1000 examples with the encoded
| rules, which a NN might then generalize correctly or it may
| come up with a completely batshit insane ruleset that
| overfits on that data. There's no way for us to actually feed
| the NN the rules directly, like one would give them to a
| human brain.
| varjag wrote:
| The main issue with expert systems appeared to be not the
| amount of necessary rules but pathway dependency on
| representation model for the problem domain. If it mapped
| well, it could perform exceptionally well. If not, you got
| into all sort of combinatory troubles real quick and had to
| wing it (or as they used to say in the 1980s, to use
| heuristic rules).
|
| The combinatory bit due to its branching nature lends itself
| poorly to leveraging SIMD hardware as statistical solutions
| do. And getting a right representation model is a substantial
| challenge. Everyone knew the importance of that ('knowledge
| engineering') by the onset of AI Winter but nobody had any
| good methods.
| thrashh wrote:
| But even us as humans (or anything with biological
| intelligence) infers the rules from data.
|
| The only thing is -- we start collecting data since birth and
| we also don't have to pay for every little thing we put in
| front of our eyes.
|
| If you're training an AI, you have to pay for data. If you
| generate some of your data, you pay a little less.
|
| (Also I think we ask of AI more than we ask of humans. For
| example, it's common to be face blind of cultures you're not
| familiar with but we get pissed if an AI is face blind at all.)
| dimatura wrote:
| I think that's an insightful point. Personally, I believe an
| important part of ML architecture engineering is figuring out
| architectures that make giving "the wrong answer" to a problem
| impossible, or at least difficult. But sometimes just throwing
| a bunch of data at a generic architecture and hoping it can
| learn the "rules" is easier.
|
| And of course, as alluded in the article, sometimes we don't
| really know the "rules", or don't know how to articulate them
| in a way we can feasibly synthesize data according to those
| rules. Then people might reach for ML to figure out those
| rules, implicitly or explicitly. So it becomes a bit of an
| ouroboros situation. But still somewhere in that loop there's
| some domain knowledge injected somewhere by whoever is
| engineering the pipeline.
| Valectar wrote:
| We don't have the rules that we want the neural network to
| learn, if we did we could just directly use those rules, and
| there would be no need for ML to solve the problem. In this
| case we want the ML to learn how to infer spatial information
| from two dimensional images, and the process that generates the
| data it trains on cannot do this at all. It can create two
| dimensional images from spatial information, which is a much
| simpler and effectively solved process.
|
| There are cases when we want a machine learning model to do the
| same thing as the process which generate it's data, like in the
| case of model's learning to replicate physics simulations, but
| even then the entire point is for the machine learning model to
| accomplish the same or a similar result but in a more
| computationally efficient way.
| carlmr wrote:
| Sounds like we've invented dreaming now.
___________________________________________________________________
(page generated 2023-06-18 23:01 UTC)