[HN Gopher] AI-Generated Data Can Poison Future AI Models
___________________________________________________________________
AI-Generated Data Can Poison Future AI Models
Author : meany
Score : 110 points
Date : 2024-03-09 15:34 UTC (7 hours ago)
(HTM) web link (www.scientificamerican.com)
(TXT) w3m dump (www.scientificamerican.com)
| Der_Einzige wrote:
| Certain words, like "groundbreaking", have been totally ruined
| for me by LLMs which are too often trained to sound like each
| other.
| hermitcrab wrote:
| See also: https://news.ycombinator.com/item?id=39422528
| buo wrote:
| I think it's interesting that human minds generally (though not
| always!) improve when exposed to the output of other human minds.
| It seems to be the opposite for current LLMs.
| diggan wrote:
| Maybe it's less about "Human VS Robot" and more about exposure
| to "Original thoughts VS mass-produced average thoughts".
|
| I don't think a human mind would be improving if they're in a
| echo-chamber with no new information. I think the reason the
| human mind is improving is because we're exposed to new,
| original and/or different thoughts, that we hadn't considered
| or come across before.
|
| Meanwhile, a LLM will just regurgitate the most likely token
| based on the previous one, so there isn't any originality
| there, hence any output from a LLM cannot improve another LLM.
| There is nothing new to be learned, basically.
| bluefirebrand wrote:
| > I don't think a human mind would be improving if they're in
| a echo-chamber with no new information
|
| If this were true of humans, we would have never made it this
| far
|
| Humans are very capable of looking around themselves and
| thinking "I can do better than this", and then trying to come
| up with ways how
|
| LLMs are not
| diggan wrote:
| > Humans are very capable of looking around themselves and
| thinking "I can do better than this"
|
| Doesn't this require at least some perspective of what
| "better than this" means, which you could only know with at
| least a bit of outside influence in one way or another?
| esafak wrote:
| Parsimony, explanatory power, and aesthetics. These are
| things that could be taught to a computer, and I think we
| will. We had to evolve them too.
| throwaway74432 wrote:
| Different loss function
| KolmogorovComp wrote:
| A more appropriate analogy would be isolating someone from the
| rest of the world and only being able to read their own
| writings from now on.
|
| While some persons can strive in these kind of environment
| (think Kant for example), many would become crazy.
| ausbah wrote:
| humans haven't been had the same set of all encompassing
| "training experiences" like LLMs have. we each a subset of
| knowledge that may overlap with some other's knowledge, but is
| largely unique. so when we interact with each other we can
| learn new things, but with LLMs I imagine it is a group of
| experienced but antiquated professors developing their own set
| of out of touch ideas
| NortySpock wrote:
| I do get to choose what I read, though.
| mewpmewp2 wrote:
| Have you ever heard of the telephone game? This is what is
| going on here. Or imagine an original story of something that
| really happened. If it goes by 100 people in a chain, how much
| do you think the story will resemble the original one?
| BobaFloutist wrote:
| I mean it makes sense that (even impressively functional)
| statistical approximations would degrade when recursed.
|
| If anything I think this just demonstrates yet again that these
| aren't actually analogous to what humans think of as "minds",
| even if they're able to replicate more of the output than makes
| us comfortable.
| orbital-decay wrote:
| Humans exhibit very similar behavior. Prolonged sensory
| deprivation can drive a single individual insane. Fully
| isolated/monolithic/connected communities easily become
| detached from reality and are susceptible to mass psychosis.
| Etc etc etc. Humans need some minimum amount of external data
| to keep them in check as well.
| ben_w wrote:
| Reproductive analogy:
|
| A sequence of AI models trained on each other's output gets
| mutations, which might help or hurt, but if there's one
| dominant model at any given time then it's like asexual
| reproduction with only living descendant in each generation
| (and all the competing models being failures to reproduce). A
| photocopy of a photocopy of a photocopy -- this seems to me to
| also be the incorrect model which Intelligent Design proponents
| seem to mistakenly think is how evolution is supposed to work.
|
| A huge number of competing models that never rise to dominance
| would be more like plants spreading pollen in the wind.
|
| A huge number of AI there are each smart enough to decide what
| to include in its training set would be more like animal
| reproduction. The fittest memes survive.
|
| Memetic mode collapses still happen in individual AI (they
| still happen in humans, we're not magic), but that manifests as
| certain AI ceasing to be useful and others replacing them
| economically.
|
| A few mega-minds is a memetic monoculture, fragile in all the
| same ways as a biological monoculture.
| nonrandomstring wrote:
| A different biological analogy occurred to me which I've
| mentioned before in a security context. It isn't model
| degeneration but the amplification of invisible nasties that
| don't become a problem until way down the line.
|
| Natural examples are prions such as Bovine spongiform
| encephalopathy [0] or sheep scrapie. This seems to really
| become a problem in systems with a strong and fast positive
| feedback loop with some selector. In the case of cattle it
| was feeding rendered bonemeal from dead cattle back to
| livestock. Prions are immune to high temperature removal so
| are selected for and concentrated by the feedback process.
|
| To really feel the horror of this, read Ken Thompson's
| "Reflections on Trusting Trust" [1] and ponder the ways that
| a trojan can be replicated iteratively (like a worm) but
| undetectably.
|
| It isn't loss functions we should worry about. It's gain
| functions.
|
| [0] https://en.wikipedia.org/wiki/Bovine_spongiform_encephalo
| pat...
|
| [1] https://tebibyte.media/blog/reflections-on-trusting-
| trust/
| analog31 wrote:
| This might be my biases speaking, but I have a hunch that
| there's still more potential for human generated content to
| poison our minds, than AI.
| GaggiX wrote:
| Unless the internet is no longer useful because there is no way
| to find anything reliable, there would be enough signal to train
| and align models.
| Iulioh wrote:
| Dead internet theory is closer and closer
|
| I don't remember wich YouTuber made a interesting video about
| it but basically communities are moving away from the free web
| in private communities (think discord or even sites that you
| are forced to register to to read the content)
|
| It's an interesting thing but I think queries on searche
| engines are becoming worse for this reason too.
| hackerlight wrote:
| I question whether it'll matter. There is so much language data
| already, unlocking a little more isn't going to be the
| difference maker for AGI.
| sophrocyne wrote:
| Some perspectives from someone working in the image space.
|
| These tests don't feel practical - That is, they seem intended to
| collapse the model, not demonstrate "in the wild" performance.
|
| The assumption is that all content is black or white - AI or not
| AI - and that you treat all content as equally worth retraining
| on.
|
| It offers no room for assumptions around data augmentation,
| human-guided quality discrimination, or anything else that might
| alter the set of outputs to mitigate the "poison"
| data-ottawa wrote:
| > Use the model to generate some AI output. Then use that
| output to train a new instance of the model and use the
| resulting output to train a third version, and so forth. With
| each iteration, errors build atop one another. The 10th model,
| prompted to write about historical English architecture, spews
| out gibberish about jackrabbits.
|
| That this happens doesn't surprise me, but I'd love to see a
| curve of how each organic vs machine content mixe ratio results
| in model collapse over N generations.
| MacsHeadroom wrote:
| This is exactly right. Model collapse does not exist in
| practice. In fact, LLMs trained on newer web scrapes have
| increased capabilities thanks to the generated output in their
| training data.
|
| For example, "base" pretrained models trained on scrapes which
| include generated outputs can 0-shot instruction follow and
| score higher on reasoning benchmarks.
|
| Intentionally produced synthetic training data takes this a
| step further. For SoTA LLMs the majority of, or all of, their
| training data is generated. Phi-2 and Claude 3 for example.
| pavel_lishin wrote:
| What happens if you train a model on nothing _but_ AI-
| generated output, recursively? Does it eventually get inbred?
| Kuinox wrote:
| Without human input, yes.
| visarga wrote:
| Why would you limit a model to be like a brain in a vat?
| Instead let the model out so people use it, then use the
| chat logs to fine-tune. A chat room is a kind of
| environment, there is a human, maybe some tools. The LLM
| text will generate feedback and right there is a learning
| signal.
|
| Even without a human, if a LLM has access to code execution
| it can practice solving coding tasks with runtime feedback.
| There are many ways a LLM could obtain useful learning
| signals. After all, we got all our knowledge from the
| environment as well, in the end there is no other source
| for knowledge and skills.
| rdedev wrote:
| Claude 3 does use publically available data. Not everything
| is synthetically generated. Look at the section for training
| data in the below link. It has an quote from the paper which
| states that it uses a mix of public data, data from labelers
| and synthetic data
|
| https://www.lesswrong.com/posts/JbE7KynwshwkXPJAJ/anthropic-.
| ..
|
| I can't find a link to the actual clause paper to verify the
| above link but a few other places mention the same thing
| about the training data. We don't know if this improved
| performance is because of synthetic data or something else.
| I'm guessing even antropic might not be knowing this too.
| Bjorkbat wrote:
| Ironically Claude 3 appears to have certain "quirks" arguably
| caused by the fact that its training data contains synthetic
| data. In one instance (https://twitter.com/DimitrisPapail/sta
| tus/176477229891207585...), it kept referring to itself as
| ChatGPT.
|
| Granted, one could argue that this only happened because the
| API version of Claude doesn't appear to use a system prompt.
| If that's the case, then the LLM lacks any identity otherwise
| defined by the initial system prompt, and thus, kind of makes
| one up.
|
| Nonetheless, point remains, it's kind of interesting to see
| that in the years since the launch of ChatGPT we're already
| seeing a tangible impact on publicly available training data.
| LLMs "know" what ChatGPT is, and may even claim to be it.
| catchnear4321 wrote:
| that is the meat the article tries to cook. the impacts so
| far aren't all that negative.
|
| but time flows like a river, and the more shit that gets
| into it...
|
| poison does not need to be immediately fatal to be fatal.
| some take a frighteningly long time to work. by the time
| you know what's happening, not only is it too late, you
| have already suffered too much.
|
| does this sound like anything more than a scary story to
| tell around campfires? not yet.
| coffeebeqn wrote:
| Wouldn't reinforcement learning just weigh any nonsense data
| very low and then spammy garbage doesn't really affect the
| model in the end much ? If the model and human experts can't
| tell the difference then it's probably pretty good AI
| generated data
| catchnear4321 wrote:
| the ideal poison tastes like nothing, or at the very least
| doesn't taste bad.
|
| you wouldn't want to alert the victim.
| __loam wrote:
| Truth and what humans think is true are different things.
| Synthetic data was created by models that were trained to
| be convincing.
| wredue wrote:
| >model collapse does not exist in practice
|
| Dude what? That's a pretty absurd claim. Most generally
| available models specifically curate their inputs for the
| express purpose of avoiding AI garbage induced collapse. It's
| literally on their cited reasons for avoiding ai generated
| data as inputs.
| jtriangle wrote:
| As someone also working in the imaging space, ai generated data
| is useful solong as it's used carefully.
|
| Specifically, we're implementing AI culled training sets which
| contain some generated data that then gets reviewed manually
| for a few specific things, then pushed into our normal training
| workflows. This makes for a huge speedup versus 100% manual
| culling and the metrics don't lie, the models continue to
| improve steadily.
|
| There may be a point where they're poisoned and will collapse,
| but I haven't seen it yet.
| Aerroon wrote:
| > _human-guided quality discrimination_
|
| This is the part that I don't really understand. Isn't this
| basically an evolutionary algorithm, where the fitness function
| is "whatever people like the most" (or at least enough to post
| it online)?
|
| People rarely generate 10 pieces of content with AI and then
| share all 10 with the world. They usually only share the best
| ones. This naturally filters for better output.
|
| Are they saying that evolutionary algorithms don't work?
| ipython wrote:
| This reminds me of how fascinated I was as a kid of the artifacts
| you get from recursively photocopying a piece of paper.
| sshine wrote:
| I watched someone in the printer room at the computer science
| department gradually photocopy from white to black, and back
| again, over the span of 300 pieces of paper, by altering the
| thresholds of the photocopyer.
|
| They didn't graduate to become computer scientists, but did
| indeed get admitted to the royal school of art the year after.
|
| I found it strangely therapeutic.
| doubloon wrote:
| reminds me of sheep and cows being fed their bretherens own brain
| matter developing spongiform encepalopathy (brain disease) or of
| course cannibals developing kuru. except a purely 'software'
| form.
| chmike wrote:
| And human generated data may not ?
| jxdxbx wrote:
| How does this relate to synthetic data?
| add-sub-mul-div wrote:
| You'd think we'd be concerned about it poisoning the culture,
| well before any concerns that it would start to interfere with
| the rich continuing to be able to profit from it doing so.
| cortesoft wrote:
| Human created content is also filled with gibberish and false
| information and random noise... how is AI generated content
| worse?
| Libcat99 wrote:
| Imagine you have a calculator that outputs a result that is off
| by one percent. That's ai right now.
|
| If you use the results of each calculation in additional
| calculations, the result will skew further and further from
| reality with each error. That's ai training on itself.
| richk449 wrote:
| In many areas of communication and information, this exact
| problem is dealt with through error correction codes. Do AI
| models have built in ECC?
| Libcat99 wrote:
| The trouble is "truth" and math are different.
|
| You can verify a mathematical result. You can run the
| calculations a second time on a separate calculator (in
| fact some computers do this) to verify the result, or use a
| built in check like ecc.
|
| There's no such mathematical test for truth for an ai to
| run.
| ben_w wrote:
| There's no _fully general_ test for truth for an AI to
| run.
|
| In some specific domains such tests exist -- and the
| result is, generally, computers wildly outperforming
| humans. But I get the impression from using them that
| current LLMs didn't take full advantage of this during
| training.
| richk449 wrote:
| Error correction doesn't insure truth. At least in
| communication, it insures that the final version matches
| the original version.
|
| For AI, you wouldn't be doing EC to make sure the AI was
| saying truth, you would be doing EC to ensure that the AI
| hasn't drifted due to the 1% error rate.
|
| Of course I have no idea how to actually do it - if it
| isn't being done now, it is probably hard or impossible.
| nyrikki wrote:
| No, LLMs with soft attention use compression, and actually
| has no mechanism for ground truth.
|
| They are simply pattern finding and matching.
|
| More correctly, they are uniform consent depth threshold
| circuits.
|
| Basically parallel operations on a polynomial number of
| AND, OR, NOT, and majority gates.
|
| The majority gates can do the Parity function, but cannot
| self correct like ECC does.
|
| The thing with majority gates is that they can show some
| input is in the language:
|
| This the truthiness of 1,1,1,0,0 being true, but 1,1,0,0,0
| would be failure as negation, but doesn't prove that
| negation, it isn't a truthy false.
|
| With soft attention will majority gates they can do parity
| detection but not correction.
|
| Hopefully someone can correct this if I am wrong.
|
| Specifically I think that the upper bound of deciding
| whether X = x is a cause of m) in structures is NP-complete
| in binary models (where all variables can take on only two
| values) and S_2^P -complete in general models.
|
| As TC_0 is smaller than NP, and probably smaller than P,
| any methods would be opportunistic at best.
|
| Preserving the long tail of a distribution is a far more
| pragmatic direction as an ECC type ability is unreasonable.
|
| Thinking of correctional codes as serial turing machine and
| transformers as primarily parallel circuits should help
| with understanding why they are very different.
| feoren wrote:
| Arsenic naturally occurs... how are automatic factories that
| dump millions of tons it in the nearby river worse?
| heresie-dabord wrote:
| > how is AI generated content worse?
|
| This is a crucial question.
|
| In human society, a feedback loop of nonsense is usually
| defeated by practical effects in physical reality and
| experience. The objective of education, for example, is to
| transmit knowledge and apply reason to important questions.
|
| In _manipulated social media_ , there is no check on the
| nonsense loop. The technology that we currently call A.I. could
| be used for educational good.
|
| How it _will_ be used, however, is likely to further distort
| discourse and generate nonsense.
| ejb999 wrote:
| It is worse, because it is faster - how many incorrect blog
| articles can a sigle typical writer publish and post on the
| internet - maybe 1-2 a day if you are a prolific writer?
|
| How many can an AI agent do? Probably hundreds of thousands a
| day. To me, that is going to be a huge problem - but don't have
| a solution in mind either.
|
| And then those 100K bad articles posted per day by one person,
| are used as training data for the next 100K bad/incorrect
| articles etc - and the problem explodes geometrically.
| richk449 wrote:
| Kessler syndrome for the internet?
| ur-whale wrote:
| > AI-Generated Data Can Poison Future AI Models
|
| Looks like we didn't learn anything from the mad cow disease!
| ein0p wrote:
| I also wonder what search engines are going to do about all this.
| Sounds to me, actually, traditional, non-intelligent search might
| be on its way out, although of course it'll take time. Future
| search engines will have to be quite adept at trying to figure
| out whether the text they index is bullshit or not.
| beeboobaa wrote:
| It shouldn't be a problem if you only train on legally acquired
| data. You will know the authors name and can contact them if you
| so wish.
| theferalrobot wrote:
| I don't think any of the major players could do that for all
| their data and they are acquiring it legally.
| coldcode wrote:
| I think AI-generated images are worse for training AI generative
| models than LLMs, since there are so many now on the internet
| (see Instagram art related hashtags if you want to see nothing
| but AI art) compared to the quantity of images downloaded prior
| to 2021 (for those AI that did that). Text will always be more
| varied than seeing 10m versions of the same ideas that people
| make for fun. AI text can also be partial (like AI-assisted
| writing) but the images will all be essentially 100% generated.
| ToucanLoucan wrote:
| That's far from unique to instagram. I loathe Stable Diffiusion
| and co solely because they've utterly FLOODED every cool art-
| adjacent website with endless mediocre derivative shit. Like
| there was always low-effort content of course, but holy fuck,
| there is SO MUCH MORE now. And some of these people are trying
| to CHARGE for this uninspired junk!!!
| 7moritz7 wrote:
| I agree with this despite using SD a lot myself. It's fun to
| use until you realize the majority of people posting stuff
| generated with it have almost no creativity, all generating
| the same things over and over again, mostly without any
| manual work involved. that uncanny realism style with the
| generic Stable Diffusion face and one of 5 different poses.
| The number of people putting any sort of effort into it is
| way, way lower than the number of users thinking they are
| making art. It's more of a slot machine in the majority of
| cases
| vunderba wrote:
| Unfortunately, yeah 99.9% of images you're going to see
| generated from stable diffusion models are going to be
| either selfies, portraits, or porn.
|
| What's you're not going to see is things like "a divine
| gigantic textile loom sewing together a white horse and a
| black horse in an interlaced pattern to create a zebra."
| for example.
| __loam wrote:
| I definitely think the flooding of art spaces is hugely
| problematic, but it is pretty funny to watch people try to
| "be an artist" by putting essentially no effort in. It
| definitely points to a lack of understanding in the field
| when all these people are basically generating a ton of
| images that are all derived from the same models. There's a
| lack of understanding of supply and demand, when the
| expectation is that your ai illustration that you made in
| like an hour with the same software as every other ai artist
| is that it's somehow going to be competitive on engagement
| with an original piece from an artist who has an audience.
| There's a lot of demand for artists like Mika Pikazo and
| Frank Frazetta, not the 100,000 ai artist
| nestorD wrote:
| I believe that this is a non-problem pushed forward by small-
| scale experiments that are not representative of what people
| actually do with AI generation. A lot of new content, while AI
| generated, has been hand picked and polished by a human (for
| example, while you might commit AI generated code to your
| codebase, you ensure that it is correct and follows your
| preferred style). Content farms will push gibberish out, but they
| did so, and worse, before and the first generation of models was
| able to train on the internet anyway.
| randcraw wrote:
| It's fascinating that error can accumulate through repeated
| trainings that 1) is undetected by humans and 2) can degrade LLM
| or diffusion models (or any transformer model?) so completely.
| This implies that not only do we not understand how latent
| knowledge is actually representated in deep nets, we don't know
| it forms or how it changes during training. If we did, we could
| have predicted the destructive impact of recycling of output as
| input. IMO, this suggests we should demand rigorous validation of
| deep nets (especially generative ones) before relying on them to
| behave responsibly.
| esafak wrote:
| Computers need to be able to learn from the world at large, not
| just their own output. World models are needed to make progress.
| Bjorkbat wrote:
| I'm not sure how much of a risk this is to LLMs in particular,
| but I feel like we're already seeing the impact on image AI
| models.
|
| Even though they're getting better at generating hands that make
| sense and other fine details, you can generally tell that an
| image is AI generated because it has a certain "style". Can't
| help but wonder if this is partly due to generated images
| contaminating the training data and causing subsequent AI image
| generators to stylistically converge over time.
| p5v wrote:
| Is there a standard objective metric that can help determine that
| the quality of a model has degraded over time. In that case, much
| like source code, you just revert to the old version.
| RecycledEle wrote:
| Synthetic data is a disaster.
|
| If you want foom (fast self-improvement in AI) use AIs to filter
| the training data for the next generation of AIs.
___________________________________________________________________
(page generated 2024-03-09 23:01 UTC)