[HN Gopher] Extracting training data from diffusion models
___________________________________________________________________
Extracting training data from diffusion models
Author : ericwallace_ucb
Score : 87 points
Date : 2023-01-31 15:22 UTC (7 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| jjcon wrote:
| Is there any reason we shouldn't view diffusion models as any
| other tool? I can infringe copyright with photoshop too... even
| accidentally. If I generate original work with either that seems
| like fair game.
|
| I imagine with the right prompt one could coax out a copywritten
| image even if it hadn't ever seen it before
| yunwal wrote:
| There's still valid reason to be concerned here. For example,
| this implies building a diffusion model based on private data
| can leak it. If I can generate a whole bunch of prompts like
| "MRI Joe Biden brain tumor" and 1 out of a million times I get
| a consistent result, that's unacceptable.
|
| Github Copilot could be leaking private code as well.
| ericwallace_ucb wrote:
| The paper shows that Stable Diffusion and Google's Imagen
| regenerate individual images from their training sets. They show
| it is very rare, but can be found reliably.
| pavlov wrote:
| Seems relevant to the Getty Images lawsuit against Stable
| Diffusion.
| sdenton4 wrote:
| Eh... The mp3 decoder can generate copyrighted music if you
| feed it the right inputs...
|
| Likewise, in this work they prime the pump by using exact
| training prompts of highly duplicated training images. And
| then you have to generate 500 images from that prompt to find
| 10 duplications. You've really gotta want to find the
| duplicates, which indicates that these are going to be
| extremely rare in practice, and even more rare once the
| training data is hardened against the attack by
| deduplication.
| PartiallyTyped wrote:
| The probability that synthesis of similar enough training
| samples given that the dataset does not contain duplicates is
| astronomically small, in this work they purposefully
| manipulated the dataset, used specific prompts and gave all
| sorts of advantages to an adversary, and created many images
| per prompt to be able to find such cases.
| roenxi wrote:
| No doubt, but how relevant? If you could somehow go through
| my brain you'd also find the occasional piece of art and
| literature - I've memorised a few good poems and songs. And I
| can recognise certain paintings on sight which would be
| difficult if they weren't accurately encoded in my mind
| somewhere. The fact that I have memorised them doesn't mean
| that I'm violating anyone's copyright if I attempted to
| compose poems and songs.
| amelius wrote:
| What do you mean by the last sentence?
| nickthegreek wrote:
| I imagine reliability means that with the exact prompt, seed,
| cfg scale, model checkpoint, blah blah and on the same
| hardware, they can continue to get an image that they
| consider close enough to the original.
| Lerc wrote:
| Figure 2 doesn't fill me with confidence as to the ability to
| detect similar images. The best example is the bottom right match
| which hits because the collar is in the same position and a bunch
| of white being in the same place outweighs a lot more meaningful
| data.
|
| This probably means there are far more matches to be found that
| would be considered clearly copies to humans. SSIM might be a bit
| heavy for the task but a simple comparison of the gradients from
| neighboring pixels might match quite a lot more.
| jxy wrote:
| I don't understand what is so surprising here. The training of
| the model consists of adding noise to training samples and
| denoising the resulting random samples to reproduce training
| samples. If you have one training sample, you can find the
| optimized random sample that reproduce the training sample.
| yorwba wrote:
| After adding noise to a training sample, there's still some
| information about the original contained in the noisy image, so
| the model only needs to fix it up a little with the help of the
| prompt. But when you start from 100% noise, _all_ of the
| information needs to come from the model and the prompt.
|
| Seen from another angle, if you take a random value for each
| pixel, you're unlikely to generate anything resembling a
| picture, let alone any given training picture you're trying to
| reproduce. That there's an input that makes the model output a
| training sample doesn't necessarily mean that it's easy to
| find. But the paper shows that you can find several by guessing
| randomly.
| chickenpotpie wrote:
| Research isn't always about finding surprising things. Some of
| the most important research ever conducted is proving things
| that seem intuitive, especially when it proves the opposite.
| sdenton4 wrote:
| An alternative framing if the article is that reproving of
| training examples is /rare/. It works for highly replicated
| images and, in larger models trained for longer periods of
| time, really unusual training examples. Both of these failure
| modes have pretty obvious mitigations: deduplicate the training
| data, and trim the outliers.
| mxwsn wrote:
| Their extraction: (1) assumes the attacker knows the caption for
| some training images, and (2) primarily works on images
| duplicated 100x-3000x in the training dataset. Their attack does
| not succeed for any singleton images. Deduplicating can be
| challenging on internet-scale datasets, but their work as
| presented does not appear to be a major concern for releasing
| diffusion models trained on other smaller datasets.
|
| On memorization - I suspect this is a great thing for downstream
| performance, and a positive indicator that diffusion models are
| actually better generative models than prior methods (VAEs, GANs,
| etc). This mirrors the finding that feedforward neural networks
| can memorize randomly labeled data very well. Intuitively it
| feels like memorization is a quantifiable behavior that is a
| foundational activity in information processing - it is one type
| of optimal usage of observed data - that superpowers downstream
| performance.
| PartiallyTyped wrote:
| > are actually better generative models than prior methods
| (VAEs, GANs, etc).
|
| Diffusion models _are_ VAEs and follow the same variational
| framework. You could imagine that VAEs are diffusion models
| with a single step in the forward and backward processes ;).
| They actually optimize the same VLB objective, but with
| diffusion models the objective is a trajectory instead of a
| single step, however, when training we are optimizing single
| step transitions. This is possible because the objective ends
| up being a sum of logarithms, thus there is no dependence
| between terms.
|
| In practice we solve a simplified objective which looks a lot
| like as we do with standard AutoEncoders ;)
|
| The key component that differentiates the two is in what we
| expect of the underlying neural network. It is far easier to
| parameterize small changes than large ones, with VAEs you ask
| the decoder to produce a large change in the latent variable,
| whereas with diffusion, we generally split them into 4000
| smaller changes assuming you are using the DDPM approach and
| not the DDIM one.
|
| Because we are improving with very small steps, we avoid the
| blurriness of VAEs, and we don't go out of distribution when
| sampling random noise. VAEs are often difficult to synthesize
| because even with KLD in the objective, the encoder produces a
| low variance distribution, and so when we sample noise from a
| high(er) variance gaussian, we are out of distribution rather
| quickly.
| mxwsn wrote:
| I agree it's illuminating to understand diffusion models in
| relation to VAEs, but I personally consider them different
| models, but the line in the sand is definitely subjective.
|
| I think this because (reasons I'm sure you're familiar with)
|
| - Diffusion model is closest to a hierarchical VAE, but
| hierarchical VAEs were significantly less popular than
| regular VAEs
|
| - The variational objective in diffusion models in practice
| is weighted
|
| - Diffusion models require unchanging latent dimension while
| VAEs aren't restricted to this
|
| - Historically, diffusion models grew out of score-based
| approaches, not from VAEs
| PartiallyTyped wrote:
| You raise good points, if anything, it'd have probably been
| more accurate of me to express that DDPMs and probabilistic
| variants fit within the same Bayesian framework as VAEs but
| with the posterior and likelihood functions simply being
| markov chains instead.
|
| This allows us to separate non probabilistic diffusion
| models e.g. cold diffusion. But then again, what's the
| difference between a deterministic model and sampling from
| a delta function? ;)
| Zacharias030 wrote:
| Super interesting post. Tyvm! Which three sources would you
| recommend for someone fluent in ML to read up on to arrive at
| your conclusions presented here (or their own)?
| PartiallyTyped wrote:
| The Variational Auto Encoder paper [1], and the DDPM
| paper[2] are pretty much all you need for this, [6[ is also
| good but covered by [2]. Going through the derivations
| helped solidify things for me. I haven't read [9] but looks
| very promising, authors include Jonathan Ho, and D. Kingma
| who authored [2] and [1] respectively.
|
| From there [3,4] show improvements to DDPMs, [5] shows that
| diffusion models can be very general. [7,8] show diffusion
| models from the view of score matching.
|
| [1] AutoEncoding Variational Bayes
|
| [2] Denoising Diffusion Probabilistic Models
|
| [3] Denoising Implicit Models
|
| [4] Improved Denoising Diffusion Probabilistic Models
|
| [5] Cold Diffusion
|
| [6] Deep Unsupervised Learning using Nonequilibrium
| Thermodynamics
|
| [7] Generative Modeling by Estimating Gradients of the Data
| Distribution
|
| [8] Score-Based Generative Modeling through Stochastic
| Differential Equations
|
| [9] Variational Diffusion Models
| Imnimo wrote:
| Only 109 retrievable images out of the 350,000 most-duplicated is
| fewer than I expected. Maybe it's just the stringent definition
| of retrieval, but I would have expected many famous works of art
| like the Mona Lisa and Girl with a Pearl Earring to be readily
| extractable. Maybe these just aren't quite pixel-perfect enough?
| saurabh20n wrote:
| The last author's tweet thread and replies have some interesting
| tidbits:
| https://twitter.com/Eric_Wallace_/status/1620449934863642624
|
| * "We propose to extract memorized images by generating many
| times with the same prompt and flagging cases where many of the
| generations are the same."
|
| * "- Diffusion models memorize more than GANs - Outlier images
| are memorized more - Existing privacy-preserving methods largely
| fail"
|
| * "Stable Diffusion is small relative to its training set (2GB of
| weights and many TB of data). So, while memorization is rare by
| design, future (larger) diffusion models will memorize more."
|
| * "It only memorizes a very small subset of the images that it
| trains on."
|
| * "our goal is to show that models can output training images
| when generating in the same fashion that normal users do."
| larsiusprime wrote:
| Some interesting commentary by AI expert Alex J. Champandard:
| https://twitter.com/alexjc/status/1620466058565132288
| polygamous_bat wrote:
| I am highly sceptical of the "Expertise" of the "expert" you
| cite, as a current Ph.D student in a relevant field. The "hot
| take" also seems highly technically inaccurate, presenting it
| as a "lossy database" when it's very clear that the average or
| even more than the 90% percent of the images could not be
| reproduced. In which way is this a "lossy db"? That, combined
| with some snark and sound bites, thoroughly turned me off from
| continuing to read the thread.
|
| I am generally concerned, these days, because it seems like the
| days when I could just keep my head down and do science is
| over, and now I have to also defend myself against opportunist
| hype-masters who probably were jumping on the crypto bandwagon
| six months ago and now are self proclaimed AI experts.
| GaggiX wrote:
| So wait they only found 109 matches after generating 175 milion
| images using the prompt from the most duplicated samples from the
| dataset and SD v1.4? Also almost all of them have more than 300
| copies in the dataset, so with a model with the same size and
| trained on a dedup dataset like SD 2.0/2.1 there will be almost
| no matches, even after generating 175 mln images and knowing the
| prompts used in the dataset. Finally Google at el need to explain
| how an attacker that want to extract images from a trained model
| somehow has the prompts for the top X duplicated images in the
| dataset but not the images themselves, and thus will going to
| spent an incredible amount of money to generate something like
| 175 mln samples and test them together to find the matches.
| yorwba wrote:
| I wonder whether the "data dimension" from https://transformer-
| circuits.pub/2023/toy-double-descent/ind... could be used to
| identify the model parameters involved in memorization and remove
| them without having to retrain from scratch on a cleaned-up
| dataset.
| mshake2 wrote:
| I expect to see this paper in many lawsuits soon as evidence of
| copyright infringement.
___________________________________________________________________
(page generated 2023-01-31 23:01 UTC)