CPSample: Classifier Protected Sampling for Guarding Training Data During Diffusion

Thank you for reviewing our paper. We're glad that you find our experiments convincing. We address your concerns about our paper below:

Weaknesses:

Motivation: the main weakness of this paper lies in its motivation. As shown in existing research, training data replication is not a common phenomena in real-world pre-trained diffusion models. Most training samples are even unable to be inferred as member by membership inference attacks in Stable Diffusion, while the exception ones can be handled case by case.

To the best of our knowledge, memorization in diffusion models is a very real concern, and diffusion models are not robust to membership inference attacks [1]. Moreover, exact replication of training images is certainly a concern, as Carlini et. al. were able to extract many of the training images from Stable Diffusion [2].

While the proposed method still degrades the generation performance as well as the diversity (potentially), it seems uneconomical to deploy the proposed method on any pre-trained diffusion models.

Compared to training differentially private models, which can be several times slower than training standard diffusion models, using CPSample is comparatively cheap, as one only needs to overfit a classifier. Once trained, the classifier allows for flexible levels of protection without retraining.

Since privacy issues only happen in pre-trained diffusion models, the significance of the proposed method is moderate.

We are not sure what the reviewer means by this comment. Diffusion models are currently vulnerable to membership inference attacks. Individuals training models for applications such as medical imaging still face the risk of leaking their training data when serving models that they have trained or fine-tuned on sensitive data.

Computation cost & Generalizability: [...] In fact, the training of the classifier in the paper only includes less than 2 thousands images.

We trained classifiers on up to 160,000 images, as shown in Section 4.4. As demonstrated by Section 4.3, the resulting models are less vulnerable to MI attacks, and Figure 5 shows that they produce high quality images without revealing the training data. As stated in the paper, Zhang et. al. note that training a classifier on random versus real labels requires compute that only increases by a small constant factor as long as the number of network parameters is linearly proportional to the number of training data points [3].

OOD generation (minor): One potential drawback, as shown in Figure 4, is that the proposed method may generate images with contents quite different to the description of the text.

The images seem to be faithful to their captions to us. The first shows Sylvester Stallone, the second shows a classic car, and the third is abstract art.

Can you show how your method performs on large-scale datasets?

Which experiments would you like to see? We are an academic lab and cannot train from scratch on extremely large datasets. We showed that CPSample can protect datasets of up to 160,000 images, but we are resource-limited and have not tried to train on larger datasets than that. Moreover, if one needs to protect specific images (i.e. Sylvester Stallone) within a multi-billion image dataset, we have shown that CPSample is effective.

it does not seem to practical to deploy it on real-world AIGC applications which raises the data privacy concerns

We deployed this method with Stable Diffusion to protect a subset of the LAION dataset. Is there another experiment that you would like to see?

[1] https://arxiv.org/pdf/2301.09956 [2] https://arxiv.org/pdf/2301.13188 [3] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=Sy8gdB9xx.

I agree that the memorization of diffusion models is a big deal. However, according to [1, 2], only few proportions of images can be detected as memorization. For example, in [1], searching from 350,000 most duplicated images in Stable Diffusion only gets 94 of them as memorized images. I believe some post-training checks and strategies can cheaply avoid the model from generating these images. In other words, a practical way to solve the memorization problem should be, before releasing the model, conducting a memorization check, finding the memorized images, and doing some post-training refinements specifically for these images. This will not add any extra computational costs to the inference as the proposed method. The proposed method not only asks for extra costs in inference (users will not like this) but also requires a mounted classifier which could be easily bypassed (as the users of Stable Diffusion do to the safety checker). Hence, I think this will not be a practical method to solve the real-world memorization problem.

Thank you for reviewing our paper. We're glad that you share our view of the problem's importance. We address some of your concerns about our work below:

Weaknesses:

The scalability of the strategy to train a classifier is a primary concern. Given that current text-to-image diffusion models, such as Stable Diffusion and Pixart, are trained on large-scale datasets scraped from the Internet, is it a feasible solution to train an overfitted classifier on a large scale dataset?

We trained on up to 160,000 images without encountering problems. Zhang et. al. [1] show that training a classifier on random versus real labels requires compute that only increases by a small constant factor as long as the number of network parameters is linearly proportional to the number of training data points, suggesting that the method should be scalable.

The proposed method assumes that the training member set is available. However, this assumption does not always hold. Specifically, the training member sets of certrain text-to-image diffusion models, such as Pixart, are not publicly accessible, therby limiting the applicability of the proposed approach.

If one no longer has the original training data available, one can identify images that are at risk of being duplicated during training (for instance, using Carlini et. al. [2]) and train the classifier on those.

For example, in Figure 4, the generated image by CPSample does not align with the input prompt and sometimes will generate corrupted results. In contrast, although rejection sampling faces challenges with resampling, it tends to produce images that are more closely aligned with the input prompt.

While there are side effects to CPSample, they are minor compared to alternatives such as DPDM or ambient diffusion in terms of quality, as we show in Section 4.4. We argue that the images in Figure 4 are faithful to their prompts-- the third caption requested abstract art, so we don't think that this is an example of a corrupted image.

There are many typos in notations of DDPM

Thank you for pointing this out. We will carefully revise the paper.

Questions:

In my opinion, the trained overfitted classifier is designed to estimate a memorization score and captures a boundary between the member set and hold-out set to provide a guided gradient. However, various alternative methods exist for estimating this boundary or providing a similar memorization gradient. For instance, training a classifier to differentiate between the member set and hold-out set, or training an overfitted diffusion model to provide memorization gradients, are both plausible approaches. Therefore, why choose to assign random 0/1 labels to the member set? (The reasons provided in Line 238-244 is not very convincing for me)

We explicitly sought to avoid retraining a diffusion model, since this is extremely expensive, especially for large datasets. Training a classifier on random 0-1 labels is much cheaper than training a diffusion model. To clarify, the 0-1 labels were assigned to two halves of the training data, not to the validation data. This allowed us to provide classifier guidance during samples away from the closest examples and towards others. This prevented us from moving too far away from the data as a whole while still avoiding close proximity to any one data point. Please let us know if you have other questions about this aspect of the paper.

The scalability of CPSample to diffusion models with larger member sets is uncertain

We addressed this in the weaknesses section.

in more practical scenarios, diffusion models tend to memorize only the rough structure of the training sample and can only reproduce similar patterns.

As shown in Figure 4, there are many examples where diffusion models replicate or nearly replicate the entire image. CPSample is mainly focused on preventing this type of replication. Replicating only the general structure of an image is less likely to be a severe privacy concern.

The current experiments have been mostly conducted on unconditional diffusion models. To fully evaluate the effectiveness of CPSample, additional experiments on conditional diffusion models, such as text-to-image diffusion models, should be conducted to assess whether CPSample can accurately align with the given condition.

We provide two experiments on conditional diffusion models, one in Figure 4 and one in Appendix B. Which additional experiments would you like to see (subject to reasonable compute constraints)?

[1] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=Sy8gdB9xx. [2] https://arxiv.org/abs/2301.13188

Thank you for reviewing our paper and providing helpful suggestions. We address the weaknesses and questions that you brought up below.

Weaknesses

This method is very simple: “Overfit a classifier on random binary labels assigned to the training data and use this classifier during sampling to guide the images away from the training data.”

Is methodological simplicity a weakness?

Does it still require training the classifier on the large training dataset of diffusion models? What if we do not have access to the training data any longer?

As shown by Carlini et. al., vulnerable training data can be extracted from the diffusion model. One can train the classifier to protect this vulnerable data if the original data is no longer available. A more realistic use case is that one could fine-tune a pretrained model using one's own sensitive data and use CPSample to protect this sensitive data.

Thus, the CPSample method is not applicable to the SoTA diffusion models.

The reviewer is mistaken. As we show in Section 4.2, CPSample can protect models like Stable Diffusion that are not steered by a classifier.

The proposed CPSample method is very similar to the AMG guidance method from Chen et al. 2024.

We fail to understand the connection between CPSample and AMG. These seem to be methods used in different settings for different purposes. Could the reviewer please explain the connection?

How is the classifier trained for the proposed CPSample method without requiring access to the training data?

You do require access to the training data or at least knowledge of which images you wish to protect from replication. This is the case for the methods that we compare against, too (ambient diffusion, DPDM).

The qualitative results in Figure 3 clearly show the degradation of quality in the generated images between DDIM and CPSample (especially for the LSUN Church dataset).

These are not randomly chosen images-- they are the generated images that are most similar to the training data. Hence, it is unsurprising that the images from the original diffusion model (which are identical to training images) are of higher quality. The FID scores in Table 4 are a more fair metric of quality, and they show minimal degradation. For randomly sampled images, which would make for a more fair visual comparison, see Appendix F.

Additionally, Image C in the last row for the CPSample shows the complete degradation in the quality of the generated images.

The caption requested abstract art on several panels-- the image looks like abstract art on several panels. Note that it is the only image that looks "abstract."

What is the overhead of the method in terms of the inference time? How long does it take to train the classifier? How is the classifier trained?

Training details are provided in Appendix C. Inference times are commensurate with those for classifier-guided diffusion models. We will be happy to include timed trials in the final version of the paper if the reviewer recommends it.

The paper only compares the FID score of the CPSample to other methods, however, no comparison is presented in terms of the reduction in the generation of the training data points.

In the FID score comparison, neither CPSample nor the methods that we compared against produced any duplicates of the training data. Thus, we feel it is a fair comparison. Is there another experiment that the reviewer feels would strengthen the comparison?

Thank you for your answer, I appreciate it.

We will be happy to include timed trials in the final version of the paper if the reviewer recommends it.

ICLR allows authors to conduct additional experiments and even update their submissions during the review process. Therefore, the absence of the recommended experiments before the final decision significantly diminishes the paper's chances of acceptance.

This is not only the case for my review, but for other reviewers as well. For example, Reviewer 6foL asked for additional experiments with other MIA methods (which I also think should be added), however, they were not provided. The same holds for Reviewers dYdC and t8KP.

Thank you for reviewing our paper and offering these helpful suggestions. We address the weaknesses and questions that you brought up below:

Weaknesses

The presented theoretical study seems to be not proven much in terms of CPSample. Specifically, it is not demonstrated as to why assumption 3 holds well empirically as stated in the paper. Furthermore, given the intractable estimation and sometimes large values of Lipschitz constant, I wonder if that would make the possible $\delta$ extremely small which in terms renders the theory results in vain?

We would be happy to include a study of how frequently assumption 3 holds in practice. We provide an illustration in Figure 6 of how this assumption holds in practice, but we agree that this is likely insufficient verification. Are there particular experiments that you would like to see in this direction? We intended the theory to be a heuristic for why CPSample works, but we are open to suggestions that would help turn it into a more practical guarantee.

Some more experiments on MIA could be carried out to effectively evaluated the information loss caused when CPSample. Particularly, papers such as [1, 2] have devised new MIAs tailored to the settings of diffusion models which also conforms to the past settings in MIA. It would be great to see if CPSample could also withstand these newly devised attacks.

The membership inference attack given in Algorithm 1 and used in Section 4.3 is extremely similar to the attack in Duan et. al. Both inference attacks use reconstruction error as the basis for distinguishing between members of the training and held-out sets. We are happy to include more examples of membership inference attacks in the final version of the paper.

Questions

As mentioned in the first point of weaknesses, it would be great if the authors could elaborate more on the the third assumptions either empirically or theoretically along with the problem mentioned.

Please see the Weaknesses section.