Single-Step Diffusion Model-Based Generative Model Inversion Attacks

Q1. According to the ablation study, the prior loss specified in your method brings negative improvement. Why do you keep this innovation point in your method?

The prior loss was proposed as part of the diffusion distillation framework to explore its potential effects on performance. However, it is not a mandatory component for optimization in our method. As shown in our ablation study, the prior loss brings negative improvement in some cases, which is why we excluded it from the main optimization. It is similar to how discriminator loss is not utilized in PPA [r3], discussed in its ablation study. Its inclusion was intended to examine its potential influence and provide a more comprehensive understanding of its role, rather than to position it as an essential element of our approach.

---

References:

[r1] Zhang et al. "The secret revealer: Generative model-inversion attacks against deep neural networks." In CVPR, 2020.

[r2] Chen et al. "Knowledge-enriched distributional model inversion attacks." In ICCV, 2021.

[r3] Struppek et al. "Plug & play attacks: Towards robust and flexible model inversion attacks." In ICML, 2022.

[r4] Nguyen et al. "Re-thinking model inversion attacks against deep neural networks." In CVPR, 2023.

Thank you for your time in reviewing our work and for your comments. Please see our responses to your comments and questions below.

W1. The method of distilling a single-step generator has been extensively studied in prior work, which limits the novelty of this research.

While the technique of distilling a single-step generator has indeed been studied in prior work, our contribution lies in its novel application within the context of model inversion attacks (MIA), which has not been explored before. Specifically, we introduce inversion-specific distillation process in Section 3.3, and embed it within a new framework diffusion distillation MIAs (DDMI) that leverages diffusion models to significantly enhance generative MIAs.

This combination of techniques and their application to MIA represents a meaningful innovation, as it not only demonstrates how single-step generators can improve attack effectiveness but also establishes a new perspective on evaluating privacy vulnerabilities through advanced generative models.

W2. The new investigation for CLIP model inversion is non-sense.

We hypothesize that utilizing more detailed templates can improve numerical performance. Our additional experiments show that detailed templates reduce the overall KNN distance, as illustrated in the table below, where we use ViT-L-14 as the vision model in three methods.

Furthermore, we suggest that the numerical results are influenced by the frequency of target entities in the LAION-400M dataset. The vision model was trained on the LAION-400M dataset, which contains a diverse range of entities, and most target entities in the FaceScrub dataset has low frequency.

W3. The experimental results are not sufficient.

As your comments are partially similar to those of Reviewer mTRw, please refer to our response to Reviewer mTRw‘s W1 for further details.

W4. The experiment setup is too weak.

• Regarding the statement that “investigation on such a scenario is non-sense to prove the superiority of your method,” we would like to clarify that using CelebA as the public dataset is a commonly adopted benchmark setup in previous works, such as GMI [r1], KEDMI [r2], PPA [r3] and LOMMA [r4], even though PLG achieves perfect attack accuracy. Our objective was to demonstrate the improvements achieved by integrating SDM, which we did by comparing performance before and after this integration.

• Besides, FFHQ as public dataset is also included to address scenarios with higher distributional shifts, allowing for a more robust comparison, particularly with PLGMI. Our paper already includes experiments using FFHQ as the public dataset, and the results with FFHQ are presented in Table 3. This setup introduces greater distributional shift and provides a more challenging scenario for evaluating our method.

• Regarding the division of CelebA, we follow default setup in the standard MIA literature, which divides CelebA into two non-overlapping sets. We will explicitly include the dataset division in our experiment setup. Thanks for pointing this issue.

W5. The experiments of defenses in Table 6 shows that the attack results are worser than the case without SDM with the BiDO defense.

It is indeed true and acceptable that every method can exhibit failure cases, especially when faced with robust defenses like BiDO. The SDM-based method features a more powerful and higher-quality generator, resulting in significantly better image quality compared to traditional methods (refer to Fig.5 in our revised paper). The generated images are more representative and reveal more privacy features. From the perspective of visual results, SDM can still more clearly demonstrate the overall characteristics of the data distribution, thereby exposing more potential privacy information.

However, from a data perspective, we believe that current metrics may not effectively measure MI performance. While SDM shows lower performance than the baseline in attack accuracy and feature distance, it exhibits an opposite trend in the FID metric.

One possible explanation for this is that SDM's generative objective may place greater emphasis on the consistency of the overall data distribution rather than precisely reproducing the fine-grained details of specific inputs. As a result, the samples generated by SDM tend to reflect the average characteristics of the distribution. Although they achieve higher visual quality, they may be limited in capturing specific target privacy details and improving attack accuracy.

Thank you for your careful review of our paper and for pointing out these issues. We have provided detailed responses to your comments below.

W3. There are two “Sec.3.1”.

Thank you for identifying the typographic errors. We have corrected the duplicate "Sec. 3.1" issue in the revised version of the paper. Your attention to detail is appreciated.

W2. The specific expression of $L_{distill}$ in formula 8 is not explained in the paper.

We appreciate the reviewer’s helpful comments regarding the explanation of $L_{distill}$ in Eq. (8). The $L_{distill}$ term in Eq. (8) can represent any suitable distillation loss function. To address this concern and improve clarity, we will add explanations in the paper to clarify any ambiguity. Specifically, in Section 3.4, we instantiate $L_{distill}$ as $L_{SiD}$, which is tailored to our application.

Q1. The trend in Fig 1(a) seems similar between lines. More explanation is expected.

In Fig 1(a), there is indeed a similar trend across all methods: attack accuracy generally increases as optimization iterations proceed. This trend reflects that samples are increasingly optimized to approximate private data with more iterations.

However, our focus here is to reveal the instability issues present in the optimization process of the SOTA methods. While all methods show an upward trend, the SOTA methods display consistent fluctuations and higher variance in attack accuracy, indicating instability. In contrast, our SDM-based LOMMA (GMI) method demonstrates a steadier increase with minimal fluctuations, highlighting a more stable optimization process.

Thank you for your constructive comments! Please see our detailed responses to your comments and suggestions below.

W1. Poor selection of baseline methods for comparison in Section 4.2.

• First, we acknowledge that KEDMI and LOMMA (KEDMI) represent strong baselines. However, KEDMI utilizes inversion-specific GAN models to derive the generator, which contrasts with our goal of using a plug-and-play generator that is applicable across various MIA frameworks. This distinction is clearly demonstrated in our experiments, where we evaluate performance using both the original generators and our single-step (conditional) diffusion model across multiple methods. By comparison, KEDMI relies on an inversion-specific GAN architecture, making it challenging to evaluate under the same plug-and-play framework.

• That said, if an absolute performance comparison is desired, comparing conditional diffusion model derived from inversion-specific distillation with KEDMI might be more appropriate. Would you consider this comparison suitable?

• We would also like to clarify that PLGMI is included as a baseline in our paper. Specifically, we discuss PLGMI in Section 4.2.1, with qualitative results presented in Table 3 and additional metrics in Table 8.

W2. No empirical evidence/ theoretical grounds for strong claims made in Section 3.1.

• We agree that the statement, “We attribute these limitations to inherent flaws in GANs,” requires a more precise and well-supported explanation. Our motivation for introducing diffusion models is to leverage the rapid advancements and proven strengths of these models in generating high-fidelity images. Inspired by these developments, we aim to explore their potential as more robust tools for investigating model inversion attacks (MIAs). We will revise the paper to provide a clearer and more solid motivation for this transition.

• Regarding the claim about instability in GAN-based MIAs, we recognize that the evidence provided in Fig. 1(a) alone may not be sufficient to justify such a strong conclusion. Based on experimental observations and further analysis, we hypothesize that the observed instability arises from the limited quality and capabilities of the GANs employed in these methods. We will revise our discussion to present an evidence-backed analysis of this aspect.

• For the suggestion to include LOKT as a baseline, we acknowledge its significance as a state-of-the-art label-only MIA. However, similar to KEDMI, LOKT depends on a GAN-specific learning framework (T-ACGAN), making it less suitable for a direct comparison with our approach. Nevertheless, we appreciate the suggestion and will clarify this distinction in our revised manuscript.

• For the standard deviation of our main experiments, please see our general reply.

Thank you authors for the extensive rebuttal and performing additional experiments. I have read all the reviews and the rebuttal. The rebuttal addresses some of my concerns (PLGMI, Error bars for experiments, clarification reg. the use of inversion-specific GANs in the proposed framework). I acknowledge the notable performance gains achieved using your proposed framework and it can be valuable to the community. However, the core questions still remain as acknowledged by the authors in the rebuttal (No empirical evidence/ theoretical grounds for strong claims in Sec 3.1, user studies).

Therefore, I’m afraid I’m unable to increase my rating, as I feel the paper is not ready for publication in its current form. I have suggested some directions for improving Section 3.1 (See my review). Another way to think more deeply about this problem is as follows: Model inversion attacks aim to search for high-likelihood samples under a target identity. Why, then, does your proposed framework yield significantly better results in this search compared to existing methods? Investigating these questions could provide a more rigorous analysis and a deeper understanding of your framework.

Overall, this work has good potential, and I wish you the best as you continue to refine it. I also encourage you to address high-resolution MIAs, as suggested by other reviewers.

Thanks again for the extensive rebuttal.

Thank you for taking the time to review our work. We have provided detailed responses to your comments below.

W1. The core idea of replacing GANs with diffusion models have limited technical depth without fundamental insights into privacy mechanisms.

According to the current theoretical understanding of model inversion attacks (MIAs) [r1], the presence of a powerful "common generator" is critical for the success of such attacks. Our proposed method, DDMI, introduces a novel perspective by leveraging diffusion models as an advanced tool in MIA research. Unlike GAN-based approaches, diffusion models, as likelihood-based methods, provide superior coverage of the data distribution. This capability enables diffusion models to learn a more effective "common generator," enhancing the MIA framework’s ability to reconstruct sensitive information with greater detail and accuracy. By adopting this approach, our work contributes to advancing the technical foundation of MIAs.

W2. No theoretical gurantee of privacy such as Differential privacy or other theoretical foundations.

Theoretical guarantees of privacy, such as differential privacy, are typically associated with defense strategies designed to mitigate attack risks. Since our work aims to explore vulnerabilities and understand the capabilities of model inversion attacks, establishing theoretical privacy guarantees is outside the scope of this research. We appreciate your feedback and hope this clarification provides a clearer perspective on our study’s objectives.

Q1. How does this method fundamentally advance our understanding of privacy vulnerabilities beyond showing that better generative models lead to better attacks?

Currently, generative MIAs serve as an empirical method for assessing privacy vulnerabilities in machine learning models. While previous MIA methods have shown strong performance in terms of specific metrics, the visual fidelity of recovered samples has generally remained limited. This limitation affects the practical effectiveness of these methods in accurately assessing privacy risks.

Our approach advances the field by introducing a powerful single-step diffusion model, enhancing the capacity of generative MIAs to produce high-fidelity reconstructions. This improvement provides a more robust evaluation of privacy vulnerabilities in machine learning models, enabling a deeper understanding of the potential risks that such models may pose. Our work goes beyond simply demonstrating that improved generative models yield better attacks; it provides a concrete method for more effectively probing and evaluating model privacy in a meaningful way.

Q2. Why choose SiD specifically for distillation? What are the trade-offs compared to other distillation methods?

Our goal is to enhance the effectiveness of model inversion attacks by leveraging a powerful diffusion model without introducing significant computational overhead. We chose SiD for distillation because it achieves state-of-the-art performance among diffusion distillation methods (up to the time of submission). Notably, SiD enables the student model to surpass the teacher model in generation quality, making it an exceptionally suitable choice for our approach.

---

References:

[r1] Wang et al. "Variational Model Inversion Attacks." In NeurIPS, 2021.

Sincerely thank you for your constructive comments and generous supports! Please see our responses to your comments and suggestions below.

W1. "Distill the Multi-Step Diffusion Model into a Single-Step Generator" has no much innovation, similar to other One-step Diffusion papers.

As your comments are similar to those of Reviewer MUun, please refer to our response to Reviewer MUun‘s W1 for further details.

W2. The paper could benefit from including a broader array of evaluation metrics that could capture other aspects of model performance and attack effectiveness, such as computational efficiency (Q1), robustness to adversarial defenses, or qualitative user studies.

We would also like to clarify that experiments on robustness to adversarial defense are included in our paper. Specifically, we discuss the results in Appendix D.1, with qualitative results presented in Table 6. As for computational efficiency and qualitative user studies, we will incorporate these metrics into the revised version of our manuscript.

Q2. Why there is no analysis of the robustness of adversarial defenses in the experiment or variance experiment?

For experiments and analysis of the robustness of adversarial defenses, please refer to the response to W2.

For variance experiment, we have conducted additional experiments to include standard deviation of main metrics and please refer to the general reply.