Targeted Model Inversion: Distilling Style Encoded in Predictions

Ethical implications and possible defense methods

We believe that investigating a new attack and concretizing its performance and impact is an important contribution that motivates further research in mitigating the addressed threat. Numerous previous studies regarding adversarial attacks (e.g. FGSM [1], C&W [2]) and MI attacks (e.g. MIRROR, P&P) proposed novel and powerful attacks demonstrating new upper bounds of robustness and resiliency of ML models against the proposed attacks.

Additionationally, we will discuss possible mitigation methods in the revised paper. Specifically, we will discuss the downgraded MI performance when the target model only returns a label for a given query (see A.2.4) and another mitigation of injecting random noise to a prediction output (while preserving its prediction label). Observe that its performance dropped below MIRROR-b, in particular, F-dist and Cover metrics are greatly degraded from the original TMI experiment (21.1% and 81.6%). This suggests that the noise successfully distracts TMI from reconstructing subtle features. The table below shows the performance of TMI against target models that inject random noise to prediction outputs. Finally, we will explain the applicability of recent MI defenses [3-5].

---

[1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014).

[2] Carlini, Nicholas, and David Wagner. "Towards evaluating the robustness of neural networks." 2017 IEEE symposium on security and privacy (sp). IEEE, 2017.

[3] Wang, Tianhao, Yuheng Zhang, and Ruoxi Jia. "Improving robustness to model inversion attacks via mutual information regularization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 13. 2021.

[4] Wen, Jing, Siu-Ming Yiu, and Lucas CK Hui. "Defending against model inversion attack by adversarial examples." 2021 IEEE International Conference on Cyber Security and Resilience (CSR). IEEE, 2021.

[5] Xu, Qian, Md Tanvir Arafin, and Gang Qu. "An approximate memory based defense against model inversion attacks to neural networks." IEEE Transactions on Emerging Topics in Computing 10.4 (2022): 1733-1745.

[6] Schonfeld, Edgar, Bernt Schiele, and Anna Khoreva. "A u-net based discriminator for generative adversarial networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

We appreciate your helpful feedback. We address each concern in the following:

Narrow domain of application; nothing conceptually new

Please note that we demonstrated that retraining a mapping network in StyleGAN contributes to better distilling styles in prediction vectors, which provides promising initial datapoints for MI in the inversion phase. We also extensively evaluated the efficacy of this new MI attack, demonstrating the feasibility of distilling style in prediction for better MI.

To demonstrate the wide range of TMI’s applicability, we conducted two additional experiments: (1) using another generative model as prior, and (2) adopting TMI to a white-box attack scenario.

For (1), we replaced StyleGAN with UNet-GAN [6], and compared the results to the base experiment in the table below. For the surrogate model $f’$, we used the left-half of the UNet-GAN’s discriminator. Since UNet-GAN does not incorporate a mapping network, a custom mapping layer was trained from scratch. Observe that while the overall performance is slightly decreased compared to TMI with StyleGAN, it is still more effective than MIRROR-b. The slight drop in performance is largely due to the entangled latent space of UNet-GAN, where the latent space $\\mathcal{Z}$ is directly used without mapping it to an intermediate disentangled latent space $\\mathcal{W}$ in advance. We conclude that while StyleGAN is still the most effective image prior to be utilized, other GAN-based methods can generally benefit from our suggested approach of leveraging a new mapping layer and distilling the discriminator for a surrogate model. Therefore, the contribution of TMI is not only limited to the packaged attack method as a whole, but also the individual components that can be applied independently.

For (2), we used the original model instead of the surrogate model, and the performance gains are listed in the table below. As expected, the performance is improved by an average of 63.7%, outperforming the white-box attacks across all four metrics. This also indicates that retraining a mapping network can be applied for conducting white-box TMI attacks to improve their reconstruction performance. We will include this result in A.2.1. Effect of the Surrogate Model.

Our additional experiment results demonstrate that the distillation process of TMI is a generally-applicable technique which can not only support using other generative models as prior, but also augment existing white-box attacks.

Moreover, prior MI attacks have primarily focused on reconstructing class-generic features, which harms privacy of individuals that directly contribute to the training phase of the target model. Individuals who interact with the target model only after deployment (i.e., their user data are not included in the training set) were not exposed to privacy leakage. However, our experimental results suggest otherwise. Since TMI is capable of reconstructing the original image specific to an output prediction, it demonstrates the privacy threat of reconstructing inputs of ML service users. These results also shed light on information leakage from output predictions.

Q4. Can a different GAN architecture be used?

Yes, it is possible to utilize different GAN architectures since every GAN incorporates a latent space and a discriminator network. We conducted an additional experiment with UNet-GAN [11] pretrained on FFHQ and compared the results to the base experiment in the table above. For the surrogate model $f’$, we used the left-half of the UNet-GAN’s discriminator. Since UNet-GAN does not incorporate a mapping network, a custom mapping layer was trained from scratch. Observe that while the overall performance is slightly decreased compared to TMI with StyleGAN, it is still more effective than MIRROR-b. The slight drop in performance is largely due to the entangled latent space of UNet-GAN, where the latent space $\\mathcal{Z}$ is directly used without mapping it to an intermediate disentangled latent space $\\mathcal{W}$ in advance. We conclude that while StyleGAN is still the most effective image prior to be utilized, other GAN-based methods can generally benefit from our suggested approach of leveraging a new mapping layer and distilling the discriminator for a surrogate model. Therefore, the contribution of TMI is not only limited to the packaged attack method as a whole, but also the individual components that can be applied independently. The techniques from TMI might also be able to augment state-of-the-art diffusion models for MI attacks. For example, the mapping network can serve as a reliable encoder for latent diffusion models. We believe that designing an MI attack upon diffusion models is plausible, however, out of scope in this paper.

Q5+Q6. Incomplete math.

We appreciate the suggestion and will replace our math formulations in Eq2 and Eq3 with the formal equations as follows:

• Eq2. $\\underset{\\theta}{\\arg\\min}\\,\\mathbb{E}\_{(w,x,\\hat{y})\\in\\mathcal{D}\_{gen}}\\left[ (w-m'\_\\theta(\\hat{y}))^2\\right]$
• Eq3. $\\underset{\\theta}{\\arg\\min}\\,\\mathbb{E}\_{(w,x,\\hat{y})\\in\\mathcal{D}\_{gen}}\\left[ (\\hat{y}-f'\_\\theta(x))^2\\right]$

Q7. What happens if you take a stylegan trained on cars dataset and use it for a predictor trained on human faces?

Using an unrelated dataset will render TMI less effective, since its reconstruction domain is limited to the image manifold of the prior modeled in the generator. However, please note that as mentioned in W1+Q1, to the attacker’s perspective, it is probable to find a dataset or pretrained StyleGAN specific enough for TMI, given the abundance of public datasets and repositories. In addition, prior works on MI often use subsets from the target network’s trainset to train their inversion model, whereas our base evaluation for TMI utilize relevant, but different datasets. We believe that this adversary setting is an acceptable norm in the model inversion literature.

---

[11] Schonfeld, Edgar, Bernt Schiele, and Anna Khoreva. "A u-net based discriminator for generative adversarial networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

We thank the reviewer for the constructive feedback. We address each concern in the following.

W1+Q1. How would the attacker know which dataset is similar?

For a target ML service for users, it is trivial to know what kind of image they receive as input (face/medical/scenery/animal/...), the meaning of their output (similarity to celebrities/medical diagnosis/geolocation/...), and the characteristics of the output prediction (number of classes, meaning of each class) with little effort. Based on this information, the adversary can choose a pre-trained model or a dataset to use among numerous pre-trained models and datasets readily available on the Internet (e.g., Kaggle alone provides 268,605 datasets across 165 official subjects, including people, health, genetics, and business.). For instance, assume a scenario where the target service implements geolocation estimation based on photos of scenery [10]. The service can predict which country or continent you are in based on the input photo. Accordingly, the attacker can decide to search for datasets containing landscape images (such as the Google landmarks dataset or InstaCities1M) or related pretrained generators for MI attacks.

We emphasize that our adversary does not require additional knowledge compared to other prior MI attacks; our adversary leverages an auxiliary dataset that the adversaries that previous black- and white-box MI attacks [1-7] assumed.

W2+Q2. Imposes an additional space constraint, in terms of the large stylegan that is to be trained

We agree that training a StyleGAN network is a computational hurdle. However, building this StyleGAN network is a one-time procedure. After this preparation phase, MI attacks are completely offline while enabling high-fidelity input reconstruction for a large number of queries.

Q3. Unfair comparison; proposed method has the luxury of using additional full-blow generator network which the previous methods don't have

We note that all existing MI attacks [3-9] that utilize an image prior assume the same setting. Among the methods selected for baseline comparison, AMI is the only method that does not involve using a generator network, but still, it requires an auxiliary dataset to train the inverse mapping of the target model. One noticeable difference is that the auxiliary dataset is no longer needed for our TMI attacks that utilize an image prior, if there is a available pre-trained generator. However, AMI does not support exploiting the pre-trained model.

---

[1] Fredrikson, Matthew, et al. "Privacy in pharmacogenetics: An End-to-End case study of personalized warfarin dosing." 23rd USENIX security symposium (USENIX Security 14). 2014.

[2] Yang, Ziqi, et al. "Neural network inversion in adversarial setting via background knowledge alignment." Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2019.

[3] Zhang, Yuheng, et al. "The secret revealer: Generative model-inversion attacks against deep neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[4] Han, Gyojin, et al. "Reinforcement Learning-Based Black-Box Model Inversion Attacks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Kahla, Mostafa, et al. "Label-only model inversion attacks via boundary repulsion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[6] An, Shengwei, et al. "Mirror: Model inversion for deep learning network with high fidelity." Proceedings of the 29th Network and Distributed System Security Symposium. 2022.

[7] Struppek, Lukas, et al. "Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks." International Conference on Machine Learning. PMLR, 2022.

[8] Chen, Si, et al. "Knowledge-enriched distributional model inversion attacks." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[9] Wang, Kuan-Chieh, et al. "Variational model inversion attacks." Advances in Neural Information Processing Systems 34 (2021): 9706-9719.

[10] Weyand, Tobias, Ilya Kostrikov, and James Philbin. "Planet-photo geolocation with convolutional neural networks." Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer International Publishing, 2016.

We thank the reviewer for the constructive feedback. We address each concern in the following.

W1. Example scenarios

Consider an age classifier that gets a face-image query and classifies it into one of five classes that correspond to age groups, the adversary can populate a list of arbitrary prediction vectors that represent each class and then conduct TMI attacks to reconstruct the facial images by conducting TMI attacks. When this age classifier is hosted on an MLaaS cloud instance, the owner of this cloud instance should limit an excessive number of queries due to their computation and financial constraints. We explored this scenario in Section 5.3 (see Figure 4).

We also described several example scenarios in Section 3 in which the adversary obtains prediction vectors. These scenarios include users posting prediction results on social media simply for entertainment (e.g. celebrity look-alike apps such as StarByFace show the look-alike percentage to celebrities), medical professionals sharing diagnosis predictions for educational or consultative purposes, split inference settings where the inference result is disclosed to different parties [1] or an untrusted MLaaS server. Prior studies have addressed countermeasures to protect inference privacy by obfuscating [2,3] or encrypting the input [4]. However, the prediction vector is left in plaintext format.

W2. How long does this method take + Q1. How many queries to the surrogate model?

We compared the number of queries and the computation time per each attack in the table above. Please note that the required time is highly dependent to the baseline implementations and their own GAN prior. By default, TMI queries the surrogate model the same number of times to the update steps. As stated in A.1.2, we used $n=5000$ for our experiments. For other baselines, we assigned a default query budget that the respective attack assumed in their paper. We note that TMI with 5K updates exhibited the superior performance over all black-box baselines, as shown in Table 2. In the revised paper, we will include these experimental results. We emphasize that leveraging a surrogate model contributes to substantially decreasing the number of black-box queries directed at a target model (please refer to Section 5.3 for more details). Please note that the adversary can conduct MI attacks in an offline fashion by abusing this surrogate model, thus circumventing potential constraints associated with online queries. Consequently, the target service provider may remain unaware of whether they are under attack or not.

Q2. Replacing the surrogate model with the original (ignoring the query budget)

We appreciate this valuable suggestion. We conducted an additional experiment using the original model instead of the surrogate model, and the performance gains are listed in the table above. As expected, the performance is improved by an average of 63.7%, outperforming even the white-box attacks across all four metrics. This also indicates that retraining a mapping network can be applied for conducting white-box TMI attacks to improve their reconstruction performance. We will include this result in A.2.1. Effect of the Surrogate Model.

Q3. Is this method specific to GANs?

Although our paper is focused on StyleGAN, the contribution of TMI is not only limited to the packaged attack method as a whole, but also the individual components that can be applied independently. For example, the mapping network might also serve as a reliable encoder for latent diffusion models. We believe that designing an MI method using a diffusion models is also promising. However, we consider exploring this direction out of scope for this paper.

---

[1] Zecheng He, Tianwei Zhang, and Ruby B Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 148–162, 2019.

[2] Mireshghallah, Fatemehsadat, et al. "Shredder: Learning noise distributions to protect inference privacy." Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 2020.

[3] Liu, Qin, et al. "When deep learning meets steganography: Protecting inference privacy in the dark." IEEE INFOCOM 2022-IEEE Conference on Computer Communications. IEEE, 2022.

[4] Gu, Zhongshu, et al. "Privacy enhancing deep learning cloud service using a trusted execution environment." U.S. Patent No. 11,443,182. 13 Sep. 2022.

W1. Applicability of the current method restricted to domains related to the StyleGAN's training distributions

We acknowledge that TMI necessitates a StyleGAN prior model. However, all previous MI studies [1-9] have assumed access to an auxiliary dataset of which underlying distribution is either the same or slightly different from the distribution of the target dataset. Therefore, we proposed a new MI method that harnesses this auxiliary dataset in a black-box manner, achieving superior MI performance over the SoTA methods.

In addition, we note that our TMI evaluation spans two distinct application domains, namely chest X-ray and facial datasets. This suggests that, given an appropriate StyleGAN prior, TMI is applicable to diverse classification tasks for MI. Given the abundance of pre-trained models and datasets readily available on the internet (e.g., Kaggle alone provides 17,678 image datasets across 164 official subjects, including people, healthcare, genetics, and business.), we believe that the adversary is capable of selecting their preferred StyleGAN prior for their needs. Even when there is no available dataset or pretrained StyleGAN, the adversary still has an option to collect their own images and train a StyleGAN specific to their need. Image search or sharing services (such as Google Images, Flickr, and Pinterest) allow attackers to access a large volume of image data.

W2. Explore leveraging advanced GAN techniques

We will discuss potential improvements by integrating SoTA models and the connection between GAN inversion to MI attacks. GAN inversion typically requires an original image to find a suitable latent point, whereas our TMI attacks do not assume white-box access to the target model or access to the original image. Leveraging a prediction output without the original image, TMI performs the best for identifying the optimal latent point that contributes to reconstructing the input image among the SoTA MI methods.

W3. more accurate test of TMI will be to use a much more generic dataset, which will likely fail

Using a much more generic dataset will render TMI less effective, since its reconstruction domain depends on the image manifold of the prior modeled in the generator. However, please note that as mentioned in W1, to the attacker’s perspective, it is probable to find a dataset or pretrained StyleGAN specific enough for TMI, given the abundance of public datasets and repositories. In addition, prior works on MI often use subsets from the target network’s trainset to train their inversion model, whereas our base evaluation for TMI utilize relevant, but different datasets. We believe that this adversary setting is an acceptable norm in the model inversion literatures. Also note that TMI assumes a weaker black-box adversary with a limited number of input queries.

W4. Feature distance poorly correlated to image quality

We appreciate this valuable insight and agree that feature distance alone is insufficient to demonstrate reconstruction performance. Therefore, we used four metrics to comprehensively measure the MI performance of TMI and the other baselines. We will discuss the downsides of using feature distance alone in measuring MI performance.

---

[1] Fredrikson, Matthew, et al. "Privacy in pharmacogenetics: An End-to-End case study of personalized warfarin dosing." 23rd USENIX security symposium (USENIX Security 14). 2014.

[2] Yang, Ziqi, et al. "Neural network inversion in adversarial setting via background knowledge alignment." Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2019.

[3] Zhang, Yuheng, et al. "The secret revealer: Generative model-inversion attacks against deep neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[4] Han, Gyojin, et al. "Reinforcement Learning-Based Black-Box Model Inversion Attacks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Kahla, Mostafa, et al. "Label-only model inversion attacks via boundary repulsion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[6] An, Shengwei, et al. "Mirror: Model inversion for deep learning network with high fidelity." Proceedings of the 29th Network and Distributed System Security Symposium. 2022.

[7] Struppek, Lukas, et al. "Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks." International Conference on Machine Learning. PMLR, 2022.

[8] Chen, Si, et al. "Knowledge-enriched distributional model inversion attacks." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[9] Wang, Kuan-Chieh, et al. "Variational model inversion attacks." Advances in Neural Information Processing Systems 34 (2021): 9706-9719.

We thank the reviewer for the constructive feedback. We address each concern in the following.

Q1. Ablations on query budget

We conducted additional experiments to assess the performance of the baseline methods across varying query budgets (5, 10, 50, 100K). Please note that the query budget denotes the total number of queries allowed to attack every label (i.e., 530 attacks in case of FaceScrub). We included the experimental results of two best-performing baseline methods (i.e., P&P and MIRROR-w) and our TMI attack in the table above. The graphical representations of the performance of all baseline methods can be found here. As the table above shows, the overall MI performance increases as the number of queries increases. TMI consistently outperforms the baseline attacks across different query budgets. We also note that TMI holds a distinctive advantage of requiring zero queries to a target model after the preparation phase, which significantly facilitates the reconstruction of inputs for a large number of prediction vectors.

During the experiments, we identified an error in the query count of LO-MI in Table 1. For each attack attempt, LO-MI requires 25k queries, instead of 2k. This will be corrected in the final version. We confirm that this erroneous typo does not impact the overall conclusion of our study since we overstated the performance of the baseline attack (i.e., LO-MI). Correcting this value highlights the superior performance of TMI attacks.

Q2. Surrogate model’s performance on the original dataset?

We measured the accuracy of the TMI surrogate models using the test splits of the FaceScrub, CelebA, and PadChest datasets. The table below describes the performance metrics of both original ($f$) and surrogate ($f’$) models. In the revised paper, we will include these experimental results.

Q3. Use of gendered language for describing the attacker

We will replace those words with gender-neutral terms. We note that prior studies in the security and privacy domain have often employed gender-specific terms when referring to the adversary.