Stealthy Shield Defense: A Conditional Mutual Information-Based Approach against Black-Box Model Inversion Attacks

We sincerely express our gratitude for dedicating your valuable time to providing positive comments that will enhance our paper. Your praise regarding our applicability, insight, methodology and experiments, have greatly encouraged and motivated us. Our detailed responses to all of your conerns are presented below.

Even though the authors claim that the computational overhead is negligible due to the efficient optimization on GPU, a more detailed analysis or benchmarking of the computational cost would greatly support this claim.

Following your suggestion, we add a quantitative experiment stated in Common Concerns on Computational Cost, where the results indicate that modifying 512 predictions for 100 times only needs 0.5 seconds with our careful optimization.

The proposed scheme requires iterating over all possible labels to perform the defense. What happens if there are a large number of possible labels? For example, a large company or region may train a face recognition model that includes thousands or millions of faces. Will this scale slow down the defense and become a bottleneck? I suggest some detailed computational analysis.

As analyzed above, we have simplified the objective function, reducing the space complexity from $\Theta(|\mathbb{Y}|^2)$ to $\Theta(|\mathbb{Y}|)$, and time complexity from $\Theta(|\mathbb{Y}|^2)$ to $\Theta(|\mathbb{Y}|\log|\mathbb{Y}|)$.

Furthermore, we add an experiment to illustrate the relationship between $|\mathbb{Y}|$ and time-cost. We generate $\pmb{x}\in\mathbb{R}^{|\mathbb{Y}|}\sim N(\pmb{0},\pmb{I})$ and let $\pmb{y}=\text{softmax}(10\cdot\pmb{x})$. It is observed that the $\pmb{y}$ generated in this way is close to the real probability distributions. We generate $\pmb{f}(x)$ and $\pmb{q}^y$ randomly in this way, and let our GPU-based-water-filling to find its optimal solution $\pmb{p}$. We take a batch with 256 pairs $(\pmb{f}(x),\pmb{q}^y)$. We record the time by torch.profiler, an official tool provided by PyTorch. Our experiment is conducted on one NVIDIA GeForce RTX 3090. Here is the result:

It shows that even when $|\mathbb{Y}|$ reaches a million, solving 256 convex optimization problems only takes us 1.3 seconds. We believe that at this point, our post-processing will not be the performance bottleneck, but the slow inferring and massive parameters of the target model will be.

Using MI/CMI in deep learning is gaining increasing attention. However, its introduction in related work lacks both depth and breadth, which makes it hard to find the role of this work into the relevant community.

Thank you for pointing this out. The application of conditional mutual information (CMI) to deep learning was proposed this year [1]. So far, there are only three published works on this topic. In the revised version of our paper, we will supplement mutual information (MI) and information bottlenecks (IB) in related work.

Even though this is a robust scheme against model inversion attack, the authors should discuss about the potential possibility of adaptive attacks. If the adaptive attack is unlikely to happen for now, the authors should also state the reason why.

Are there any manifest adaptive attack schemes that can target at this defense? How difficult is it to design/launch such attacks?

Yes! Your question has prompted us to think deeply about adaptive attacks. In Common Concerns on Adaptive Attacks, we discuss the adaptive attack strategies that may apply to us, and our countermeasures against them.

[1] En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning: Framework and preliminary results. IEEE International Symposium on Information Theory, 2024.

This is indeed a profound and pivotal question. Since the first day we saw it, we have been thinking about it. Now, we can finally provide a satisfactory clarification on this, both theoretically and experimentally.

Theoretically, higher-dimensional $X$ lead to a higher conditional mutual information $\mathbb{I}(X;\hat{Y}|Y)$, not lower. As you said, higher-dimensional $X$ contains more information irrelevant to $\hat{Y}$, but it also contains more information relevant to $\hat{Y}$. Our CMI only quantifies the relevant parts (by its definition). The more relevant information, the greater the CMI.

Furthermore, we provide a mathematical proof. Let $X_{high}$ be the high-dimensional input, $X_{low}$ be the low-dimensional input, and $X_{diff}$ be the difference between them. There is

where (2) is based on the chain rule of MI, and (3) is based on the non-negativity of MI. Now we have proven that $X_{high}$ has a higher CMI than $X_{low}$.

Theoretically, deeper neural networks lead to a lower conditional mutual information $\mathbb{I}(X;\hat{Y}|Y)$, as you said. Without considering the residual connections, neural networks can be regarded as a Markov chain: $Y \rightarrow X \rightarrow Z_{1} \rightarrow Z_{2} \rightarrow \ldots \rightarrow Z_{d} \rightarrow \hat{Y}$ where $Y$ is the true label, $X$ is the input, $Z_{1},Z_{2},...,Z_{d}$ are intermediate features, $\hat{Y}$ is the predicted label, and $d$ is the depth of the neural network. Based on the data processing inequality [1], there is $\mathbb{I}(X;Z_{1})\geq\mathbb{I}(X;Z_{2})\geq\ldots\geq\mathbb{I}(X;Z_{d})\geq\mathbb{I}(X;\hat{Y})$ which means that MI decreases as the neural network becomes deeper. We can prove that CMI also has a similar inequality: $\mathbb{I}(X;Z_{1}|Y)\geq\mathbb{I}(X;Z_{2}|Y)\geq\ldots\geq\mathbb{I}(X;Z_{d}|Y)\geq\mathbb{I}(X;\hat{Y}|Y)$ which means that CMI decreases as the neural network becomes deeper. So your concern is justified!

Fortunately, there are two factors that prevent CMI from being too small to optimize:

1. Deeper networks are often used for higher-dimensional inputs in practice. As we have demonstrated above, higher-dimensional $X$ brings higher CMI, offsetting the negative impact of deeper models.
2. Deeper networks have residual connections, which disrupt the Markov property. Their Markov chains should be changed into $Y \rightarrow X \rightarrow (Z_{1},Z_{2},\ldots,Z_{d}) \rightarrow \hat{Y}$ which greatly shortens the length of the data processing inequality.

To verify our theoretical analysis, we conducted quantitative experiments. We select models with different depths and train them on FaceSrube with different resolutions. We calculated their CMI on the test set, and the results are:

We can see that:

• CMI is higher in A than in B, confirming our theory that higher-dimensional inputs lead to a higher CMI.
• CMI is lower in B than in C, confirming our theory that deeper neural networks lead to a lower CMI.
• The three CMIs differ little and are on the same order of magnitude, indicating that CMIs are numerically stable and will not be too small to optimize.

In fact, the MIA robustness and model's utility of Experiment A are shown in High resolution and RLB results, and the ones of Experiment B and C are shown in Tables 1-3 in our paper, indicating that our defense hold effective in various input dimension and model depth.

Thank you again for asking such a profound question! In the process of thinking and responding, we realized that MI/CMI might be the key to explaining the effectiveness of residual connections! This is certainly worth a further special research. We will try to explain deep learning via information theory in the future.

If you have any more questions, please feel free to continue asking us.

[1] https://en.wikipedia.org/wiki/Data_processing_inequality

Thank you for careful review and positive feedback! We are honored that you appreciate our writing, ideas, experiments and applicability.

Below are our point-by-point responses to your questions.

The proposed defense could be susceptible to an adaptive attack. An adversary could query the same input multiple times to obtain multiple predictions from the model. Since the defense produces outputs by perturbing the original prediction, the adversary could compute an average over multiple outputs to get a better estimate of the model’s true output. Such an adaptive attack is not discussed by the paper.

Can you address the adaptive attack discussed in Weakness#1?

Thank you for pointing out an interesting point of adaptive attacks, which has inspired our deeper thinking. Our detailed discussion and solution is stated in Common Concerns on Adaptive Attacks. We will supplement the discussion in our paper.

The defense requires a validation dataset to implement, which could limit its adoption.

We conducted additional experiments (as stated above), and our conclusion is: if the model owner has not prepared a validation set, using the training set is also acceptable. This will not affect the robustness of MIA, but only slightly affect the model's utility.

In line 228, the authors state that “the objective function is too complex for the convex optimizer to solve.” and use this as the motivation to minimize a simplified objective $\mathbb{KL}(\pmb{p}||\pmb{q}^{y})$ instead (by sampling $y\in\mathbb{Y}$). Why is this the original objective too complex? Wouldn’t you be able to use gradient descent to solve for? Would this lead to a better solution?

The original optimization objective $\sum_{y\in\mathbb{Y}}{\mathbb{P}(y|\pmb{x})\sum_{\hat{y}\in\mathbb{Y}}{\mathbb{P}(\hat{y}|\pmb{x})\log\frac{\mathbb{P}(\hat{y}|\pmb{x})}{\mathbb{P}(\hat{y}|y)}}}$

is too complex because it involves $|\mathbb{Y}|$ vectors of length $|\mathbb{Y}|$, resulting in a space complexity and time complexity of $\Theta(|\mathbb{Y}|^2)$, which is unacceptable. After sampling $y\in\mathbb{Y}$, the simplified objective only involves one vector of length $|\mathbb{Y}|$, with a space complexity of $\Theta(|\mathbb{Y}|)$ and a time complexity of $\Theta(|\mathbb{Y}|\log|\mathbb{Y}|)$ for sorting, which is acceptable.

Convex optimization problems can indeed be solved by gradient descent, which is the approach taken by most famous optimizers. However, the computational cost of gradient descent is unacceptable because we require a solution for each prediction. Based on the simplified objective, we are able to propose water-filling to solve it, and efficiently implement it on GPUs. More details are stated in Common Concerns on Computational Cost.

In terms of mathematical expectation, our simplified form is equivalent to the original form. We sample based on probability $\mathbb{P}(y|\pmb{x})$, while the original form is a probability-weighted-sum. However, we introduce randomness, vulnerable to adaptive attacks as you mentioned. Our discussion and solution are stated in Common Concerns on Adaptive Attacks

How was the temperature $T$ picked in Algorithm 1?

What was the value of $\varepsilon$ for the proposed defense in the experiments?

All hyperparameters are listed in Table 4 in our paper, which is

Why is the Acc@1 for IR-152 lower for “no defense” compared to prior defenses? This suggests that adding prior defense improves the attack success rate, which is very strange.

That's exactly what happened. Previous defenses have become ineffective against the latest and advanced attacks, and may even facilitate them. This also highlights the importance of our work. In the Table 5 of the latest MIA survey [1], there are similar results.

Thank you again for your recognition! If you have any other questions, please feel free to ask us.

[1] Qiu, Yixiang et al. “MIBench: A Comprehensive Benchmark for Model Inversion Attack and Defense.” ArXiv abs/2410.05159 (2024): n. pag.

We would like to express our gratitude for your valuable suggestions and meaningful feedback. We are honored that you consider us to provide new insights into MIA field, with a rigorous theoretical structure and excellent experimental results.

Our detailed responses are provided below.

Key concepts, such as the water-filling method, lack adequate coverage in the main body, making it challenging for readers to fully grasp the methodology without delving into appendices. Greater emphasis on these aspects within the main text would improve the paper's accessibility.

Thanks for the advice. We fully agree with you. We will carefully revise our paper, placing the optimal solution of water-filling in the main text. Due to the page limitations, the mathematical derivation of water-filling has to be placed in the appendix.

The experimental setup omits certain critical information, such as the dataset used for GAN prior training and specifics for Figure 1, leaving questions regarding reproducibility and completeness.

Could the authors provide additional details on the GAN prior training and any specific hyperparameters used for Figure 1?

All our GANs are trained on the public dataset FFHQ. For each attack method, our GAN training is consistent with the original paper. Specifically, for Mirror and C2F, we use StyleGAN2-ADA [1], a well-known pre-trained GAN. Its weights can be downloaded on github. For BREP, we follow the official implementation to train a GAN, changing the public dataset from CelebA to FFHQ. For LOKT, we generate pseudo-labels by the target model (with each defense) and train an ACGAN [2]. The trainning details are consistent with the official code.

The detailed settings for Figures 1~2 are presented in Section 5.3. We use the VGG-16 trained on FaceScrub as the target model, consistent with Tables 1-3. Our defense hyperparameters are $T=0.1, ε=1.5$ as presented in Table 4. We employ TSNE implemented by sklearn.manifold, where we set metric=jensenshannon and all other parameters to be default. The CMI is calculated by Equation (2) in the paper.

Thank you again for pointing out these details. Your suggestions will greatly improve the quality of our paper. If our paper is accepted, we will open all code so that readers will have no concerns about reproducibility.

The study exclusively benchmarks against state-of-the-art white-box defenses but omits comparisons with black-box defenses, which would be more pertinent for the black-box MIA context.

Our competitors, MID, BiDO, LS and TL, can all be considered as special black-box defenses. Because they not only modify the model's weights, but also the model's outputs (including soft-labels and hard-labels). In fact, in the Table 8 of [3], the authors of LS conducted experiments on defending against black-box attacks, indicating that LS is designed for both white-box and black-box scenarios.

To our knowledge, our SSD is the first defense specifically designed for black-box scenario. Our paper thoroughly demonstrates the necessity and significance behind this. Our competitors, MID, BiDO, LS and TL, are all SOTA defenses (and naturally, SOTA black-box defenses as well). According to the latest MIA survey [4], we have compared with all the competitors that should be compared.

[1] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NeurIPS, 2020.

[2] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning, pages 2642–2651. PMLR, 2017.

[3] Struppek, Lukas et al. “Be Careful What You Smooth For: Label Smoothing Can Be a Privacy Shield but Also a Catalyst for Model Inversion Attacks.” ArXiv abs/2310.06549 (2023): n. pag.

[4] Qiu, Yixiang et al. “MIBench: A Comprehensive Benchmark for Model Inversion Attack and Defense.” ArXiv abs/2410.05159 (2024): n. pag.

Thank you for careful review and insightful comments, as well as your appreciation for our method and experiments. We have carefully read all your comments. Here are our replies one by one.

Please provide an explicit expression for $D_{valid}$.

We employ Leave-One-Out Cross-Validation (LOOCV), where one sample in $D_{train}$ is used to validate for each label. During training the target model, $D_{valid}$ is used to evaluate utility and adjust hyperparameters. After training, the latest $D_{valid}$ is used to estimate the $\pmb{q}^y$ in our defense. Thank you for the inquiry and we will include these details in our paper.

Furthermore, it would be interesting to understand the impact of $D_{valid}$’s size on the final output. Were all training images used as $D_{valid}$?

Based on the LOOCV setting above, we have $|D_{valid}|=|\mathbb{Y}|=530$ for FaceScrub dataset and $|D_{valid}|=|\mathbb{Y}|=1000$ for CelebA dataset. Your question has sparked our interest, so we conducted a new experiment by using all training samples as $D_{valid}$, i.e. $|D_{valid}|=|D_{train}|=27018$ for CelebA. All other experiment settings are the same as the Tables 1~3, where the target model is IR-152 and the dataset is CelebA.

The MIA robustness under Mirror attack:

And the MIA robustness under BREP attack:

We find that there is almost no difference in MIA robustness between the original and the new.

For model's utility, the evaluation results are

It can be seen that using $D_{train}$ as $D_{valid}$ leads to a decrease in model utility. We think the reason is: the estimating of $\pmb{q}^y$ by $D_{train}$ is not as accurate as the one by $D_{valid}$. However, the decrease is very minor and acceptable. If the model owner has not prepared validation set, we suggest they could use training set as validation set.

SSD requires $D_{valid}$, which I believe should be real data (either the training dataset or its validation set). This means the user must store raw training data or predictions on the training data to perform predictions, potentially increasing the risk of data leakage.

Under our careful design, model owners need not to store the complete $D_{valid}$, nor the raw predictions of private samples. In fact, they only need to store the average prediction for each label, i.e. ${\tilde{\pmb{q}}}^{y}\coloneqq\underset{(\pmb{x},y)\in D_{valid}}{\text{mean}}{\pmb{f}(\pmb{x})}$ for each $y\in\mathbb{Y}$. We believe that the averaging operation can greatly mitigate the risk of data leakage [1].

SSD's prediction process involves an optimization step for each image, leading to significantly increased computational costs and slower inference times compared to other models.

I would like to see a comparison of the prediction time between SSD and baseline methods like NoDef or other defenses.

As we stated in Common Concerns on Computational Cost, we have completely addressed this problem. Modifying 512 predictions for 100 times only needs 0.5 seconds with our careful optimization.

The experiments about RLB-MI and high-resolution are ongoing. The results will be available before the end of the discussion.

[1] Dwork, Cynthia. “Differential Privacy.” International Colloquium on Automata, Languages and Programming (2006).

We are pleased to reply to you with the results for high resolution and RLB, once again proving the superiority of our defense. We now address your remaining concerns.

please provide more details on how high-resolution MI attacks like MIRROR are adapted to low-resolution scenarios.

To adapt to low resolution, we use StyleGAN2 trained on FFHQ with a resolution of 256×256 [1]. The generated images are center-cropped to 176×176, resized to 64×64, and input into the target model. All other attack settings are consistent with the original code.

Can the author evaluate the proposed method's performance on high-resolution images?

Yes! We chose Mirror as the attacker. We use StyleGAN2 trained on FFHQ with a resolution of 1024×1024 [1]. The generated images are center-cropped to 800×800, resized to 224×224, and input into the target model. The target model is ResNet-152 and the evaluation model is Inception-v3. The first 10 classes of FaceScrub are attacked and each class reconstructed 5 images. The attack results are:

It can be seen that our defense has the best MIA robustness in high-resolution scenarios.

The target models' utility and defenses' hyperparameters are:

Due to the deadline, we have no time to adjust our hyperparameters to achieve the best $\text{max }L_{1}$. However, other metrics are sufficient to prove that we have the best model's utility in high-resolution scenarios.

[r1] was omitted in the paper. I suggest the author should add [r1] in the experiments

Following your suggestion, we supplement a new experiment on RLB. All attack settings are consistent with the original code [2]. The target model is IR-152, the evaluation model is Inception-v3, and the GAN is trained on FFHQ. The first 10 classes of CelebA are attacked and each class reconstructed 5 images. The attack results are:

It can be seen that our defense has the best MIA robustness against RLB. The target models' utility and defenses' hyperparameters are consistent with the Table 3~4 in our paper, which shows that we have the best model's utility, too.

The above two new experiments show that both in high-resolution and RLB-attack scenarios, our defense remains the best. We will add them to our appendix. If you have any other questions, please feel free to ask us.

[1] https://github.com/NVlabs/stylegan2

[2] https://github.com/HanGyojin/RLB-MI

Thank you for your advice! We will maintain consistency in the revised paper.

To clarify, are the reported results in this rebuttal for LS based on positive LS (α = 0.01) or negative LS?

We set $\alpha=-0.01$ for LS. Thank you for pointing out this typo and we have corrected it in the rebuttal.

We appreciate your feedback and contribution to improve our work! If all your questions have been addressed (validation set, time overhead, high resolution, and RLB attack), we kindly ask you to consider raising your rating. If you still have any doubts or reservations about our work, we are more than willing to engage in further discussion with you.

The discussion will end in 10 hours. If you still have any doubts or reservations (including but not limited to RLB attacks, high resolution, time consumption, validation sets), please leave a message before it ends. We are more than happy to respond promptly.

Thank you for providing a detailed response that addresses the concerns regarding computational cost. I appreciate the theoretical and empirical efforts you have undertaken to address the issue. Including them in the appendix will help clarify this point for potential readers.

Thank you for providing us a detailed comment regarding the concerns about adaptive attacks. I appreciate your explanation and the effort to propose a memory-free, low-cost improvement to enhance the robustness of your defense mechanism.

You make a valid point about the practical challenges attackers encounter in black-box scenarios. Indeed, launching adaptive attacks in such settings requires significant resources, and your argument about the realistic limitations attackers encounter strengthens the case for the feasibility of your defense. Furthermore, the introduction of the hash function h(x) to randomize the sampling process and ensure "same-input-same-output" is an intriguing idea. The connection to the similarity between features through z(x) is particularly compelling, as it aligns well with the intuitive understanding of feature-space robustness. I look forward to seeing these ideas expanded in the final version of the paper.