Adversarial Machine Learning in Latent Representations of Neural Networks

Regarding Weakness 1 & Question 1

On the claim that "assuming the same level of information distortion, latent features are always more robust than input representations", does this hold even if the latent embedding dimension is larger than the input dimension?

We thank the reviewer for this comment. To this end, we remark that Theoretical Finding #2 indeed holds when the latent embedding dimension is larger than the input dimension. Our experiments shown in Figure 4 include the original VGG and ResNet, where the space is respectively $\times 5.33$, $\times 2.66$ larger.

To better prove our point, we are actively working on a new experiment with respect to the depth of latent space in the revision. We present some preliminary results in Table 1 and Table 2.

Table 1: Adversarial robustness of ResNet152 as a function of depth with perturbation budget $\epsilon = 0.003$

Table 2: Adversarial robustness of VGG16 as a function of depth with perturbation budget $\epsilon = 0.003$

The experiment is based on a smaller model VGG16 and a deeper model ResNet152. Same level of $l_\infty$ norm distortions are introduced to all feature maps before max pooling layers. As shown in Table 1 and Table 2, all the latent feature maps are more robust than the input regardless of dimensionality and depth. Furthermore, ASR reduces with respect to the depth of latent space, which is in line with our Theoretical Finding #2.

If yes, why it can be more robust?

This finding holds because it is derived directly from the chain rule of mutual information which does not depend on cardinality $|\mathcal{T}|$.

it has been proven in the ICML 2019 paper "First-order Adversarial Vulnerability of Neural Networks and Input Dimension" that the minimal adversarial perturbation scales inversely with the data dimension.

We thank the reviewer for bringing [1] to our attention. We want to address the key difference between the work [1] and our Theoretical Finding #1. In our Theoretical Finding #1, the robustness is jointly determined by its upper bound $I^*(Y;T)$ and $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$. Although $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$ indicates that a larger cardinality will result in less robustness, which is in line with [1], $T$ is a DNN-specific space depending on the DNN architecture design. In stark opposition, Theoretical Finding #1 addresses the variance-bias tradeoff in the distributed DNN design, which is also supported by our experiments as shown in Figure 6. Moreover, it also explains why latent space is more robust even if it has larger cardinality, since the latent robustness depends both on the cardinality and $I^*(Y;T)$.

Regarding Weakness 2 & Question 2

The assumption on the same level of information distortion seems very strong and lacks justification

We thank the reviewer for this insightful comment. We remark that such constraints on information distortion are necessary to formally analyze robustness of distributed DNNs and provide a fair comparison with attacks in the input space. Moreover, although there is much work focused on unconstrained attacks – such as [2, 3, 4] —, the constrained distortions are still a popular way to evaluate robustness [5, 6]. Since this is the first work to investigate this topic, we aim at formally analyzing the robustness with information constraints. However, we also point out that in real-world scenarios the attacks in the latent space can have a different level of information than the input. We will clarify this crucial aspect in the final version of the paper.

It's also not clear what is the threat model (what the attacker can do) that leads to a latent representation $T_{adv}$

In a distributed DNN scenario, latent representations are sent to the local DNN that can be exposed to attackers. We refer the reviewer to Figure 1 for a visual depiction of the attack. We assume that attackers can directly add perturbations to latent representations in the same manner as attacks performed in the input space. Thus, $T_{adv}$ denotes the adversarial latent representation generated by attackers. For a detailed attack model formulation, we kindly refer the reviewer to Section 4.1.

I thank the reviewer for providing the responses. Although some of my concerns have been partially addressed, some major concerns remain:

1. Lack of robustness evaluation on adaptive attacks - without this experiment, it's hard to argue, even numerically, that the analysis and the assumptions hold in practice. As I pointed out, many earlier defenses based on latent representation regularization were later broken by adaptive attacks. This setting should apply to the considered distributed DNN setting as well.
2. I appreciate the new experiments from authors. However, it is implemented with a very small $\epsilon$ budget and on simple attacks like FGSM and PGD. It would be more convincing to try on larger $\epsilon$ values with AutoAttack.

Regarding Question 3

Did you think about analyzing the latent features robustness in LLMs?

We believe the adversarial robustness of latent features in LLMs is an intriguing and extremely timely topic, and we are planning to have a separate investigation into this matter.

What do you think the challenges would be?

We believe one big challenge could be the intrinsic difference between textual and image data. For example, textual data are non-differentiable due to their semantic nature, while image data are differentiable RGB values. As a result, the problem formulation cannot be directly applied to LLMs.

Regarding Question 4

What would be the key element in designing defenses for this attack?

We believe the latent representations intrinsically have more semantic information compared to input. And there is evidence that adversarial input and benign input have different distributions in hidden layers [4]. We believe this can also be generalized to latent representations. In order to develop defense specifically for latent representations, we believe one key element is to understand the semantic meaning of the distribution change caused by adversarial perturbations.

Reference:

[1] Zhang, Gaoyuan, et al. "Distributed adversarial training to robustify deep neural networks at scale." Uncertainty in Artificial Intelligence. PMLR, 2022.

[2] Wong, Eric, Leslie Rice, and J. Zico Kolter. "Fast is better than free: Revisiting adversarial training." arXiv preprint arXiv:2001.03994 (2020).

[3] Hong, Sanghyun, et al. "A panda? no, it's a sloth: Slowdown attacks on adaptive multi-exit neural network inference." arXiv preprint arXiv:2010.02432 (2020).

[4] Galloway, Angus, et al. "Batch normalization is a cause of adversarial vulnerability." arXiv preprint arXiv:1905.02161 (2019).

Regarding Weakness 1 & Question 1

The paper does not take into consideration attacking the latent representations of a DNN that was adversarially trained.

We thank the reviewer for pointing this out. We have developed a new experiment to evaluate the input and latent robustness of two adversarial training approaches presented in recent work [1] and [2] in Table 1.

Table 1: Robustness of ResNet50, DAT [1] and FastAT [2] with perturbation budget $\epsilon = 0.003$

In Table 1, DAT and FastAT are pretrained ResNet50 with different adversarial training approaches. We evaluate the robustness in both input and latent space as we did with the original ResNet50. The results show that latent representations of adversarially trained DNNs are also more robust than input, which is in line with our theoretical findings.

how do you think the adversarial training would change the success rate of these attacks?

We have evaluated this aspect from an experimental standpoint in the table above. From a theoretical perspective, while the adversarial training should be able to improve the robustness in both input and latent space to some extent, it will not affect our Theoretical Finding #2. Indeed, in our Theoretical Finding #1, the upper bound is defined by a term $\mathcal{O}( |\mathcal{X}||\mathcal{Y}| / \sqrt{n})$, where n is the number of samples. Adversarial training actually introduces more samples, resulting in a smaller $\mathcal{O}( |\mathcal{X}||\mathcal{Y}| / \sqrt{n})$ as well as a larger upper bound in robustness. We also point out that our Theoretical Finding #2 holds irrespective of the sample size n, therefore adversarial training will not impact this finding. We are going to include these explanations in the final version of the manuscript.

Regarding Question 2

Recently there was another adversarial attack introduced for slightly more particular DNNs architectures with multi-exits (or early-exits)

We thank the reviewer for bringing [3] to our attention. We point out that [3] can be explained with information bottleneck theory which is related to our Theoretical Finding #2. We explain it as follow:

It is pointed out in [3] that conventional adversarial attacks cannot slow down the early-exit networks. Since we proposed to measure distortion with residual information $I(X;Y|T)$, the residual information of adversarial samples $I(X_{adv};Y|T’)$ will be larger than the residual information of benign samples $I(X;Y|T)$, i.e.,

$I(X_{adv};Y|T') \geq I(X;Y|T)$

From Equation 8 in our paper, it follows that

$I(X;Y) - I(X_{adv};Y) \leq I(T;Y) - I(T’;Y).$

In other words, the information loss between latent $T’$ and ground-truth $Y$ is larger than the information loss between input $X_{adv}$ and $Y$. Thus, a small information loss gets amplified in hidden layers, which is consistent with Figure 3 in [3].

It follows that in early-exit models, the adversarial samples and benign samples will have less distortion in early layers, and thus early exits can be activated.

We thank the reviewer for pointing out [3] which could help us enhance our theoretical aspect. We are working on a revision to include [3] and the discussion.

I would like to thank authors for addressing my comments.

About Table 1. I think these results should be added to the paper because I believe they will add value to the paper for the simple fact that attacks on latent features are even less successful than attacks on inputs for adversarially trained networks.

I agree with the authors' explanations to my questions, they are intuitive and I really appreciate the effort you put into this. As a result, I will increase my score to 8.

Regarding Weakness 1

the appropriateness of imposing an identical distortion level in this context needs more clarification

We thank the reviewer for this insightful comment. We point out that estimating the distortion is necessary to formally compare the robustness of the DNN against adversarial attacks in the latent space with respect to attacks in the input space. Specifically, to compare input space and latent space attacks, we assume an identical adversarial effort in both input and latent space through the $l_p$ norm constraint. We point out that the $l_p$ norm is routinely used as a metric to characterize such effort, as it is used in existing work, for example [1, 2, 3]. Moreover, we also point out that in real-world scenarios the attacks in the latent space can (and most likely, will) have a different level of information than attacks in the input space. We will clarify this crucial aspect in the final version of the paper.

with no human observer in the loop, the attacker might as well nullify all latent features, potentially achieving an ASR close to 100%

We strongly agree with the reviewer that without a human observer in the loop, unconstrained perturbations in latent space can lead to ~100% ASR. However, we also believe latent representations intrinsically contain more semantic information than the input, which may result in easier outlier detection. For example, image data usually has redundant information such as background. Therefore, changing one pixel in the background might not be perceptible by humans. Conversely, since semantic representations capture only the sufficient statistics of the input, changing one element of them can result in invalid interpretations (take one-hot encoding as an extreme case example). As such, an outlier detection system which can estimate the perturbation magnitude in latent space can play a similar role as the human observer in input space. We will definitely delve deeper into this intriguing aspect in our future work as it deserves a separate and exhaustive investigation.

Regarding Weakness 2

I would appreciate it if the authors could highlight the distinctions between their work and prior research

We appreciate the reviewer bringing [4] to our attention. We point out that paper [4] uses the latent representation as a help to craft the adversarial input. In stark opposition, our work focuses on the perturbation of the latent space itself. We are working on a revision to address the distinction between our work and suggested work [4].

and attempt to apply the attack methods delineated in [4] where feasible.

We thank the reviewer for suggesting to compare our paper to [4], which we definitely plan to do in future work.

Regarding Weakness 3

I am confused about two terminologies in the paper: feature compression and bottleneck.

We are sorry for the confusion. In the literature, we refer to “bottleneck” as a neural network architecture that can be trained in different ways to reduce the dimensionality of feature representations. For this reason, we present different training strategies for bottlenecks – Supervised Compression (SC), Knowledge Distillation (KD), Entropic Student (ES) and BottleFit (BF).

Conversely, other feature compression approaches such as JEPG Compression (JC) and Quantization (QT) compress features by digitizing the feature map in its frequency domain and original domain respectively. These approaches compress the data size but are not able to reduce the dimensionality of the latent space.

In short, feature compression can be done in different ways, compressing the dimensionality (bottleneck) and compressing the data size (JC and QT).

Additionally, the authors use the same bottleneck design as Matsubara et al. (2022b) and denote the new architectures with feature compression as Resnet50-fc, etc. However, the specific design mentioned in Matsubara et al. (2022b) does not seem to belong to any of the six feature compression approaches in Table 1.

Thank you for your remark. Since there can be different bottleneck architectures, we choose the same architecture that is used in [5] which is a benchmark paper. Notice that similar to our work, [5] did not present the architecture but different training strategies as different approaches. However, in their codebase, they use a consistent bottleneck design.

Table A-3 is also confusing, as I cannot understand what the authors mean by 'JC and QT are DNNs without a bottleneck,'

We are sorry for the confusion. Since JC and QT are not related to the DNN architecture, such approaches can be add-ons for any DNN with or without bottleneck to reduce the communication overhead. However, as we want to verify the point that dimensionality helps to enhance latent robustness, we choose JC and QT without bottlenecks in our experiment for comparison.

I appreciate the detailed explanations and great efforts made by the authors. Thus, I increased my score to 6. However, as mentioned in the reply, the explanations need numerical experiments to support the claims, which can give strong motivation of this work.

Regarding Weakness & Questions

However, I am concerned about whether the distributed DNNs are a necessary background as the motivation to investigate the problem. Since I think the local DNN and mobile DNN are very like the architecture of an autoencoder, and there are many works about the adversarial robustness of autoencoders.

We agree with the reviewer that distributed DNNs have an encoder-decoder structure similar to autoencoders, and we appreciate the reviewer pointing out adversarial autoencoders. However, there is a stark distinction between robust distributed DNNs and robust autoencoders. In robust autoencoder, adversarial training is used to enforce the encoder to learn adversarially robust latent representations – for example, for speaker de-identification [1] or outlier detection [2]. Such latent representations are not considered to be exposed to attackers thus the “robustness” in the robust autoencoder context is actually “semantically meaningful representations”.

In our case, we consider attacks adding perturbations directly to the latent space. The scenario considered is extremely relevant to mobile scenarios since distributed DNNs have been proven to be extremely effective in reducing network load while preserving accuracy [3, 4, 5]. In this case, latent representations are exposed to the attacker, which can lead to performance degradation. To this end, the “robustness” in latent representation means “adversarially robust to attacks”, which is a remarkably different concept than robust autoencoders.

To the best of the authors’ knowledge, there is no such prior work to investigate adversarial perturbation in latent representation. Thus, we would like to highlight the significance and timeliness of our work in the context of distributed DNN.

We appreciate the constructive feedback to help us clarify our novelty and contribution. We are actively working on a revision that includes the suggested work [1] and clarification.

Reference:

[1] Espinoza-Cuadros, Fernando M., et al. "Speaker de-identification system using autoencoders and adversarial training." arXiv preprint arXiv:2011.04696 (2020).

[2] Salehi, Mohammadreza, et al. "Arae: Adversarially robust training of autoencoders improves novelty detection." Neural Networks 144 (2021): 726-736.

[3] Matsubara, Yoshitomo, et al. "Bottlefit: Learning compressed representations in deep neural networks for effective and efficient split computing." 2022 IEEE 23rd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 2022.

[4] Matsubara, Yoshitomo, et al. "Supervised compression for resource-constrained edge computing systems." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[5] Shao, Jiawei, and Jun Zhang. "Bottlenet++: An end-to-end approach for feature compression in device-edge co-inference systems." 2020 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2020.

Regarding Questions

For a correct evaluation of the suggested bound on a given DNN architecture, the experimental settings should present attacks on all the latent spaces in the architecture and not only those available in specific distributed DNN settings.

We strongly agree with the reviewer that to support our general theoretical findings, the attacks should be performed on all the latent spaces in different depths, and we are currently working to include this experiment. We present some preliminary results in Table 1 and Table 2. We are going to add more results in our final revision. As far as the latent cardinality is concerned, we do have an experiment in the current paper which is shown in Figure 6.

Table 1: Adversarial robustness of ResNet152 as a function of depth with perturbation budget $\epsilon = 0.003$

Table 2: Adversarial robustness of VGG16 as a function of depth with perturbation budget $\epsilon = 0.003$

We choose one simple model VGG16 and one relatively deeper model ResNet152 for the experiment. In VGG16 and ResNet152, multiple feature extraction blocks which consist of several identical convolutional layers are used to extract features. After each block a max pooling layer is used to reduce the feature dimensions at different depths. We report the ASR in all feature maps generated by feature extraction blocks. As shown in Table 1 and Table 2, irrespective of the depth and dimensionality, latent representations are always more robust than input, which can support our theoretical analysis.

As the effectiveness of perturbations depends on the input size, the results should be normalized accordingly.

We completely agree that the effectiveness of perturbations depends on the input size. For this reason, in the first version of the paper, we have rescaled the perturbation with respect to cardinality, similar to what is done in Equation 3 in [7]. Please refer to Section 4.1 for more details. Furthermore, we are following the literature where it is not common to normalize the Attack Success Rate.

Regarding Weakness 1

The experimental setup is lacking and insufficient to support the authors' claims. Only attacks on several distributed DNN architectures were reported.

We appreciate the reviewer’s concern. On the other hand, we point out that it is hardly feasible to investigate each and every distributed DNN architecture since each of them exhibits its own customized design for feature compression and model partitioning.

Moreover, it is well known that neural networks share similar robust and non-robust features in the same dataset [1]. Thus, it has been shown that adversarial samples generated by one model can transfer to another [2, 3, 4]. For this reason, the target of this paper is to provide theoretical proof of the robustness of distributed DNNs and evaluate the soundness of the proof on the 6 most widely-used approaches in recent years derived from the VGG and ResNet family. This was the same procedure followed by other benchmarking papers in the distributed DNN community [5] and adversarial machine learning community [6].

Thus, we think that the combination of theoretical proof and experimental evaluation put forth by the paper provides a strong contribution to the field. We believe that the extensive evaluation of 6 distributed DNN approaches, 6 model architectures as well as 10 attack algorithms is considered a sufficient experimental setup.

Regarding Weakness 2

No novel attacks targeting latent spaces were suggested, or even settings in which both the input and latent space are attacked

Thank you for your comment. We point out that proposing new adversarial attacks is out of the scope of the paper, which is instead evaluating the robustness of distributed DNNs to existing attacks, while achieving a fundamental understanding of the robustness of latent spaces.

Moreover, for a fair comparison, we needed to guarantee that each attack we performed could be applied to both input and latent space. Proposing new and advanced attacks specifically tailored to distributed DNNs – for example, the suggested joint input/latent space attacks – is an intriguing direction and deserves a separate and dedicated investigation.

Regarding Weakness 3

The second key finding of "DNN robustness is intrinsically related to the cardinality of the latent space" is not a phenomenon exclusive to latent spaces.

We agree that there is other work with similar findings in input space [7]. However, our Theoretical Finding #1 is different from [7] because in our analysis the upper bound is jointly determined by $I^*(Y;T)$ and $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$. Theoretical Finding #1 focuses on the variance-bias tradeoff in distributed DNN designs, while previous work only focuses on the input dimension. We prove this novel finding with experiments shown in Figure 6.

On the other hand, if we limit our scope to data space $X$, the robustness provided by dataset $X$ can be described by its upper bound $I^*(Y;X)$ and $\mathcal{O}( |\mathcal{X}||\mathcal{Y}| / \sqrt{n})$. Since $X$ represents the dataset, $I^*(Y;X)$ is constant. The adversarial robustness for a given dataset is directly described by the cardinality, which is in line with the findings reported in [7]. For this reason, our analysis from the information perspective can also generalize to input space.

Reference:

[1] Ilyas, Andrew, et al. "Adversarial examples are not bugs, they are features." Advances in neural information processing systems 32 (2019).

[2] Liu, Yanpei, et al. "Delving into transferable adversarial examples and black-box attacks." arXiv preprint arXiv:1611.02770 (2016).

[3] Dong, Yinpeng, et al. "Evading defenses to transferable adversarial examples by translation-invariant attacks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[4] Xie, Cihang, et al. "Improving transferability of adversarial examples with input diversity." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[5] Matsubara, Yoshitomo, et al. "SC2 Benchmark: Supervised Compression for Split Computing." Transactions on Machine Learning Research (2023).

[6] Dong, Yinpeng, et al. "Benchmarking adversarial robustness on image classification." proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[7] Simon-Gabriel, Carl-Johann, et al. "First-order adversarial vulnerability of neural networks and input dimension." International conference on machine learning. PMLR, 2019

5. Additional Citations

Adversarial Auto-encoders:

[1] Espinoza-Cuadros, Fernando M., et al. "Speaker de-identification system using autoencoders and adversarial training." arXiv preprint arXiv:2011.04696 (2020).

[2] Latif, Siddique, et al. "Multi-task semi-supervised adversarial autoencoding for speech emotion recognition." IEEE Transactions on Affective computing 13.2 (2020): 992-1004.

[3] Sahu, Saurabh, et al. "Adversarial auto-encoders for speech based emotion recognition." arXiv preprint arXiv:1806.02146 (2018).

[4] Salehi, Mohammadreza, et al. "Arae: Adversarially robust training of autoencoders improves novelty detection." Neural Networks 144 (2021): 726-736.

[5] Pidhorskyi, Stanislav, Ranya Almohsen, and Gianfranco Doretto. "Generative probabilistic novelty detection with adversarial autoencoders." Advances in neural information processing systems 31 (2018).

[6] Beggel, Laura, Michael Pfeiffer, and Bernd Bischl. "Robust anomaly detection in images using adversarial autoencoders." Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part I. Springer International Publishing, 2020.

[7] Makhzani, Alireza, et al. "Adversarial autoencoders." arXiv preprint arXiv:1511.05644 (2015).

Latent-aware Adversarial Attacks:

[8] Yu, Yunrui, Xitong Gao, and Cheng-Zhong Xu. "Lafeat: Piercing through adversarial defenses with latent features." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[9] Sabour, Sara, et al. "Adversarial manipulation of deep representations." arXiv preprint arXiv:1511.05122 (2015).

[10] Inkawhich, Nathan, et al. "Feature space perturbations yield more transferable adversarial examples." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[11] Huang, Qian, et al. "Enhancing adversarial example transferability with an intermediate level attack." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[12] Inkawhich, Nathan, et al. "Transferable perturbations of deep feature distributions." arXiv preprint arXiv:2004.12519 (2020).

[13] Luo, Cheng, et al. "Frequency-driven imperceptible adversarial attack on semantic similarity." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[14] Mopuri, Konda Reddy, Utsav Garg, and R. Venkatesh Babu. "Fast feature fool: A data independent approach to universal adversarial perturbations." arXiv preprint arXiv:1707.05572 (2017).

[15] Wang, Zhibo, et al. "advpattern: Physical-world attacks on deep person re-identification via adversarially transformable patterns." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

DeepSloth:

[16] Hong, Sanghyun, et al. "A panda? no, it's a sloth: Slowdown attacks on adaptive multi-exit neural network inference." arXiv preprint arXiv:2010.02432 (2020).

Adversarial Training:

[17] Zhang, Gaoyuan, et al. "Distributed adversarial training to robustify deep neural networks at scale." Uncertainty in Artificial Intelligence. PMLR, 2022.

[18] Wong, Eric, Leslie Rice, and J. Zico Kolter. "Fast is better than free: Revisiting adversarial training." arXiv preprint arXiv:2001.03994 (2020).

Adversarial vulnerability is related to dimensions:

[19] Simon-Gabriel, Carl-Johann, et al. "First-order adversarial vulnerability of neural networks and input dimension." International conference on machine learning. PMLR, 2019.