Consensus-Robust Transfer Attacks via Parameter and Representation Perturbations

W.1 The paper explicitly states that it follows prior work in reporting only average results without error bars or statistical significance tests. This is a significant methodological flaw, as it hinders the assessment of result reliability, especially given the variability inherent in adversarial attack experiments. For example, Table 1 and Table 2 report TSRs without confidence intervals, making it difficult to evaluate the consistency of CORTA's performance across runs or datasets.

A.1 Thank you for highlighting the importance of reporting error bars or statistical significance tests. While prior papers do not include error bars, we appreciate your suggestion, as providing confidence intervals offers a clearer assessment of result reliability—particularly given the inherent variability in adversarial attack experiments.

We re-ran the experiments from Table 1 and Table 2 on 100 randomly selected samples, repeating each experiment 20 times with different random seeds. The results are now reported as mean ± standard deviation. Please note that these are preliminary results; full experimental results will be included in the revised paper.

Table.1 :

Table.2:

W.2 While CORTA is tested with ResNet-18 and ViT-Tiny as surrogates, the baselines Ens and AdaEA use multiple surrogates (2 CNNs and 2 ViTs). This discrepancy makes direct comparisons less fair, as ensemble methods inherently leverage more diverse surrogate information. The paper would benefit from evaluating CORTA with multiple surrogates to better align with these baselines and demonstrate its robustness.

A.2 To ensure a fair comparison, we implemented "CORTA-ensemble" using the exact same surrogate set as Ens and AdaEA—ResNet-18, Inception-v3, ViT-Tiny, and DeiT-Tiny. The results on ImageNet are shown below:

In the identical 4-model ensemble setting, CORTA significantly outperforms both Ens and AdaEA, demonstrating its effectiveness in multi-surrogate scenarios and its ability to further enhance cross-architecture transfer.

Leveraging multiple surrogate models increases transfer success rates across both CNN-based and ViT-based target models. For example, CORTA improves transfer performance to ViT targets by 24.1% compared to using a single ResNet-18 surrogate. Similarly, compared to a single ViT-Tiny surrogate, transfer performance to CNN and ViT targets increases by 25.1% and 12.8%, respectively. This demonstrates that using multiple, diverse surrogates helps bridge the gap in transferability between very different model architectures.

W.3 This paper briefly mentions hyperparameters (e.g., perturbation probability $\rho = 0.5$, blending proportion $\lambda$ sampled from U[0.25,1], and trade-off coefficient $\beta$ tuned empirically), but lacks a detailed ablation study to justify these choices. For instance, the choice of $\lambda_{min}=0.25$ for feature blending is not explained, nor is the reananalysis of how sensitive CORTA's performance is to variations in $\beta$ or $\rho$. This omission limits the understanding of the method's robustness to hyperparameter settings.

A.3 In the following, we provide a more comprehensive explanation of the reasoning behind the selection of the key hyperparameters: $\rho$, $\lambda_{min}$, and $\beta$ -- we will revise the paper to better explain our selection of these hyperparameters.

1. Choice of $\rho$: We set $\rho = 0.5$ based on optimization success on the surrogate model. If $\rho$ is too high (e.g., 0.8), the feature perturbation becomes overly strong, disrupting gradient signals and reducing optimization success. Conversely, if $\rho$ is too low (e.g., 0.2), the perturbation is too weak, resulting in limited transferability to the target model. Therefore, we selected $\rho = 0.5$ for all experiments to achieve a balanced trade-off.

2. Choice of $\lambda_{min}$: Similarly, setting $\lambda_{min}$ too high or too low degrades optimization success on the surrogate model. We observed stable performance with $\lambda_{min}$ in the range [0.1, 0.3], and thus chose $\lambda_{min} = 0.25$ for all experiments in the paper.

3. Choice of trade-off coefficient $\beta$: $\beta$ is chosen to balance the magnitudes of the two optimization losses in the surrogate model, ensuring that their contributions are comparable. Consequently, $\beta$ is not task- or dataset-specific, but rather related to the model architecture. For example, we used $\beta = 0.1$ for ResNet-18 on both ImageNet and CIFAR-100. Additionally, our hyperparameter sensitivity analysis (see Figure 1 and Line 297 of the paper) demonstrates that $\beta$ yields stable results within the range [0.01, 0.1].

We address the sensitivity of these hyperparameters in Lines 294–299 of the paper and provide visualizations in Figures 1, 2, and 3 of the paper, which illustrate the effects of varying $\beta$, $\rho$, and $\lambda_{min}$. As shown, CORTA maintains stable performance for $\beta$ in the range [0.01, 0.1], $\rho$ in [0.5, 1], and $\lambda_{min}$ in [0.1, 0.3].

We hope this clarifies our hyperparameter choices and their sensitivity. We will ensure this discussion is included in the revised paper.

W.1.1 Line 148: "small $\Delta W$ - a very weak hypothesis as the setup is a black-box and targeting any neural network

A.1 For $\Delta W$, we lower the upper bound to minimize loss variations across models (see Eq. at line 170 of the paper, which holds for all $\|\Delta W\|_{F} \leq \rho$, note that $\rho$ can be large). This approach addresses the worst-case scenario, ensuring the DRO objective is satisfied even under the most adverse conditions. We have made a minor correction to the equation at line 170 in the revised paper, but the underlying idea remains unchanged.

W.1.2 Line 173: a Lipschitz constant $L_l$ can be quite huge so the last term in the optimization objective (6) can be much higher than the $||g_{W}(x, \delta)||_{F}$

A.2 The Lipschitz constant $L_{l}$ is a model-dependent term that appears in the upper bound of the optimization objective, but it does not directly participate in the optimization of adversarial samples. It is independent of the gradient term $||g_{W}(x, \delta)||_{F}$, which is minimized during optimization. Therefore, even if $L_l$ is large, CORTA only optimizes components directly related to the perturbation $\delta$, and does not rely on $L_l$. As a result, the magnitude of the Lipschitz constant has no impact on our optimization process.

W.2 No real data (e.g., a comparison table) for the optimization success rate - it was briefly mentioned in lines 308-309, but no the real analysis.

A.3 We present the following comparison table, which shows the optimization success rates of baseline methods on ImageNet using ResNet-18 as the surrogate model:

As shown in the table, most baseline methods (Ens, AdaEA, DHF, BFA) achieve optimization success rates close to 100%, whereas our method (CORTA) achieves a success rate of 69.9%.

CORTA’s surrogate success rate is lower than that of baseline attacks mainly due to two factors inherent to its DRO formulation: (1) CORTA optimizes two objectives—representation and parameter channels—rather than a single objective, increasing optimization difficulty; and (2) the feature blending operation, which incorporates the original sample’s latent features, can interfere with the adversarial objective and further reduce success rates on the surrogate model.

However, this lower surrogate success rate is not a practical issue. Attackers can simply discard unsuccessful adversarial examples using the surrogate model and retain only those that succeed, which tend to have higher transfer success rates. In practice, this only slightly increases computational cost, as generating the same number of successful examples requires optimizing about 1.43 times more samples (e.g., 100/69.9 compared to attacks with 100% surrogate success rate).

We will include these updated results and explanations in the revised version.

W.3 Minor remarks

A.4 We will make all necessary revisions in accordance with your valuable suggestions.

Q.1 Why is stochastic feature blending applied when the output size is $\leq 1/16$ of the input (lines 244-245)? No reason behind it, no ablations What about transformers?

A.5 We adopted the "output size $\leq 1/16$ of the input" setting to align with the CFM method ([21] in the paper), which is similar to DHF but focusing on targeted adversarial attacks.

However, this constraint is not essential. Our experiments show that applying stochastic feature blending to all layers—using ResNet-18 as the surrogate—still outperforms baseline methods, as shown in the table below:

or Transformer architectures, we apply stochastic feature blending to all linear layers. This setting will be clearly explained in the revised version of the paper.

Overall, this parameter is not critical, and we will update its description accordingly in the revision.

Q.2 Unclear why the blending probability $\rho = 0.5$ is chosen from the ablation study (Figure 2), because it seems that the best is somewhere around 0.8?

A.6 Thank you for this insightful question. Our choice of $\rho = 0.5$ was based on optimization performance on the surrogate model. When $\rho$ is too high (e.g., 0.8), the feature perturbation can overly disrupt gradient signals, reducing optimization success on the surrogate. Conversely, when $\rho$ is too low (e.g., 0.2), the perturbation is insufficient, limiting transferability. Thus, $\rho = 0.5$ was selected as a balanced value.

While the ablation study indicates that $\rho = 0.8$ achieves better performance on the target model, our choice of $\rho = 0.5$ does not yield the highest transfer success rate on the target. However, as black-box attackers, we cannot tune hyperparameters based on the target model. Therefore, we selected $\rho = 0.5$ based on the optimization success rate on the surrogate model, even though it may not be optimal for the target model.

Thank you for your thoughtful review and positive feedback. We are glad to hear that the theoretical foundation and empirical results of CORTA were well-received. We appreciate your recognition of the strengths of the paper, and we will continue to refine and expand on the ideas presented in future work.

Q.1 How does CORTA's computational cost scale with model size (e.g., ViT-Large)? Could approximations mitigate this?

A.1 The computational cost of CORTA is closely related to the model size. As the model size increases, the time required to optimize each image also increases. The following table illustrates the optimization time per image using various surrogate models on ImageNet:

As shown in the table, the smallest model (ResNet-18) requires only 1.7 seconds of optimization time per image, while the largest model (ViT-Large) requires 5.1 seconds. Notably, even with the ViT-Large model, CORTA’s computational time remains within an acceptable range compared to ensemble methods that utilize multiple smaller models (Ensemble: 5.2s, AdaEA: 18.8s). Furthermore, CORTA achieves high transfer success rates even when using smaller models.

Thank you for the valuable suggestion regarding the use of approximations to mitigate computational costs. Reducing computational overhead is indeed a central focus of our ongoing research. In particular, we are actively exploring gradient approximation techniques to minimize computational expenses. Our goal is to develop a more lightweight approach for generating adversarial examples, while still maintaining high effectiveness.

Q.2 Optimization Trade-off: Why does CORTA's surrogate success rate drop compared to baselines? Is this inherent to the DRO formulation?

A.2 CORTA’s surrogate success rate is lower than that of baseline attacks mainly due to two factors inherent to its DRO formulation: (1) CORTA optimizes two objectives—representation and parameter channels—rather than a single objective, increasing optimization difficulty; and (2) the feature blending operation, which incorporates the original sample’s latent features, can interfere with the adversarial objective and further reduce success rates on the surrogate model.

However, this lower surrogate success rate is not a practical issue. Attackers can simply discard unsuccessful adversarial examples using the surrogate model and retain only those that succeed, which tend to have higher transfer success rates. In practice, this only slightly increases computational cost, as generating the same number of successful examples requires optimizing about 1.43 times more samples (e.g., 100/69.9 compared to attacks with 100% surrogate success rate).

Q.3 Hyperparameter generalization: (1) Are the chosen hyperparameters (e.g., $\beta$=0.1) task-specific, or can they be generalized? An analysis on additional datasets would help. (2) eps = 16/255 is quite large in AEs. What are the results for typical, smaller values of eps?

A.3 (1) $\beta$ is chosen to balance the magnitudes of the two optimization losses in the surrogate model, ensuring that their contributions are comparable. Consequently, $\beta$ is not task- or dataset-specific, but rather related to the model architecture. For example, we used $\beta = 0.1$ for ResNet-18 on both ImageNet and CIFAR-100. Additionally, our hyperparameter sensitivity analysis (see Figure 1 and Line 297 of the paper) demonstrates that $\beta$ yields stable results within the range [0.01, 0.1].

(2) We selected $\epsilon = 16/255$ in our main experiments to align with baseline methods such as Admix, DHF, and BFA, since black-box attacks are considerably more challenging than white-box attacks. To further assess the robustness and generalizability of our method, we also conducted experiments with a smaller, more typical value of $\epsilon$ ($\epsilon = 8/255$).

For these experiments, we used ResNet-18 as the surrogate model on the ImageNet dataset, setting $\epsilon = 8/255$, step size $\alpha = 0.8/255$, and number of iterations $T = 100$. The table below presents the Transfer Success Rate (TSR) of CORTA and various baseline methods under these settings:

The results show that even with a reduced perturbation budget ($\epsilon = 8/255$), CORTA consistently outperforms baseline methods. This demonstrates that our approach maintains strong attack effectiveness within a smaller perturbation range, further validating the robustness and generalization capability of CORTA.