Adversarial Inputs for Linear Algebra Backends

Thank you for your feedback on our paper!

Experiments with larger models. We have extended our evaluation to include the ImageNet dataset, using the more complex architectures ResNet18 (Top-1 Accuracy: 69.7%) and EfficientNetV2S (Top-1 Accuracy: 84.2%). For these experiments, we used a reduced backend set (OpenBLAS, MKL, BLIS, and cuBLAS) and a smaller sample size of 128. In these new experiments, our attack achieves a 100% success rate for CPU-GPU backend combinations on both models. For CPU-CPU backends, we observe success rates of approximately 29% for ResNet18 and 22% for EfficientNetV2S. We will include these results in the paper.

Loss function in our attack. Thank you for pointing out an issue in our presentation. Indeed, the description of the loss functions in Section 3.3 is imprecise, as it omits a key point: Following Definition 3.1, we define the target labels of the backends in opposition, such that always $y_i \neq y_j$. This represents an important distinction from the standard search for adversarial examples. Moving the input towards $y_i$ usually moves it away from $y_j$, and vice versa. This “tug of war” between the target labels effectively pulls the input closer to the decision boundary. We will add the precise definition of the target labels to our paper.

Role of quantization function. We performed an ablation study to assess the influence of the quantization function on our attack. In this experiment, we compute the gradient of our attack without applying quantization. We find that our attack performs worse in this setting. Quantization is particularly important for CPU-based backends, yielding approximately 2.2× more Chimera samples for CIFAR and between 1.1× and 7× more for FMNIST. For GPU-based backends, we observe no such benefit. In fact, quantization slightly reduces performance by about 1-2%. We will include these results in our paper.

Intuition of search algorithm. Our search algorithm brings us close to the decision boundary and then proceeds by alternating back and forth using the conflicting loss functions described above. Theoretically, for every gradient direction we take, there exists a point that lies exactly on the decision boundary and hence is potentially close to a Chimera example. By alternating through these directions in a tug-of-war manner, we converge toward the region where numerical deviations effectively determine the final decision.

Improvement over baselines. All baselines were given the same iteration budget of 3000 steps as our attack, except for binary search that ran until convergence. Despite the same budget, our attack outperforms the baselines for the following reasons:

1. Although binary search can approach the decision boundary arbitrarily closely, it often lands in infeasible regions, making it impossible to elicit the required numerical differences from input samples.
2. The Carlini & Wagner attack was designed to target a single backend and searches for a point close to—but still across—the decision boundary. In contrast, our method explicitly targets the decision boundary and seeks inputs that yield conflicting predictions across backends.
3. The original boundary sample search, like binary search, was not designed to produce valid inputs. While it can find boundary points in feature space, the corresponding input samples often do not exist in the input domain.

Chimera examples in practice. Chimera examples pose a threat whenever two different systems evaluate the same model on the same data. For instance, in forensic investigations, it is no longer sufficient to use only the same model and input to replicate an incident—such as a malware detection, the censoring of media content, or the decision of a hiring decision. Instead, the entire original system—including all backend libraries—must be fully replicated to ensure consistent and reliable results.

Noisy gradient descent. When evaluating our defense, we already assume an attacker who is aware of the defense and uses noisy gradients. A key distinction of our defense is that the noise is deterministic and unique for each data point—that is, the noise remains fixed for the same input and key. As a result, an attacker cannot accumulate additional information by averaging over multiple runs. The noise creates a shattered view of the feature space, making fine-grained gradients ineffective except by chance. Our experiment in Section 5 shows that such chance-based success is too low to be practically viable.

Supplementary Material. We have uploaded our source code to https://gitlab.com/anonymized-code/2025-icml and will make it available as open source to the community.

Image on first page. You are absolutely right—this is a Chimera example with $n = 3$ only. We will replace it and plan to use an example from ImageNet to take advantage of the higher image resolution.

Thank you for your feedback on our paper!

Experiments with larger models. We have extended our evaluation to include the ImageNet dataset, using the more complex architectures ResNet18 (Top-1 Accuracy: 69.7%) and EfficientNetV2S (Top-1 Accuracy: 84.2%). For these experiments, we used a reduced backend set (OpenBLAS, MKL, BLIS, and cuBLAS) and a smaller sample size of 128. In these new experiments, our attack achieves a 100% success rate for CPU-GPU backend combinations on both models. For CPU-CPU backends, we observe success rates of approximately 29% for ResNet18 and 22% for EfficientNetV2S. We will include these results in the paper.

Efficiency and generality of our defense. The computational complexity of the proposed defense depends only on the size of the input, not on the model itself. We leverage standard cryptographic libraries, which allow us to compute keyed noise for thousands of inputs per second. As a result, our defense adds less than 0.5% overhead to the overall inference process, making it practical for most application scenarios.

In our defense, we deliberately avoid making any assumptions about the design or functionality of the employed backends. The only parameter we rely on is the empirically measured size of the pockets containing Chimera examples. A practitioner with knowledge of the specific backends used for a model can measure this size and configure our defense accordingly. However, you are correct that the required magnitude of defense noise depends on the complexity of the model, as more complex models typically accumulate larger discrepancies between backends during inference. Therefore, the noise level must be empirically calibrated on a per-model basis. Nonetheless, this does not affect the efficiency of our defense

White-box access. It is correct that our attack hinges on white-box access to the model and knowledge of the employed linear algebra backends, as shown in our experiments. However, finding Chimera examples is tricky, even in the white-box setting. Black-box attacks are, therefore, likely to suffer from low performance in practice. Keeping models confidential might, therefore, seem like a possible defense. We would still argue, however, that this is not a reliable protection strategy, as it requires keeping information confidential that is typically not considered secret. Moreover, in standard machine learning frameworks and environments, the available backends might be known to the adversary by default. We will acknowledge this setting in our paper.

Thank you for your feedback on our paper!

Experiments with larger models. We have extended our evaluation to include the ImageNet dataset, using the more complex architectures ResNet18 (Top-1 Accuracy: 69.7%) and EfficientNetV2S (Top-1 Accuracy: 84.2%). For these experiments, we used a reduced backend set (OpenBLAS, MKL, BLIS, and cuBLAS) and a smaller sample size of 128. In these new experiments, our attack achieves a 100% success rate for CPU-GPU backend combinations on both models. For CPU-CPU backends, we observe success rates of approximately 29% for ResNet18 and 22% for EfficientNetV2S. We will include these results in the paper.

Other defense strategies. We initially considered different defense strategies but ultimately chose to randomize the inputs (rather than the weights) using keyed noise. This approach offers two advantages: First, the noise remains fixed for each input and, therefore, cannot be averaged out over multiple runs. Second, the runtime overhead of this defense does not depend on the model’s size and complexity. Note that we employ standard cryptographic libraries in our defense, which enables us to calculate thousands of inputs with keyed noise per second. An alternative strategy would be to reject predictions near the decision boundaries to prevent classifications. However, this comes with drawbacks: First, the margin around the decision boundary would still need to be chosen based on the pocket sizes of the backends. Second, introducing such a margin effectively creates a new attack surface for Chimera examples—this time between “rejected” and “accepted” predictions.

Analysis of cuBLAS. We have further investigated the differences in attack performance between CPU-based and GPU-based backends. We found that the considered CPU-based backends employ identical implementations for the convolution operator. Hence, the numerical differences between them stem from matrix multiplication only. In contrast, the GPU-based backend cuBLAS uses a fundamentally different implementation for convolutions. As a result, numerical deviations arise from both convolution and matrix operations. As a result, we observe varying results for the FMNIST and CIFAR models. Since the FMNIST model consists solely of dense layers, the observed differences remain consistent across all backends. In contrast, the CIFAR (and ImageNet) models are largely composed of convolutional layers, which exhibit substantial variation when using cuBLAS. This variation makes it easier to identify Chimera examples in these models. We will include these results in our paper.

Thank you for your feedback on our paper!

Chimera vs. adversarial examples. We are sorry that a key distinction between Chimera examples and adversarial examples did not come across clearly. Constructing a Chimera example always requires considering at least two backends simultaneously. If only a single backend is attacked, the resulting adversarial example is likely to transfer to other backends as well—and thus does not qualify as a Chimera example. This is why standard algorithms for generating adversarial examples are typically unable to produce Chimera examples. As shown in Table 2, for instance, the attack by Carlini & Wagner (labeled Adv. Examples) failed to find a single Chimera example for the CPU-only backend pairs. We will clarify this point in the revised version of the paper.

Robust Accuracy. We have considered different metrics for our evaluation but ultimately decided against using robust accuracy, as our focus is not on measuring changes in model robustness but on assessing the attack surface of backend pairs. Nonetheless, the presented attack success rate—shown in Figure 5 for different backend pairs—can be interpreted as a normalized version of (1 - robust accuracy). Since we introduce a new attack, we present this success rate, as it better highlights the efficacy of the attack rather than the resulting degradation in model performance.

Adversarial Training. Thanks for this suggestion. Indeed, we had not considered adversarial training as a potential defense. However, this concept cannot be directly applied in our setting, as it would require training the same model across multiple backends simultaneously. Nonetheless, we conducted two experiments to investigate this defense in slightly different settings. First, we added Chimera examples to the training data of the considered models, which corresponds to a simplistic form of adversarial training. In this setting, however, we observe no impact on the success of our attack. Second, we performed adversarial training using a standard method for generating adversarial examples (PGD). Interestingly, this reduces the success rate of our attack by 30–40%. We attribute this drop to our attack’s initialization step, which leverages an adversarial example to reach the decision boundary. By increasing the number of iterations in this initial step, we can improve the attack’s performance again. This highlights an interesting property of our attack: it initially behaves like a standard adversarial example to reach the decision boundary but then searches for Chimera examples in its vicinity. We will include this additional experiment in the revised version of the paper. Still, our defense (Section 5) is capable of completely eliminating the attack without affecting model performance, unlike adversarial training.

Quantization and reparameterization. The quantization function $q$ maps a vector $x$ in the feature space to the nearest element in a discrete set $S$ forming the input space. For instance, in the case of 8-bit images, it converts floating-point values in the feature space back to 8-bit pixel values. Similarly, reparameterization is a technique used to enforce box constraints on the input, ensuring that all pixel values remain within the valid range [0, 255]. This approach is derived from the Carlini-Wagner attack. We will clarify this in the paper. (Infeasible regions) Infeasible regions arise from the mismatch in granularity between the input space (e.g., 8-bit pixels) and the feature space (e.g., 32-bit floats). While Chimera examples may exist for certain combinations of 32-bit floating-point values, these may not necessarily be reachable using 8-bit inputs, hence lying in an infeasible region. We will clarify this difference and provide a more formal definition of this problem using the quantization function $q$.

Key differences. Our attack differs from the classic PGD attack in two key aspects. First, we calculate the gradient from a quantized input in every iteration. Quantization is necessary because we want to elicit a Chimera example from a feasible (discrete) input. Second, the perturbation is computed from multiple linear algebra backends with conflicting loss functions. Each backend aims to push the input toward a different class—like in a tug-of-war scenario.

Defense effectiveness. Our defense remains effective even when gradients are approximated or accumulated. It is effective because Chimera examples lie in extremely narrow regions of the feature space. As a result, adding noise with a magnitude larger than these regions during inference makes it very unlikely to discover them. As this noise is fixed for each input $x$ (keyed), it cannot be averaged out through repeated computations. Consequently, the computed gradients may move around the regions of Chimera examples but can only locate them by chance.

Drop-out layers. We use drop-out layers only during training and not inference.