We sincerely thank you for your valuable time and comments. We will alleviate your concerns in this rebuttal as follows.

The "Free" detection threshold seems ill-defined for the backdoor data detection method, which is essentially a binary classification.

We'd like to highlight that the baseline methods propose determining this threshold either by utilizing a statistic from a subset of clean data distribution or by employing knowledge of the poisoning ratio or through heuristic methods.

For example, methods like SPECTRE and Spectral Signature assume the knowledge of some upper bound of the poisoning ratio. On the other hand, CD suggests calculating the threshold from the mean and standard deviation of a subset of clean samples. We provide the various existing thresholding methods in Table A1 in the Appendix.

Thus, we think that a backdoor data detection method (which assign scores to training data samples) cannot be trivial unsupervised binary clustering. For example, the scores (which are scalar values) may have multimodal distributions.

Essentially, MSPC still needs to use loss value as the score, except the threshold is optimized through Eq 5.

We would like to point out that the MSPC loss values are used as the score only after the optimization of Eq. 5 is over. This is in general not possible if the mask $\mathbf m$ is not optimized.

Moreover, the threshold is not selected by users - it is fixed at 0 (by definition) as given by Eq. 2. However, that threshold will not work if we do not have a good mask, which leads to our proposed bilevel formulation of Eq. 5.

After relaxing the mask to continue values, Eq.4 is very similar to CD (Huang et al., 2023). It seems the difference is to replace the absolute difference with the KL divergence.

We strongly disagree with this statement.

Firstly, the mask in (Huang et al., 2023) represents the minimal mask that is responsible for the same prediction before and after applying the mask, while the mask in Eq.4 is responsible for maintaining the scaled prediction consistency signature.

Notably, Eq.4 includes the variable $\mathbf w$. Based on the fact that the scaled invariance property is only true for backdoor samples, Eq. 4 will work if we only consider the backdoor samples. This is the reason why $\mathbf w$ is an important part of the equation and the bilevel formulation of Eq.5 is a natural consequence of moving towards finding that optimal $\mathbf w$.

It would be great to include the CD in the experiments for comparison, as well as recent works Meta-SIFT (Zeng et al., 2022) and ASSET (Pan et al., 2023).

We include results with CD in the general response. However, we would like to emphasize that in light of the practical conditions (Section 1 and General response), these are not comparable.

We have highlighted the disadvantage of Meta-SIFT in this problem in Section 5.2 and Figure 5.

It should be noted that Meta-SIFT is developed to identify a small set of clean samples from the dataset (as explained in Section 1). To identify backdoor samples accurately, Meta-SIFT needs to identify all clean samples accurately. However, as shown in Fig. 5, it fails to do so (% of Clean Samples Selected = 100) because of the high NCR value.

We should also note that both Meta-SIFT and ASSET violate the practical conditions.

What happens to SPC if the input is not constrained to 0 to 1 or the input is normalized?

We provide the AUROCs obtained with SPC if the input is not constrained to 0 and 1.

SPC completely fails in such a scenario.

lower poisoning rates.

1. We have performed experiments with low poisoning rates ($\gamma = 0.05$). We have provided these additional results in Appendix G (Fig A3, Table A3).
2. Moreover, we point to the Adaptive Blend attack (Table 1, row 8). This attack was specifically designed to work with an extremely low poisoning rate ($\gamma = 0.003$). Our method achieves an AUROC of 0.9046, where all other baselines fail.

Dear Reviewer nd7k,

Thank you very much for the careful review and the valuable comments. It is our great pleasure to learn that your concerns have been addressed and the rating is increased. We will try our best to further improve our paper based on your suggestion.

Thank you very much,

Authors

We sincerely thank you for your valuable time and comments. We will alleviate your concerns in this rebuttal as follows.

The paper is poorly written, it takes the reviewer to read many times to get the core idea.

Thank you for this feedback. We respectfully disagree. We have made a substantial effort to improve the presentation of the paper, including clarifying practical assumptions, providing more illustrative examples, and illustrating the rationals behind our proposal. However, we are always eager to learn and would greatly appreciate any specific suggestions you might have on how we can enhance our writing.

Additionally, we would like to gently mention that Reviewer YD42 and Reviewer V1zd have commended our paper for being well-written and easy to follow.

key technical novelty is the introduction of a learnable mask into the MSPC loss

We would like to point out that the we introduce the MSPC loss. So the key technical novelty are:

1. Introduction of the MSPC loss (Eq 2)
2. Development of the bilevel formulation (Eq 5), which finds the optimal mask and automatically identifies backdoor datapoints. We also highlight our overall contributions at the end of Section 1.

the technique is very similar to an existing backdoor detection method proposed by (Huang et al. 2023).

We would strongly disagree with this statement.

We would like to point out that the only similarity with (Huang et al. 2023) is that we also penalize the l1 norm of the mask.

As mentioned before, our key technical contributions are the introduction of the novel MSPC loss and the novel bilevel formulation. This also helps us in satisfying P1 and P2, whereas (Huang et al. 2023) assumes the presence of clean data for identifying backdoor datapoints (thus violating P1).

The automatic filtering variable seems unnecessary, as at the end of Section 4, the authors stated that "backdoor samples will simply be the ones with MSPC loss greater than 0".

We would emphasize that the statement "backdoor samples will simply be the ones with MSPC loss greater than 0" is only true after the optimization of Eq 5 is complete.

We introduce the automatic filtering problem as a bilevel problem in Eq. 3, which contains the variable $\mathbf w$. Without this variable, it would be impossible to formulate our problem as a bilevel optimization.

The reviewer understands it is nice to satisfy both conditions but does not think this makes the proposed method fundamentally superior to other detection methods.

We elucidate the importance of the practical conditions in the General response. Given the existence of scenarios where it can be difficult / impossible to collect clean data, we believe that algorithms satisfying P1 and P2 will be favourable to practitioners.

In Table 1, the detection performance shown on CIFAR-10 is worse than that reported in the Cognitive Distillation (CD) paper (Huang et al. 2023) (their AUC is above 90%), and the results on test (and poisoned) samples should also be reported.

1. We would like to point out that our AUROC for CIFAR-10 is also more than 90% for all attacks other than DFST.
2. We aim to identify backdoor datapoints from the training set. If identified correctly, the user can either remove those backdoor data or unlearn them (as mentioned in 'Model Retraining Effect.' in Section 5.2).

This means the use of CD in the proposed way actually hurts the detection performance, which may be caused by the automatic search process with and the SPC loss.

Firstly, we would like to emphasize that we do not use CD. Our contributions include introduction of the MSPC loss (Eq 2) and development of the bilevel formulation (Eq 5).

We added experiments using CD in our general response and compared its performance with ours. We would like to highlight the following points:

1. CD performs well in cifar10. However, CD fails completely for the recently introduced attack Adaptive Blend. However, our method maintains an AUROC > 90% for this attack.
2. Our method performs better than CD in all attacks for both TinyImagenet and Imagenet200.
3. Most importantly, CD violates P1 from our practical conditions. Backed by our discussion of practical conditions (Section 1, Table 1 and General Response), we emphasize the fact that we cannot compare CD and our method fairly because CD uses clean data to find the threshold but ours do not.

Dear Reviewer wHYC,

Thank you very much for the careful review and the valuable comments. It is our great pleasure to learn that your concerns have been addressed and the rating is increased. We are happy to know that you consider our proposed automated detection is a good contribution. We will try our best to further improve our paper based on your suggestion.

Thank you very much,

Authors

We sincerely thank you for your valuable time and positive feedback. We will alleviate your concerns in this rebuttal as follows.

closely aligned with the existing Scaled Prediction Consistency (SPC) approach

We would like to argue that our work leverages this signature and provides a non-trivial algorithm to identify backdoor data samples without any additional assumptions as discussed in Section 1 , Table 1, General Response.

1. Our proposed algorithm shows massive amounts of improvements over SPC across a variety of attacks as shown in Table 1.
2. We argue that our work propose in a novel algorithm that can identify backdoor data obeying the practical conditions, while SPC is incapable of doing so.

Extending the experiments to include more diverse model architectures, such as attention-based ViT

We have included results with ViT-Ti/16 [1] in the general response. As shown in the results, our method achieves similar AUROC values (sometimes better) when compared to Resnet18. This demonstrates the robustness of our method across different architectures.

1. Touvron, Hugo, et al. "Training data-efficient image transformers & distillation through attention." International conference on machine learning. PMLR, 2021.

We sincerely thank you for your valuable time and comments. We will alleviate your concerns in this rebuttal as follows.

Notably, in the case of CIFAR-10, the proposed method significantly outperforms the STRIP method solely under the AdapBlend (γ = 0.3%) condition. Surprisingly, in the context of Tiny ImageNet, the STRIP method even surpasses the proposed method.

We would like to point out that this is in fact incorrect.

1. Our method surpasses STRIP in both Wanet and AdapBlend for CIFAR-10. Notably, STRIP completely fails in AdapBlend (AUROC = 0.1378).
2. In Tinyimagenet, our method is better than STRIP for both Badnet and Wanet.
3. For Imagenet200, our method outperforms STRIP on average (as given in the general response). STRIP's performance considerably suffers for Badnet attack on Imagenet when compared to our method.

Could you provide a runtime analysis of the algorithms employed to solve the bi-level optimization problem?

We provide runtime (in mins) for our algorithm for different datasets.

We would like to point out that our runtime is dependant on the number of epochs we perform for solving the inner optimization (Epoch_in) and the number of overall epochs (Epoch_out) for solving the bilevel optimization.

We present the time and AUROC values for Badnet attack for different datasets and different configurations of numbers of epochs. We find that our method maintains high AUROC values across various configurations of epoch numbers.

STRIP runtime

We have included new results for STRIP with the Imagenet200 dataset (given in the general response). For comparison, this takes about 70 minutes.

proposed method does not introduce significant advantages, either in terms of computational complexity or performance enhancement, when compared to the STRIP method.

We strongly disagree with this statement. We introduce two key innovations:

1. Introduction of the MSPC loss (Eq 2)
2. Development of the bilevel formulation (Eq 5), which finds the optimal mask and automatically identifies backdoor datapoints.

We also highlight our overall contributions at the end of Section 1.

We are not claiming advantages of computation complexity nor massive performance gains. As explained in Section 1 , Table 1 and in our general response, our method satisfies the practical constraints which other baseline methods do not. Even while doing so, our approach often outperforms or performs at par with baseline methods (which do not satisfy the practical constraints).

We thank the reviewer for their reply and valuable comments. We hope to alleviate the additional concerns in this rebuttal.

Looking at the means and stds reported for Badnet and Wanet in Table 2, your statement might not have statistical significance.

Could you kindly share the number of independent runs for further clarification?

Thank you very much for the suggestion. Currently, our results are obtained over 3 independent runs. Encouraged by the comments, we conducted 2 more runs and performed a Welch's one-tailed t-test between the AUROC values of Badnet and Wanet. Our results have a significance level of 0.08 and 0.06 (<0.1) for these attacks in Tinyimagenet, respectively.

We will confirm this significance result with 10 independent runs later and can update it upon the reviewer's request.

Hence, at least from an effectiveness standpoint, your method doesn't seem superior to STRIP. I acknowledge that your method is built on less stringent assumptions, but for a more accurate interpretation of your results, it's crucial to articulate your claims precisely and rigorously.

Thank you for this suggestion. We want to clarify that we did not intend to assert any advantages in terms of computational efficiency or promise absolute massive performance gains. As the reviewer pointed out, "I acknowledge that your method is built on less stringent assumptions," and as we explained in Section 1, our achieved performance, which is on par with or potentially outperforming baseline methods, is contingent upon practical conditions. It's important to note that the baseline methods may not necessarily meet these same conditions. In this context, it holds particular significance for us because our method, which relies on fewer assumptions, manages to achieve highly competitive performance.

About STRIP: Please allow us to make further clarifications on this matter. The AUROC (Area Under the Receiver Operating Characteristic) is determined by calculating the area under the curve of the True Positive Rate (TPR) and False Positive Rate (FPR) across various detection thresholds. Consequently, when employing a backdoor identification method, it is not necessary to find a precise detection threshold for AUROC computation

The STRIP work suggests to find this detection boundary using the mean and variance of the entropy distribution of clean samples. In contrast, our method can automatically detect the backdoor samples without the necessity of clean samples.

Therefore, the computation of AUROC values for STRIP does not require detection boundary computation. Consequently, in scenarios where clean samples are absent, this limitation does not manifest in the AUROC values, as they are computed by directly utilizing the entropy values of all samples.

Considering the fact that AUROC offers a limited representation of the overall problem, we posit that relying solely on this performance metric may not be a comprehensive measure of the success of our method. We appreciate your consideration of these points.

Could you provide the training loss curve for your optimization procedure? I am concerned that your choice of both inner and outer rounds is relatively small, which may lead to insufficient convergence of the training loss.

Thank you for raising this question. We consider 2 epochs of outer optimization and 2 epochs of inner optimization.

We present the masking loss convergence curve for CIFAR10 in this Figure and in this Figure for TinyImagenet. For CIFAR10, each epoch contains 50 iterations, resulting in a total of 200 iterations, while for TinyImagenet, there are a total of 400 iterations because of 100 iterations per epoch. We will present similar loss curves for Imagenet200 soon, if needed.