Thank you for your comment.

R1 Assumption&Novelty

The so-called implicit assumption that implicitly assumes ... backdoor data was not a weakness of our paper, and it does not influence the novelty when taking [1] into account. See below for details.

R1.1 Clarification

As the reviewer may have interpreted the background differently, we would like to re-clarify the defense scenario. Our focus is on a practical scenario where defenders attempt to train a benign model using partially poisoned backdoor datasets without known to be clean subsets (See lines 41-76, Introduction; lines 94-121, Preliminary).

To formally define, the defender has a training set $D = D_{c} \cup D_p$ , but the defender doesn't know which part of the samples is clean, and has neither known clean samples $D_{c*}$($D_{c*} \subset D_c$) nor external clean samples $D_{extra}$($D_{extra}\cap D=\varnothing$).

In our previous submission, we consistently use the terms clean subsets / extra clean subsets / clean seed rather than clean data throughout the main text and explicitly formulate the training set as $D = D_{c} \cup D_p$ instead of $D = D_p$ (Line 107, Preliminary). While other reviewers have not raised concerns regarding this aspect, we appreciate the opportunity to refine our explanation and will further clarify this definition in our revision.

R1.2 Free clean-data assumption

As we re-clarified above, we didn't make such an assumption as you interpreted, thus it does not constitute a weakness.

To understand why our BSD works on mixed datasets without known-clean samples, that's because we leverage general attack-related priors, including: 1) From the perspective of loss statistics, neural networks tend to overfit backdoor samples, resulting in lower loss values(ALS); 2) From the perspective of semantic information, identifying poisoned samples from backdoor attacks can be reframed as an open-set recognition problem(OSS). (See lines 152-160, page 3)

For your claim that without any information ... filtering-based defenses, there may be a misunderstanding regarding our background:

• It seems that you are referring to a scenario where there is absolutely no clean data in the training set, it's indeed an impossible task. But this scenario is not what our paper focuses on, “no clean data at all” is quite different from the “no additional clean subsets” scenario we address. Your comment below, In other words ... no clean data is available at all seems to suggest that we should solve a task that just described as impossible.

• If you mean that the defender has no assumptions regarding clean data information but is dealing with a mixed dataset, whether defense is possible depends on how you define “clean data information” and whether general attack-related priors count. Your [1] is a good case. It proposes a method that does not require known clean subsets but distinguishes between samples within a mixed dataset. It managed this using attack-related priors, treating poisoned samples as the minorities within each category.

R1.3 Novelty

Our novelty is already claimed (lines 75-98, Introduction), and Ref[1] doesn't affect our novelty, in detail:

• While [1] also assumes no extra clean sets, the core defense mechanism differs. It detects poisoned samples as intra-class outliers, considering only the samples within each current class. In contrast, our OSS module innovatively reframes the problem based on open-set recognition, OSS jointly considers the target class and all other classes for effective distinction. After identifying $y_t$ and warming up the main model, the OSS module distinguishes samples based on different feature distances between the target class $D_{t}$ (i.e., UKCs + UUCs) and the remaining classes $D_{nt}$ (i.e., KKCs). (Details in Section 4.1.1)
• [1] relies on the assumption that poisoned samples are intra-class minorities. As the intra-class proportion of poison samples($\approx\rho$) increases, class-wise mean representation goes closer to poison samples, reducing detection effectiveness. When poisoned samples became the intra-class majority (for large $\rho$ values or in imbalanced datasets like GTSRB), clean samples became outliers instead. In contrast, our method is robust against this issue and is verified by experiments under a larger $\rho$ range (Figure 3, page 8).
• Additional differences in other aspects: adaptive vs. one-time partitioning and a semi-supervised framework vs. detect-and-retrain framework, etc.

---

R2 Additional Experiments

Our experiments already cover 7 attacks in the main text and more attack variants in the appendix, along with 6 defenses and additional recent defenses like VaB, D-ST, and D-BR. This is sufficiently comprehensive since 1) the threat model specifies poisoning-based attacks; 2) related works like ABL and DBD evaluate fewer cases. More results in https://postimg.cc/DSn6S9nG.

Thank you for your constructive feedback and for recommending our paper for acceptance. We sincerely appreciate your thorough review and valuable insights. In response to your remaining concern, we present more baselines.

Additional results

As you suggested, which aligns with other reviewers, we added the PIPD as an additional baseline, together with NAB [1] and NPD [2]. As presented in Extended Table 1, while these methods demonstrated exceptional defense performance against specific attacks, our implementation suggests that they fail to achieve consistent defense across all seven attacks under a single set of parameter settings. We believe that with attack-specific parameter tuning, they could potentially achieve the performance claimed in their paper. However, we must emphasize that such attack-specific tuning is impractical for defenders in real-world scenarios.

Extended Table 1. Addition baselines on CIFAR-10. Since the table is too large, we present the full version in this anonymous link: https://postimg.cc/ftmSjbvR. PIPD1 uses the Adam optimizer (learning rate = 1e-3), PIPD2 resets the optimizer (clearing momentum and resetting the learning rate) before the Selective Training stage, and PIPD3 applies both a larger penalty ($\lambda = 5$) and the optimizer reset.

Implementation Description

The rationale for excluding PIPD in the previous submission:

• Methodologically, while PIPD's approach appears intuitively feasible, it relies on good starts from Pre-isolation using Local Gradient Ascent (LGA), proposed in ABL. Facing varying backdoor attacks, we found that it may require attack-specific tuning of hyper-parameters, which is not allowed for defenders. This concern, also supported by our early experimental observations, was one of the reasons we ultimately replaced our old version $y_t$ estimation (basically based on LGA).
• Before incorporating every method as a baseline, we conduct preliminary experimental validation to assess its reproductivity, such as evaluating its defense performance on the typical CIFAR10-BadNet combination. However, we observed that PIPD has not provided comprehensive open-source code. Specifically, after reviewing the main text, appendix, and GitHub repository of PIPD, we found PIPD: (1) provides empty README files and has no demos; (2) does not include any attack implementation (not even BadNet); (3) does not provide the implementation of Pre-isolation and Selective training stage; and (4) omits several critical training parameters, including optimizer selection, learning rate, the choice of the $\gamma$ in the Pre-isolation stage, the choice of $\lambda$ in selective training, and whether the optimizer/learning rate needs any resetting during the training.

Settings:

• To ensure experimental fairness, we merge the poisoned training data from BackdoorBench to generate datasets aligning with PIPD’s PyTorch dataset definition.
• For the non-open-sourced components of PIPD, we implement them based on their pseudocode and equations. For key parameters not explicitly specified in PIPD, we set $\gamma = 0.5$ in LGA for Pre-isolation, as recommended in ABL, and applied a default penalty of $\lambda = 1$ in Selective Training.
• Besides PIPD, we add two more baselines, namely NAB and NPD, with NAB being an in-training backdoor defense and NPD being a recent post-training defense.

---

$1$ : Liu M, Sangiovanni-Vincentelli A, Yue X. Beating backdoor attack at its own game, ICCV 2023.

$2$ : Zhu M, Wei S, Zha H, et al. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features, NIPS 2023.

Thank you for your comments.

Extended Figure/Tables in anonymous link: https://postimg.cc/68rDsgN3.

R1 Understanding BSD's Robustness Against Clean-Label Attacks

First, in response to Given that clean-label...single benchmark dataset?, we provided the experiment results of BSD on GTSRB (Extended Table 1).

The primary reason BSD resists clean-label backdoor attacks: 1) MixMatch could properly process unlabeled poisoned clean-label data; 2) BSD effectively split poisoned samples. Specifically, MixMatch's mixup operation visually weakens the trigger in the unlabeled data, preventing trigger-target label association. So, as long as the poisoned samples are correctly placed into the poison pool (unlabeled data), BSD effectively mitigates their impact.

We start by investigating semi-supervised involved defenses like ASD and DBD. These works report counterintuitive good performance against clean-label attacks. DBD included a brief explanation of its effectiveness against clean-label attacks (in their Appendix O, page25). Reproducing ASD showed that failures occurred when poison samples remained in the clean pool. Reviewing logs from two failed ASD experiments, we found many poison samples misclassified into the clean pool(Extended Table 2).

This led us to hypothesize that MixMatch could properly process unlabeled poisoned clean-label data, but DBD and ASD failed for not robust split methods instead.

Further investigation highlighted MixMatch’s mixup operation as critical. MixMatch removes original labels and mixes multiple inputs on the image level, effectively weakening the visual impact of the trigger:

$\tilde{x} = \lambda' x_l + (1 - \lambda') x_u, \quad \tilde{y} = \lambda' y_l + (1 - \lambda') y_u, \quad\lambda\sim Beta(\alpha,\alpha),\quad \lambda'=\text{max}(\lambda,1-\lambda).$

With $\alpha=0.75$ by default, expectation of $\lambda'$ ≈0.78, reducing trigger prominence in $x_u$. We visualize the mixed samples in Extended Figure 1 to better present this operation. Plus, we follow ASD set a 5x smaller $\lambda_u$, further reducing the influence of unlabeled data and mitigating clean-label attacks.

To experimentally verify this hypothesis, we enforced a secure clean-poison split where no poison samples were included in the clean pool. Under this condition, MixMatch effectively nullified the impact of clean-label attacks, as shown in Extended Table 3.

That said, it's still a necessity that we do not split poison samples into clean pools. Therefore, since the final model does not collapse (Table 2, page6), it supports the reliability of ALS’s split results. Moreover, while OSS seems ineffective against clean-label poisoned samples, it still serves two important purposes: 1) It provides a secured warm-up phase for the main model, preventing an immediate collapse (see our definition in Appendix C.3); 2) Although OSS may not directly identify poison samples, it selects samples from $D_t$ that are farthest in feature space from those in $D_{nt}$, i.e., representitive samples of $D_t$. This result intersects with the clean pool identified by ALS, improving the quality of init clean samples. Therefore, our statement is that ALS “compensates for the limitations of OSS”(line 375, page7), instead of ALS alone is sufficient.

---

R2 Important Baseline

Respecting the reviewers’ suggestions, we made efforts to implement PIPD(Extended Table 4).

Note that we did not intentionally avoid citing higher-performing works. The exclusion of PIPD (AAAI24) was due to:

• We conduct reproductive evaluations before adding any baselines, and PIPD's reproductivity is poor. Specifically, across its paper, appendix, and GitHub repository, we found many essential code/parameters/setting omissions.
• Methodologically, while PIPD’s approach is intuitively good, it relies on good starts from pre-isolation using LGA. However, our analysis suggests LGA alone may require attack-specific adjustments to hyperparameters (e.g., optimizer choice/learning rates). That's also why we replaced our early version of LGA-based $y_t$ estimation.

---

R3 Estimation of y_t

We would like to emphasize that the estimation method for $y_t$ is not our major contribution, and we have verified its robustness(Table 4, page7; Figure 9, page19; Table 8, page19; Table 11, page20) and discussed multiple insurance measures(line 1011-1032, page19). To answer how this method performs when the number of poisoned samples is relatively small, see Figure 3 (a), page8.

---

R4 Others

Model structures: Since we have no model-specific assumptions, our method is inherently model-agnostic. Additionally, our experiments on MobileNet further support this claim(Section 5.3). We also add a brief experiment in Extended Table 5.

Presentation: We appreciate your insights about strengthening the presentation of our work. We will update our modifications accordingly in our revision (once we get a chance).

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of our contributions.

We address your remaining concerns in the following sections.

Potential Failure of Target Label Estimation

We would like to kindly emphasize that while target label estimation is an important part of our method's success in defending against backdoor attacks, we believe that based on existing literature and our experimental results, target label estimation is a relatively less challenging task compared to the defense itself. We have verified its robustness through extensive evaluations (Table 4, Figure 9, Table 8, Table 11). Moreover, target label estimation is not the primary contribution of our work. Our main contribution is the innovative reconstruction of the backdoor defense task into an Openset Recognition-like task.

For potential failure cases, we have summarized four mitigation solutions:

1. In common scenarios where (1) the dataset is large and well-known with publicly available information on the number of samples per class, or (2) the dataset is well-balanced, target labels can be reasonably approximated using label statistics alone. This simple yet effective strategy ensures reliable target label estimation in practice.
2. We propose more than one target label estimation approach. Specifically, we introduce an alternative method based on local gradient ascent in Appendix D.3 (line 1012), which has proven effective against most of the discussed attacks. Furthermore, this alternative method may offer better target label estimation for unknown attack methods.
3. In scenarios where computational overhead is not a concern, we can apply the OSS algorithm to all possible classes and then aggregate the clean samples identified across all classes as the final OSS result. This approach has demonstrated effectiveness in our experiments (Figure 11).
4. For easy attacks, even if target label estimation fails, our pool update strategies can effectively correct the model through subsequent iterations (Figure 10).

The selection of OSS and ALS

We sincerely appreciate your insightful comments. Our decision to adopt OSS and ALS was primarily driven by considerations of computational overhead, as well as the challenges posed by the non-clean subset and model-agnostic defense settings.

Regarding gradient-based metrics, we agree that they can be effective. In fact, during the early stages of our research, we explored using per-sample gradient information and techniques like Grad-CAM to identify poisoned sample triggers. We further considered combining inpainting and denoising techniques in reverse diffusion to remove both local triggers (e.g., BadNets) and global triggers(e.g., Blended attacks). However, we ultimately abandoned these approaches as they significantly increased training time, particularly when performing adaptive pool updates.

Regarding alternative feature distributions, most of them are designed for backdoor detection, often assuming a clean subset. However, this violated the clean-data-free scenario. Additionally, potential feature extraction from intermediate layers will undermine the model-agnostic feature of the algorithm. Defenders would require knowledge of the model structure to select intermediate layers, undermining the practicality of defenses.

In contrast, our chosen OSS and ALS methods effectively integrate the defense task into the training process without making model-specific assumptions. Empirically, we observed that the OSS module, inspired by open-set recognition, complements the loss-based ALS, leading to robust defense performance.

Extreme hyperparameter situations

We sincerely apologize if the statements(lines 434-439, page 8) caused concerns regarding the extreme hyperparameter scenarios. Our intention was to demonstrate the robustness of our method to the settings of the two key hyperparameters, $\alpha$ and$\beta$ . We believe the results presented in Table 7 provide a solid reference for choosing hyperparameters, offering practical guidance for defenders and future researchers. Specifically, both values can be set as floating-point numbers within the range of (0,1), with $0.3<\alpha<0.8$ and $\beta<0.5$ being relatively safe choices.

To better clarify our points, we have revised the relevant section (lines 434-439, page 8) as follows:

We here present the influence of the main parameter, i.e., the parameters $\alpha$ and $\beta$ controlling the pool size. As revealed in Table 7, our BSD has robust performance against all the attacks with a relatively loose range of $\alpha$ and $\beta$ , and we recommend using the default setting in normal cases, and a reasonable range for adjustments is $0.3<\alpha<0.8$,$\beta<0.5$.

Minor issues

Our adjustment allows the figure to be displayed at a larger scale on the page, see this anonymous link: https://postimg.cc/bsztqjQS.