Rethinking Lipschitzness Data-free Backdoor Defense

The key weaknesses of the paper can be summarized as follows:

1. Limited Originality: The paper's claim to 'rethink' Lipschitzness Data-free Backdoor Defense appears to be a minor adjustment to the existing CLP framework. Specifically, the changes involve substituting the spectral norm with the L2 norm and adjusting parameters close to the pruning threshold. While there is some innovation, the overall enhancement seems minimal.

2. Insufficient Experiments: Despite the claim of thorough experimentation, the paper only evaluates and reports results for 4 out of 12 implemented attacks and 4 of 16 defense methods, in the provided code. This raises questions about the completeness and robustness of the evaluation. Additionally, the paper does not compare its approach against state-of-the-art (SOTA) methods such as RNP, which were discussed but not included in the comparative analysis.

3. Clarity and Writing Issues: The manuscript suffers from clarity issues that make it difficult to comprehend certain sections. Key problems include but are not limited to:

   • Sections 3.1.2 and 3.2 start with equations without an introduction, which causes confusion to the readers.
   • Equation 4 is confusing, why $f^{(l)}(x)$ has two different outputs?
   • The text from Line 273 to Line 278 is hard to follow, and I cannot understand it when I write this comment.
   • The using of $|$ and $\\|$, as well as the shape of $W_j$ are ambiguous in Line 279-292. Note that $\\|\cdot\\|_2$ has different definitions for matrix and vector.
   • In Algorithm 1 line 18, should it be "remove $T_{idx}$ parameters in W"?
   • The blue portion in Figure 2 (Line 338) is confusing as all of them are blue.

   I strongly suggest the authors have some proof-readers to check the notations and statements.

4. Inconsistencies Between Paper and Code: There is a discrepancy between the methodology described in the paper and the implementation provided in the code. The paper mentions a single parameter $u$ for pruning, whereas the code uses two parameters $u_1, u_2$. This inconsistency needs to be resolved.

5. Lack of Experimental Details: Crucial experimental parameters, such as pruning thresholds $u_1, u_2$, the extreme parameter number $k$, and the bias rate $b$, are omitted. According to the issues in the CLP Official repo on GitHub, CLP is sensitive to the choice of parameters, and the best choice depends on the model and dataset. Given that similar methods have shown sensitivity to parameter choices, the absence of these details raises concerns about the stability and reproducibility of the results.

6. Discrepancy in Claims: The paper suggests that CLP is not suitable for fully connected layers, whereas their proposed method can handle linear layers. However, the proposed method only prunes CNN layers similarly to CLP. Therefore, the stated limitation of CLP might not be a significant issue.

7. Further Analysis of Originality: Upon reviewing CLP, it was noted that CLP was initially designed for linear layers and then adapted for CNNs. If fully connected layer are treated as vectors (each slice as a vector), the spectral norm used in CLP reduces to the Euclidean norm used in the new method (LPP). Thus, CLP essentially addresses the same issue with a similar approach to LPP.

It should be noted that I am unable to fully comprehend Issue 2; therefore, its significance has not been validated by me.

1. There are some problems with the principle of categorizing defenses.

• Passive defense should also include the detection of backdoored models instead of just detecting poisoned samples
• There are still many other important defense paradigms, such as poison suppression [1]. Please refer to [2, 3] for more details.

2. Missing important references.

• Line 47: no references for active defenses
• Line 56: I believe there are still many other data-free defenses, such as [4]. We can easily find more on Google Scholar
• Line 125-143: Missing advanced backdoor attacks (e.g., [4-6]). There aren't even any works after 2023. The presentation of existing work is also not systematic.
• Line 144-153: Missing advanced backdoor defenses, such as the SOTA Trigger Inversion [7] and SOTA Input-level Backdoor Detection [8].

3. The contributions of this paper. This is my biggest concern. The authors should explicitly compare their method to CLP and highlight their main contributions.

4. The authors claim that they revisit the Lipschitz functions. However, the authors failed to highlight their main results and findings.

5. A systematic, structured description of the methodology is missing.

6. No discussion about the resistance to potential adaptive attacks. The proposed method relies on a latent assumption that the backdoor-related features can be decoupled from normal ones. However, adversaries can easily bypass this by reducing poisoning rates, using [4], or designing regularization terms.

7. The results of many important baseline attacks and defenses that I mentioned above are missing.

8. No discussion about potential limitations and future directions.

References

1. Backdoor Defense via Decoupling the Training Process
2. Backdoor Learning: A Survey
3. Defenses in Adversarial Machine Learning: A Survey
4. Revisiting the Assumption of Latent Separability for Backdoor Defenses
5. Narcissus: A practical clean-label backdoor attack with limited information
6. Backdoor Attack with Sparse and Invisible Trigger
7. Towards Reliable and Efficient Backdoor Trigger Inversion via Decoupling Benign Features
8. SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency

1. In this paper, although the Lipschitzness Precise Pruning (LPP) method is proposed based on Channel Lipschitznes based Pruning (CLP) with some extensions. However, the core mechanism of this work is still neuronal pruning using Lipschitz constants, which is similar to the core idea of CLP, so it fails to demonstrate significant innovation.
2. Although the article performs a comparison with the CLP, the overall experimental design lacks diversity, especially in the selection of defense models and attack methods used for comparison. In order to highlight the advantages of the LPP method more effectively, more experimental setups inherited from CLP experiment should be introduced for an exhaustive side-by-side comparison.
3. The article did not explore the relationship between the Lipschitz constant and model accuracy (ACC) when performing ablation experiments

Addressing the data-free issue and introducing Lipschitzness-based Pruning (CLP) were originally done by the CLP paper. While the paper aims to build upon this method, it does not adequately highlight its contributions or novel improvements over the original CLP approach. Explicitly emphasizing how the proposed method optimizes or enhances CLP would better underscore the paper’s contributions. For instance, the authors should elaborate on what specific aspects of the Lipschitzness-based method’s computational costs or processing efficiency they improved, as well as provide more detail on the optimization process and its outcomes.

The enhancement presented in this work appears to be incremental rather than a substantial advancement over the 2022 CLP method. The optimization does not represent a transformative improvement that would justify the novelty of the contribution. To strengthen the paper, the authors should consider reframing their approach to more clearly articulate how it moves beyond CLP in a fundamental way or consider expanding the method’s theoretical grounding to elevate the significance of its contributions.

The experimental evaluation would benefit from more comprehensive and targeted experimentation. For instance, utilizing updated and more representative datasets, backdoor attack scenarios, would make the results more convincing and aligned with current research challenges. Additionally, rather than demonstrating a single performance advantage in specific scenarios, the authors should design a set of experiments that systematically illustrates the superiority of the proposed method over CLP. This could include: A theoretical analysis of the source of advantages, which would form a basis for designing comparative experiments. Comparative evaluations that emphasize and clearly demonstrate the specific performance gaps between this method and CLP on critical metrics.

Relying on only one baseline (the 2022 CLP method) does not sufficiently showcase the method’s relative performance or unique advantages. Expanding the scope of baseline comparisons to include more recent methods would make the results more compelling and help position this work more effectively in the broader context of current research on data-free and pruning methodologies.