Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing

1. My major concern is related to the novelty and significance of this work:

Novelty: The white-box and black-box tuning methods in this paper are all existing work. This paper just utilizes the existing PEFT on a large-scale classification model and tests its performance under randomized smoothing. This raises concerns regarding the paper’s technical contribution.

Significance: The primary challenge in randomized smoothing (RS) is not training noise-robust classifiers but reducing the extensive inference time required for certification, often involving a majority vote over 10,000 samples. While the paper aims to reduce model training time through PEFT, this does not address the more critical issue of RS's substantial inference delay. This raises concerns about the practical impact of the proposed work.

2. Some experimental settings are unfair:

In Figure 2, it's unfair to compare the PEFT methods with those training-free denoising methods.

In Table 1, clarification is needed regarding the base models used for RS and DS methods. Based on the reported performance, it appears these methods are evaluated using ResNet architectures, while PEFTsmoothing employs ViT-L and ViT-B models.

3. An interesting result is that the LORA-based tuning method outperforms the full fine-tuning-based method. Typically, the performance of parameter-efficient methods could be close to but not surpass the full fine-tuning method. A more detailed analysis of this would be valuable.

1. Lack of novelty. The training objective (4) is not novel, see baselines below. The overall premise of the paper does not seem novel enough for ICLR standards. Combining two well known methods, smoothing and PEFT, does not cross the novelty threshold. There is also a very large literature on certified robustness that the authors do not compare against, see https://sokcertifiedrobustness.github.io/

2. Limited baselines. Compare against and cite the following relevant works:

• S. Srinivas et al., Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness, NeurIPS 2023
• R. Shao et al., On the Adversarial Robustness of Vision Transformers, TMLR 2022, https://arxiv.org/pdf/2103.15670
• T. Tsiligkaridis et al, Diverse Gaussian Noise Consistency Regularization for Robustness and Uncertainty Calibration, IJCNN 2023, https://arxiv.org/pdf/2104.01231

The datasets are also quite limited and simple so I suggest a more thorough experimental study involving more domain-specific datasets, e.g. WILDS.

3. How far is empirical robustness from certified robustness in your setting? A comparison is needed. Approaches for gradient masking mitigation have been studied. What are the tradeoffs? What about computational complexity?

4. How does the geometry of the loss landscape depend upon the robustness properties of the large vision models you study? These are intimately related based the adversarial robustness literature.

This work currently has several weaknesses.

• There are several instances when the writing is imprecise. For instance
  • The claim made on lines 144-146 "Theoreticaly, Theorem 2.1 ... with Gaussian noised inputs" is a strong one, and requires either a citation or some kind of rigorous or empirical justification in the paper.
  • Lines 150-156 are also unclear. For instance, the authors should clarify "large-scale vision model ... acquire such potential ...". What does the phrase "acquire such potential" refer to?
  • lines 346-347: "this is mainly due ... especially for high resolution images" is another strong claim that either requires a citation or empirical justification
• It is unclear what the authors mean by "certified." Is it that the PEFT can be used to guarantee the conditions of Theorem 2.1? If so, can the authors provide a formal proof for this? In order to make a claim that a method is 'Certified', a rigorous guarantee must be provided.
• Equation (6) is written poorly and appears to be key to the entire experimental slate (and thus, this paper). Specifically:
  • The expression should be written in terms of indicator functions
  • The use of the '&' is unclear - do you mean the product?
  • The authors state that "certifiedCheck returns 1 if Theorem 2.1 is satisfied". However, the conditions in Theorem 2.1 are all probabilistic in nature. How are the probabilities computed? Specifically, how is $\mathrm{Pr}[F(x+\varepsilon) = c_A]$ computed?

The presentation in the paper needs to improve. The paper assumes a lot of background on PEFT on the reader. Given that most of the results in the paper are based on LoRA, it might be useful to provide a detailed explanation of the setup and training of the LoRA model. Similarly, I would urge the authors to provide a more detailed overview of the four different PEFT approaches considered in the paper. Figure 1 does not adequately explain the PEFT setting.

The Imagenet results in the appendix are quite a lot worse compared to the Diffusion denoiser. While the authors suggest that this can be attributed mostly to the fact that the diffusion-based denoiser is more powerful for higher-resolution images, is there a smoother trade-off that can be established between the two models? A small discussion on this could be quite helpful.