Slightly Harmonizing Certified Robust Radius and Accuracy

1. The paper studies a variant of randomized smoothing that I have not seen elsewhere, which smooths the model weights in addition to the input. As a result, the theory does not apply to the large body of existing work on randomized smoothing. While proposing variations to existing methods may be beneficial, it’s important to include justification. (I could not find justification for smoothing the model weights in the paper.)

2. The experiments in Section 5.2 are missing an important baseline: randomized smoothing following Cohen et al. (2019). Currently results are only presented for the variant of randomized smoothing proposed in the paper that smooths the model weights and input. It's therefore not possible to assess whether the proposed variant improves on the baseline or not.

3. As someone without a strong background in PAC-Bayes theory, I found the paper challenging to follow. One sticking point for me is the fact that the smoothed classifier in eq (4) is not random, since the expectation is taken w.r.t $\mathbf{u}$. This lead me to wonder how PAC-Bayes is applied, given it is presented as a method for analyzing randomized predictors (see p. 3). I think further explanation would be helpful in the body, even if it is ultimately covered in the proofs. Another point where further clarification would be helpful is in the interpretation of the prior and posterior on the model weights. My understanding is that the prior is the distribution used to initialize the model weights, and the posterior is the distribution of the model weights after training, however the posterior and prior do not seem to be connected via Bayes rule.

4. Some steps in the derivation of the regularizer in Section 4 are unclear. It appears the original objective is to minimize $\prod\_{i} \lVert \mathbf{W}\_{i} \rVert\_{2}^{2}$ so as to maximize the certified radius. However the original objective is replaced by a lower bound $\lVert \mathbf{W}\_{n} \cdots \mathbf{W}\_{1} \rVert\_{2}^{2}$, which is then replaced by an upper bound $\lVert \mathbf{W} \mathbf{W}^\top \rVert\_\infty$ in eqn (9). I am confused by the mixing of lower and upper bounds – normally one would minimize an upper bound on the original objective. Another step that is confusing is eqn (10). I don’t immediately see why it is true, and there is no proof provided.

Minor:

1. Eqn (4): $\mathcal{G}$ is indexed by $\mathbf{u}$, however $\mathbf{u}$ is integrated out.
2. Eqn (4): overloading $\mathbf{u}$ is confusing.
3. Section 3.1: $\Omega_r$ and $\Omega_\mathrm{ge}$ are not defined anywhere.
4. Theorem 3.3: is it correct to say that the “generalization bound holds for any data perturbation within the radius”? For this to be true, the loss must be zero for the data point in question. And as I understand it, the generalization bound is a statement about the expected loss for a randomly drawn data point.
5. Section 4: $\cos$ does not typically have two vector arguments. For clarity, should define as cosine similarity.
6. Section 4: using $\mathbf{\eta}$ where $\mathbf{u}$ was used previously.

1. Since you don't prove Lemma 2.1, please provide a reference for it.
2. It is more common to use the terminology "isotropic Guassian noise" rather than "spherical Gaussian noise", so you may want to change that.
3. I suggest you replace the intersection symbol $\cap$ in Lemma 3.1 with the word "and," since you are not writing those two conditions as sets, but rather as mathematical predicates.
4. The authors make many connections to related works in the literature throughout the paper, yet, in the experiments, they do not compare their method to such related works. In my opinion, there could be more done in the experiments to further strengthen the contributions.

1. One primary concern is why we need to smooth both model weights and inputs. Although, by doing this, one can also obtain a certified region, it is not apparent that the new bound is tighter than the original one [Cohen et al., 2019]. Also, it is noteworthy that perturbing both weights and inputs would become more challenging to optimize. If it is not tight, it is meaningless. The authors should give more explanations about this and prove the tightness from the theoretical perspective.
2. With respect to the generalization bound, we should hold a strong condition $\mathbb{P}_{u}(...) \ge 1/2$. Is it achievable? In the training phase, we merely adopt the smoothed classifier to optimize. What is the relationship between $\sigma$ and this condition?
3. For the bound Thm.3.3, how to determine $\tau$ for Eq.(8)? It is recommended to validate the theoretical results by some empirical experiments.
4. Proof B.3 needs more details for easy understanding.
5. Does model weights and inputs adopt the same $\sigma$? What will happen if using a different one?
6. Finally, the empirical studies are also not unconvincing. 1) At first, I noticed that the authors leverage a different $\sigma$ value for normal and the proposed methods. Why? 2) Some sota randomized smoothing methods are overlooked [1,2,3]. The authors should conduct some ablation studies of $\sigma$ (such as $0.25, 0.5, 1.0$) and add more competitive methods. 3) Last but not least, the authors must consider some $\ell_2$-adversarial attacks to evaluate the effectiveness of the proposed method.

Ref:

• [1] MACER: attack-free and scalable robust training via maximizing certified radius.
• [2] Consistency regularization for certified robustness of smoothed classifiers.
• [3] Provably robust deep learning via adversarially trained smoothed classifiers.

• While the initial part of the paper mainly focuses on the Pac-Bayesian bounds for generalization, the empirical section doesn't provide any validation for the non-vacuousness of the given bounds. It is hard to follow the flow of the paper as presented now. It would be great if the authors could tie in the first theoretical part better with the later parts. It is well-known that the spectral norm bounds give the effective Lipschitz constant for a given ReLU neural network model. As RS effectively measures the local Lipschitz constant of a model to give guarantees, having a known smaller global Lipschitz constant should lead to better bounds. Thus, the theoretical results do not seem to be necessary to justify the motivation for lowering the spectral norm of the network.
• The empirical results are only given for small neural network models. It is unclear if the proposed method scales well to larger neural network models. Scalability and fast inference are some of the main advantages of randomized smoothing. It is not clear if those still hold for this proposed version of the RS.