CTBench: A Library and Benchmark for Certified Training

We $\newcommand{\Rj}{\textcolor{purple}{VJvs}}$are happy to hear that Reviewer $\Rj$ finds our work interesting and well motivated, our library and benchmark useful and comprehensive, and our experimental results insightful. Due to the word limit, we address major questions raised by Reviewer $\Rj$ below and are happy to discuss more in the follow-up. We include new results as Table S1 etc., in the anonymized link.

Q1: Could the authors comment on why the improvement of certified accuracy sometimes accompanies a decrease in natural accuracy?

The robustness-accuracy trade-off is well-known, where higher certified accuracy often comes at the cost of natural accuracy. Most methods, including SABR and MTL-IBP, have hyperparameters (e.g., $\lambda$, $\alpha$) that directly regulate this trade-off. Our goal, like in prior work, is to maximize certified accuracy, with natural accuracy improvements seen as a bonus. For completeness, we further provide robustness-accuracy curves, as shown in Figures S1–S3.

Q2: Could the authors provide an ablation study to evaluate the individual impact of CTBench strategies?

Please refer to our reply to Q4 of Reviewer $\textcolor{blue}{mj6P}$.

Q3: Can differences in implementation change the test-time certification?

No. Test-time certifications are conducted using third-party tools like MNBaB and OVAL. Our changes only affect the training process, while the final trained network can be exported and verified independently, ensuring direct comparability with the literature.

Q4: Could the authors discuss the connection between this work and probabilistic certification methods such as Randomised Smoothing?

Please refer to our reply to Q1 of Reviewer $\textcolor{green}{vyCo}$.

Q5: Could the authors discuss the possibility of extending this work to training and certification for other norms?

Our work focuses on deterministic certified training using bound propagation for the $L_\infty$ norm, as it remains the most reliable and widely adopted approach for robustness guarantees. While [1] explores various norms for certification, it also limits deterministic certified training to $L_\infty$, reflecting the current state of the field, with practical deterministic methods focused on $L_\infty$.

Certification under other norms, such as $L_2$, faces scalability challenges. For example, [2] evaluates $L_2$ certification on small models with only 192 hidden nodes, while our CNN7 network has over 10M parameters, making their method impractical. Similarly, [4] uses expensive SDP methods, limiting their approach to synthetic toy datasets (Spheres) and does not naturally extend to $L_\infty$. Furthermore, while [4] addresses $L_2$-norm robustness, their methods do not naturally extend to $L_\infty$.

We acknowledge that exploring deterministic certified training for other norms is a valuable future direction. However, due to scalability limitations and the lack of effective methods for other norms (even on MNIST), our focus remains on $L_\infty$. If Reviewer $\Rj$ knows of scalable methods for other norms, we would be happy to include them in our study. In addition, we will revise our statement to say that “we only focus on $L_\infty$ robustness because there exists no scalable deterministic certified training algorithm regarding other norms”.

Q6: Could the authors clarify the difference between the experiment with loss fragmentation in Section 5.1 and the findings of Shi et al. [3]?

We clarify that Shi et al. [3] analyze only IBP-based instability, which is an over-approximation of the real instability of neurons. In contrast, our analysis applies an estimate of the true number of unstable neurons. To illustrate this difference, we provide a comparison between the two variants in Table S5. We observe that the gap between our lower bound estimate and IBP is larger for SOTA methods, which also reflects in the certification gap between IBP and MN-BAB for these models (Table S4).

Q7: Do the authors have an intuition as to why certified training performs so poorly (and even worse than adversarial training, which is much less expensive) for some OOD corruptions?

On corrupted datasets (MNIST-C, CIFAR-10-C), adversarial and certified training improve robustness against localized perturbations like blur, noise, and pixelation but struggle with global shifts like brightness and contrast changes. This aligns with the intuition that these methods enhance robustness mainly in the immediate neighborhood of the original inputs, whereas global changes fall outside this region. Moreover, the stronger regularization induced by certified training when compared to adversarial training often exceeds what's needed for untargeted corruptions. Addressing this may require diverse augmentations, though it could reduce certified adversarial accuracy.

We $\newcommand{\Rm}{\textcolor{blue}{mj6P}}$thank Reviewer $\Rm$ for their insightful review. We are happy to hear that Reviewer $\Rm$ finds our work interesting and well motivated, our library and benchmark useful and comprehensive, and our experimental results insightful. In the following, we address all concrete questions raised by Reviewer $\Rm$. We include new results as Table S1 etc., in the anonymized link.

Q1: Can the findings of this study be generalised to other architectures than CNN7?

Our study focuses on the CNN7 architecture, which is consistently adopted by SOTA works to enable direct comparison. Table 14 in Appendix C2 compares CTBench training with the baseline training code from [1] using a smaller CNN5 on MNIST with $\epsilon = 0.3$. Results show that CTBench improves IBP, SABR, and MTL-IBP performance on CNN5, proving that the benefits extend beyond CNN7. Moreover, it is confirmed on CNN5 that IBP can match SOTA methods under large perturbations with fair comparisons and well-chosen hyperparameters.

Furthermore, Table S2 examines shared mistakes between CNN5 and CNN7, revealing common patterns across architectures. We will include additional results on generalizability in the revised manuscript.

Q2: The number of unstable neurons as the metric of smoothness may be challenging to generalize to non-ReLU networks. Could the authors comment on this and consider alternatives, such as measuring the change in loss value within a neighborhood of an input sample?

We agree that the number of unstable neurons is hard to generalize to non-ReLU networks, as a metric of smoothness. However, it is of critical importance to the certification of ReLU networks, beyond measuring the difficulty of adversarial attacks which may also be indicated by other smoothness metrics. Concretely, branch-and-bound (BaB), the dominating strategy for complete certification of ReLU networks, directly branches the unstable neurons; thus, the number of unstable neurons provides a direct metric for the difficulty of certification. Since ReLU networks dominate certified training, we adopt the number of unstable neurons as the main metric. We will clarify this and comment on generalization to non-ReLU networks in the revised manuscript.

Q3: CTBench achieves SOTA performance by incorporating different strategies. Could the authors discuss the individual impact of these strategies?

We acknowledge the importance of ablation studies and discuss in Appendix A the challenge of fully disentangling these effects due to their interconnected nature.

While full disentanglement is infeasible, we conduct a preliminary study to separate implementation advantages from hyperparameter tuning. Table 14 compares CNN5 performance using CTBench and the SOTA codebase, applying CNN7-tuned hyperparameters to both to reduce tuning bias, showing CTBench's universal implementation benefits. Additionally, Table S3 (L1 regularization on IBP-trained networks) and Figure S3 (effects of varying $\lambda$ for SABR and STAPS and $\alpha$ for MTL-IBP on the robustness-accuracy trade-off) illustrate hyperparameter impact.

Q4: Complexity is a crucial metric, but the running time is only reported in the appendix. Could the authors provide more analysis and discussion on the complexity of different training and certifying methods?

We agree that complexity is crucial. Therefore, in addition to the running times reported in Appendix (Table 8), we provide a more detailed complexity analysis for each training method in Table S6.

For certification, we use complete certification algorithms based on branch-and-bound techniques that have exponential complexity, depending on the number of unstable neurons that need to be analyzed. Due to this exponential growth, we use a timeout of 1000 s per sample. Also, we report a new ablation study on certification algorithms in Table S4, including IBP and CROWN-IBP (efficient but incomplete certification algorithms) certified accuracies.

Q5: The experiment on shared mistakes appears to provide dataset-specific results rather than general insights. Could the authors clarify why curriculum learning improves certified training?

We would like to clarify that we do not claim curriculum learning directly improves certified training. Our claim is that data points exhibit varying levels of difficulty for certified training, as evidenced by the systematic deviation from independent errors. The connection between data point difficulty and curriculum learning is discussed in [2], which is why we suggest its potential benefit. Additionally, Appendix C.3 provides a more extensive evaluation across different datasets and certification algorithms, demonstrating that the results are not specific to a single dataset.

References

[1] arxiv.org/abs/2305.13991

[2] arxiv.org/abs/1705.08280

We $\newcommand{\Rv}{\textcolor{green}{vyCo}}$thank Reviewer $\Rv$ for their insightful and careful review. We are happy to hear that Reviewer $\Rv$ finds our work interesting and well motivated, our experimental results convincing, and our ablation studies insightful. In the following, we address all concrete questions raised by Reviewer $\Rv$. We include new results, named with Table S1 etc., in the anonymized link.

Q1: How are the issues of unfair comparison highlighted in this work relate to the broader area of certified robustness, namely non-deterministic methods such as randomized smoothing (RS)?

We agree that unfair comparison may also be present in the RS literature. After a brief survey, we find that recently published randomized smoothing techniques vary significantly in network architecture, training schedules, hyperparameter choices, and the noise distributions used for certification, all of which may contribute to unfair comparisons. Therefore, introducing a standardized fair benchmark as well as unified implementation for RS is also important. However, while a meta-survey on the issue may be easily performed, establishing a benchmark at the same level of CTBench requires a large amount of effort, and is way beyond the scope of this work. Concretely, one needs to unify all algorithms in a single library (which has never been developed, to the best of our knowledge), validate that all implementations match (or exceed, as in CTBench) the original reports, and then may start to evaluate them in fair settings. We remark that even the last step, which requires the least human effort, takes months’ compute on four GPUs in CTBench. Therefore, it is impractical for us to conduct the same study for RS in this work. However, we strongly believe that this is a good future work, and we will discuss this direction and the meta-survey in the revised manuscript.

Meanwhile, we conduct a preliminary study on comparing $L_\infty$-norm robustness certified by RS to our results based on deterministic algorithms. Specifically, we compare the numbers by the state-of-the-art $L_\infty$-norm RS algorithm [1] on CIFAR-10 $\epsilon=2/255$ and $\epsilon=8/255$ with CTBench results in Table S1. We find that the current RS approaches yield lower certified accuracy compared to CTBench, in agreement with the literature that deterministic methods dominate the $L_\infty$ robustness.

Q2: The paper includes results for multiple datasets, but each of them is analysed separately. Could the authors provide a further analysis and comparison of performance trends across the datasets?

Sure, we provide a preliminary analysis below, and will include more in the revised manuscript.

Across the datasets considered in this work, several performance trends emerge, offering insights into how different certification and training methods generalize. For both MNIST and CIFAR-10, we observe that Interval Bound Propagation (IBP) demonstrates great performance at larger perturbation sizes, while other methods show limited improvement over IBP. This suggests that as perturbations increase in magnitude, stronger regularization is crucial for maintaining certifiability. In the context of corrupted datasets (MNIST-C and CIFAR-10-C), adversarial and certified training methods effectively enhance robustness against localized perturbations such as blur, noise, and pixelation. However, these methods remain less resilient to global transformations like brightness and contrast changes compared to standard training. This observation aligns with the intuition that adversarial and certified training primarily improve robustness in the immediate neighborhood of the original inputs, whereas global changes fall outside this region. Addressing this limitation could involve more diverse data augmentation strategies, though this may come at the cost of reduced certified adversarial accuracy.

When examining network-level properties such as neuron instability and network utilization, trends across datasets are less straightforward. In all cases, standard training results in the highest neuron instability, as expected due to the absence of regularization aimed at minimizing this effect. However, network utilization does not follow a consistent pattern. In some scenarios, certified training increases network utilization compared to adversarial training, indicating the learning of more complex patterns and relationships. Yet this trend is not universally observed, suggesting that the underlying dynamics of network utilization are context-specific and not easily generalizable.

Overall, these findings highlight that while some performance trends persist across datasets, others are context-dependent, underscoring the need for context-specific analysis when evaluating the current stage of certified robustness methods.

References

[1] arxiv.org/abs/2406.10427