Efficient Robust Conformal Prediction via Lipschitz-Bounded Networks

Common response:

First of all, we would like to thank our reviewers for their time spent reviewing our paper along with the insightful comments they provided. Their reviews highlight that our method is “highly efficient and scalable” while underlining its innovative nature and our comprehensive experiments. Moreover, reviewer X44i points out the novelty of the theoretical framework we developed to ensure worst-case coverage bounds of vanilla CP. We appreciate all these comments.

To address common questions shared among reviewer AUox and reviewer 3iVr, we add the following experiments which support our initial results:

• Shared model comparison: we run a comparison of robust CP methods on an identical model, see answer to reviewer AUox.
• Lipschitz bound tightness estimation: we empirically evaluate the Lipschitz constant of our network by running adversarial attacks, see answer to reviewer 3iVr.

Reviewer response:

We thank the reviewer for their positive feedback and strong endorsement of our work. We have incorporated an impact statement, as well as additional experiments (shared model comparison and empirical tightness estimation) to further highlight the strengths of our approach. We hope these enhancements further clarify and strengthen our contribution.

Common response:

First of all, we would like to thank our reviewers for their time spent reviewing our paper along with the insightful comments they provided. Their reviews highlight that our method is “highly efficient and scalable” while underlining its innovative nature and our comprehensive experiments. Moreover, reviewer X44i points out the novelty of the theoretical framework we developed to ensure worst-case coverage bounds of vanilla CP. We appreciate all these comments.

To address common questions shared among reviewer AUox and reviewer 3iVr, we add the following experiments which support our initial results:

• Shared model comparison: we run a comparison of robust CP methods on an identical model, see answer to reviewer AUox.
• Lipschitz bound tightness estimation: we empirically evaluate the Lipschitz constant of our network by running adversarial attacks, see answer to reviewer 3iVr.

Reviewer response:

Foremost, we would like to thank Reviewer AUox for their comprehensive comments. We appreciate the comments about our methods' novelty and empirical validation.

In the following paragraphs, we address their main concerns.

About comparing with different models (experimental weakness 1):

“The experiments comparison use different backbone model [...] potentially biasing results”

To alleviate these concerns: we train a VGG-like 1-LipNet with 10.7M parameters from Boissin et al. 2025 on CIFAR-10. Then, we evaluate robust CP methods on this network under the conditions of Figure 3 .

CROWN does not scale to such a deep network due to its inner complexity.

Therefore:

• Using a deeper network further improves our results (10.7M parameters compared to 4.5M parameters for the ResNeXt).
• LipNets allow for tighter conservative score estimations than CAS since its certificate is deterministic and does not require finite sample corrections.
• This LipNet makes the CAS method perform better than with a ResNet50 due to its robustness (cf. Figure 3).

These results will be added to the final version of the paper. We thank the reviewers for their comments which prompted this evaluation highlighting the performance and efficiency of our method.

About the ImageNet split sizes (experimental weakness 2):

To validate lip-rcp's performance without calibration set size differences we evaluate our method using only 500 calibration samples as done in the CAS article due to the method's high computational demands:

Set sizes: 118.5 (111.0 originally) Coverage : 97.6% (97.4% originally)

The performance gap between methods remains.

Theoretical questions (Claims 2 & 3):

While the manifold of images encountered in practice is typically nonconvex, Theorem 3.3 only requires that the input distribution and the score $x \mapsto s(x,y)$ be defined on a subset of a convex space such as $\mathcal{X}=[0,1]^{3 n_{\mathrm{pix}}}$, where $n_{\mathrm{pix}}$ is the number of pixels. Theorem 3.3 is then valid for any such distribution, even with finite support included in that space $\mathcal{X}$. We will add this fact within footnote 3 of Page 5.

Also, our coverage bounds (15) are not closed-form, but (as mentioned after (15)) they can be quickly and tightly computed via a binary search. See also Langford and Schapire (2005, after Def 3.2) for closed-form yet looser bounds.

About model independence (Claim 1):

As correctly pointed out by the reviewer, we use additional information on the model to improve over black-box (model-free) approaches. We propose to insert the following clarification in Section 4:

``In practice, our method uses Lipschitz-by-design networks. This limits the model independence of our robust CP method, but it was key to obtaining efficient and competitive robust CP metrics for the first time. Interestingly, our methodology, which applies more generally to any network and score that are Lipschitz continuous, can also benefit from future research on Lipschitz constant estimation.''

About computational efficiency (Claim 5):

With several recent developments mentioned in the related works of our article, Lipschitz-by-design networks have become easier to train, and well performing, and libraries exist to train LipNets with minimal effort (cf geotorch, deel-lip, etc…). Training overheads are described in Appendix E, for an exhaustive study of the overheads of LipNets, we refer the reviewer to (Boissin et al. 2025, Table 3).

For reference, our ResNeXt model of Figure 3 introduces a 9.6% runtime overhead on TinyImageNet compared to an identical unconstrained model.

Details: The article by Liu et al. (2024) mentioned by the Reviewer is already cited in Page 1 L46.

Common response:

First of all, we would like to thank our reviewers for their time spent reviewing our paper along with the insightful comments they provided. Their reviews highlight that our method is “highly efficient and scalable” while underlining its innovative nature and our comprehensive experiments. Moreover, reviewer X44i points out the novelty of the theoretical framework we developed to ensure worst-case coverage bounds of vanilla CP. We appreciate all these comments.

To address common questions shared among reviewer AUox and reviewer 3iVr, we add the following experiments which support our initial results:

• Shared model comparison: we run a comparison of robust CP methods on an identical model, see answer to reviewer AUox.
• Lipschitz bound tightness estimation: we empirically evaluate the Lipschitz constant of our network by running adversarial attacks, see answer to reviewer 3iVr.

Reviewer response:

We would like to thank Reviewer 3iVr for their insightful comments. Importantly we appreciate their comments about our methods' efficiency, scalability and performance.

Below we answer the reviewers main concerns.

About the tightness of Lipschitz bound estimations (Questions 1 & 3):

We run a direct tightness estimation of our Lipschitz upper bound with Lipschitz-by-design networks. We compute the maximum ratio between logit variations under attack and adversarial attack budgets on the CIFAR-10 test set.

This quantity offers a lower bound to the actual Lipschitz-constant of our network. Using PGD attacks of budget $\epsilon=0.05$ we get a Lipschitz constant lower bound of 0.917 when our by-design Lipschitz bound is 1. This demonstrates that our bound is relatively tight in practice.

Under these same attacks, the empirical coverage of our $\epsilon$-robust CP sets on the test split is 92.06% (94.7% under AutoAttack attacks) which approaches the desired robust coverage of 90% under $\alpha=0.1$.

About the choice of $L_n$ (Theoretical Claim 1):

To avoid any ambiguities, we address your question from two different angles.

• Setting any $L_n \neq 1$ as the network constraint: this would have limited impact since constraining the Lipschitz constant of a classifier is not a limitation when using an appropriate optimization objective (Béthune et al. 2022). Moreover, $L_n = 1$ avoids gradient vanishing or explosion (Béthune et al. 2024, Thm 1).
• Computing robust prediction sets using the l.h.s. of (19) and other values for $L_n$​: the approximation quality for the Lipschitz constant of the network is crucial, under-approximating it leads to uncertifiable results, while strongly over-estimating would lead to pathologically large prediction sets. Theory-wise, the impact of $L_n$​ can also be analyzed in a toy setting, with data points $(X_i,Y_i)$ drawn i.i.d. from a mixture of two Gaussians, and a model given by the Bayes rule.

About the LAC sigmoid score (Question 2):

Upon further inspection, the LAC sigmoid score yields a slight degradation of vanilla CP.

CIFAR-10 / ResNet50 / $\alpha=0.1$:

• LAC softmax coverage & set size: 90.04% / 1.088
• LAC sigmoid coverage & set size: 90.28% / 1.144

With similar tendencies for lower $\alpha$. We will add the following sentence to our paper:

“This ablation study results in marginally bigger vanilla CP set sizes for LAC sigmoid scores compared to softmax ones with scaled temperature. Investigating Lipschitz conformal scores represents a promising direction for future research to further enhance robust CP performance.”

Regarding the tradeoff between accuracy and robustness (Theoretical claim 2 & Question 5):

Interestingly, the trade-off between accuracy and robustness for robust CP is not straightforward.

Indeed, robust networks exhibit poorer accuracy which penalizes vanilla CP performance in small $\alpha$ regimes, consequently impacting robust CP performance. Similarly, accurate but brittle classifiers exhibit smaller vanilla CP set sizes yet the robust CP sets are large given the fine margins between conformal scores. To keep our method as simple, reliable and reproducible as possible we used standard hyperparameter values (which will be detailed in the Appendix) since our method exhibits SOTA behaviour without tuning.

We would like to thank the reviewer for pointing out this promising aspect.

Details:

We will fix the typos that Reviewer 3iVr kindly pointed out. Furthermore, we propose the following caption for Figure 5:

"Illustration of the proof. On the ball $\mathcal{B}\_{\epsilon + \delta}(x)$, the score $x \mapsto s(x,y)$ is minimized at some $\tilde{x}$. On the smaller ball $\mathcal{B}_{\epsilon}(x)$, the minimum can only be larger, but not larger than $s(x'',y)$, which is close to $s(\tilde{x},y)$ by continuity of $s(\cdot,y)$".

Common response:

First of all, we would like to thank our reviewers for their time spent reviewing our paper along with the insightful comments they provided. Their reviews highlight that our method is “highly efficient and scalable” while underlining its innovative nature and our comprehensive experiments. Moreover, reviewer X44i points out the novelty of the theoretical framework we developed to ensure worst-case coverage bounds of vanilla CP. We appreciate all these comments.

To address common questions shared among reviewer AUox and reviewer 3iVr, we add the following experiments which support our initial results:

• Shared model comparison: we run a comparison of robust CP methods on an identical model, see answer to reviewer AUox.
• Lipschitz bound tightness estimation: we empirically evaluate the Lipschitz constant of our network by running adversarial attacks, see answer to reviewer 3iVr.

Reviewer response:

We would like to thank Reviewer X44i for the time dedicated to reviewing our paper and the suggestions they provided. Also, we appreciate the reviewer's comments regarding the soundness and novelty of our work.

We develop answers to the reviewer's questions below:

About the connection with robust learning:

The field of Robust Conformal Prediction is quite specific, as it studies the robustness of guaranteed prediction sets under i.i.d conditions. It is not directly related to robust learning beyond reusing properties of a neural network, as the (split) Conformal procedure is post-hoc, and conducted on an already trained model.

However, a comparison to any certifiably (or not) robust prediction set is feasible, based on other theories or heuristics. Yet, to our knowledge, robust prediction sets have only been studied in the Conformal Prediction setting.

About our theoretical contributions:

While one aspect of our work concerns developing a highly efficient method to compute certifiably robust prediction sets (lip-rcp) in the classic Robust CP setting of RSCP (Gendler et al., 2022). Our theoretical work of Section 3 introduces a novel complementary theoretical approach:

In essence, we conduct a vanilla CP procedure (without robust score computations) and use an additional holdout data split to estimate maximum coverage variations for worst-case attacks. Those estimates are guaranteed for any budget simultaneously (uniformly). As opposed to Robust CP, this implies that under normal conditions, our method allows to retain the superior informativeness of vanilla CP sets while having an associated guarantee on how significantly their coverage under attack may evolve. As mentioned in the article, two previous works have proposed a weaker although incorrect guarantee, but with a similar intuition. Finally, the general formulation of our theoretical work allows us to extend our method to formal verification solvers (cf. Fig 4 - left).