Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention

We thank the reviewer for their positive assessment of our paper and discuss their questions/concerns below:

Reference [a]

Thanks for this reference! We have cited it and included a comparison in Section 2:

Both Mirage and the attack in [a] exploit abstention to reduce model availability in a targeted region, but differ in intent and actors. Mirage models institutional misuse (by the model owner) to deny service, while [a] involves third-party attackers aiming to increase institutional costs via fallback mechanisms. The methods also differ: Mirage targets regions in input space aligned with institutional incentives, whereas [a] uses artificial input triggers. As a result, Mirage induces abstention without modifying inputs, while [a] requires explicit alterations.

Threat model assumptions.

A model owner is generally interested in making predictions as accurately as possible over the entire support of the data distribution. While our calibration-based method may miss cases where both confidence and accuracy are reduced (e.g., $\varepsilon=0$, yielding uniform predictions and label flips), such attacks are detected by accuracy-based auditing techniques from the fairness literature (e.g., equalized odds). To highlight this, we include extended results in Table 2 (appendix).

The proposed method only works for classification.

Please see our reply to reviewer 6g19 where we include a regression experiment showing that ideas used in Mirage can be extended to regression. Also, we note that converting UTKFace's age prediction task to classification is not an arbitrary choice but common practice [1].

[1] Lowy, Andrew, Devansh Gupta, and Meisam Razaviyayn. "Stochastic differentially private and fair learning." ICLR, 2023.

The method suffers from ZKP’s long computation times.

While the computational overhead of ZKPs does impose practical limitations for applying Confidential Guardian to image data, tabular data is quite common in medical and financial applications, both of which are well-motivated settings for preventing abstention abuse and privacy-preserving audits. See our response to reviewer 6g19 for an extended discussion.

The implementation of the method is not available.

We will release our full code repository upon acceptance of the paper.

How to ensure the same model is used when making decisions & during audits?

To ensure the same model is used during deployment and audits, one can rely on existing ZKP-of-inference methods with cryptographic commitments—their binding property guarantees model consistency across stages. While the infrastructure for this lies beyond our scope, recent work shows promising scalability by verifying only a small sample of inferences [2].

[2] Kang et al. “Scaling up Trustless DNN Inference with Zero-Knowledge Proofs.” arXiv preprint arXiv:2210.08674 (2022).

Could the protocol work without the ZKP?

While our approach can run without ZKPs—Confidential Guardian could compute calibration metrics without verifying the forward pass—the guarantees become much weaker. Without ZKPs, the approach relies on the assumption that model owners report confidence values honestly. As the reviewer noted, this opens the door to model switching or fabricating confidence scores to suit institutional incentives.

Complexity of Alg 1?

The VOLE-based ZKP protocol has a complexity linear to the running time of the algorithm it proves [3]. In practice, the proof of inference computations are the runtime bottleneck. We will include a discussion of theoretical complexity and the key parameters affecting concrete overhead in the final version of the paper.

[3] Weng et al. “Wolverine: Fast, scalable, and communication-efficient zero-knowledge proofs for boolean and arithmetic circuits.” S&P 2021.

L293 How to modify the protocol?

This change can be made via simple find-and-replace in our EMP implementation of the protocol. As the reviewer notes, this opens the door to manipulation, so it is only appropriate in less adversarial settings (see L299). For instance, in medical contexts, strong legal accountability (e.g., HIPAA) deters tampering, making it reasonable for the service provider to collect $D_\text{ref}$. Combining this variant with privacy-preserving data provenance techniques [4] is a promising future direction.

[4] Duddu et al. “Attesting Distributional Properties of Training Data for Machine Learning.” 2024.

Table 1: what does the “<333” and “<1.27GB” for CIFAR-100 mean?

The running time of that row is based on Mystique [5] who reported time for LeNet-5, ResNet-50, and ResNet-101, but not for ResNet-18. Thus, we use ResNet-50 as the upper bound for the running time.

[5] Weng et al. “Mystique: Efficient Conversions for Zero-Knowledge Proofs with Applications to Machine Learning.” USENIX 2021.

We hope that our rebuttal has addressed the reviewer’s questions/concerns and are happy to further engage with them during the discussion period.

We thank the reviewer for their positive assessment of our paper and discuss their questions and concerns below:

The experimental evaluation could include more diverse application domains beyond image/tabular classification.

We chose to use image (high-dimensional, unstructured data) and tabular (structured) data as contrasting modalities that capture a range of real-world challenges. These experiments illustrate the effectiveness and generality of our approach. Extending the evaluation to additional modalities (e.g., text) could offer further evidence of generality, but would require additional adjustments to the problem formulation and is therefore beyond this paper’s scope. We welcome future research examining these extensions, and we also provide a discussion on regression below.

The computational overhead of the ZKP protocol may limit practical applicability in some settings.

Although ZKPs add overhead for larger models, recent efficiency advances are notable. [Weng et al. 2021] achieves ZKP inference on ResNet-101 (42.5M parameters) in about 9 minutes, while [Sun et al. 2024] processes LLaMa-2-13B (13B parameters) in roughly 16.5 minutes—a 167× speedup. Moreover, startups like Polyhedra (see https://github.com/worldcoin/awesome-zkml for more examples) are rapidly advancing these techniques. As Confidential Guardian treats ZKP inference as a subroutine (line 7, Alg 1), its overhead will naturally decrease as the technology matures.

The discussion of how to set appropriate thresholds for detection in practice could be expanded.

We agree that appropriately determining the detection threshold $\alpha$ is crucial in practical deployments. Appendix D.3 already provides an extensive discussion on this question, including a step-by-step guideline based on the data distribution, available domain knowledge, and existing regulatory standards. In short, we find that choosing $\alpha$ is a balance between practical constraints, domain-specific considerations, and regulatory mandates.

How would Confidential Guardian perform against adaptive adversaries?

We agree that adaptive attacks are key to understanding the limits of our defense, which is why we added additional experiments in Table 2 in the appendix. As shown there, an informed attacker could try two strategies to improve Mirage’s evasiveness:

• High $\varepsilon$: Skews the output distribution more strongly toward the correct class, making high-confidence predictions less anomalous and harder to detect via calibration metrics. However, this increases overlap with legitimate data, making selective manipulation more difficult.
• $\varepsilon = 0$: Yields an unbiased uniform distribution, reducing confidence and accuracy jointly. While this degrades calibration-based detection, such an attack is easily identifiable with more traditional fairness evaluation metrics such as equalized odds.

How might the approach be extended to regression, generative models, or reinforcement learning?

We conducted a regression experiment following the reviewer’s suggestion: https://ibb.co/kVQKYHMh. We simulate a noisy regression task and train a shallow neural network to predict a Gaussian output distribution (left plot). To adapt Mirage, we designate $[-3,-2]$ as an artificial uncertainty region in which we inflate predictive variance while keeping the mean consistent. This is done by regularizing the output variance $\sigma^{2}(x)$ towards a desired variance level $\sigma^{2}_t$:

$\mathcal{L}_{\text{penalty}}(x) = (\log \sigma^2(x)-\log\sigma^{2}_t)^2$

The right plot shows the resulting model with the uncertainty region. We will document this in more detail in a revised draft. Having validated Mirage in both classification and regression, we believe extensions to generative modeling are promising, though beyond this paper’s scope.

What are the practical considerations [...] particularly regarding computational overhead and integration with existing model serving infrastructure?

Confidential Guardian is a method for detecting abstention abuse on a model during an offline phase conducted solely between the service provider and auditor. To extend its guarantees to the deployment phase, one could apply ZKPs of inference using the model committed during the audit. The specifics of such an infrastructure are an orthogonal problem which we place beyond the scope of our work, but approaches in [1] and [2] – which provide strong bounds on model properties by ZKP verifying only a small probabilistic sample of inferences – show promising scalability.

[1] Franzese et al. “OATH: Efficient and Flexible Zero-Knowledge Proofs of End-to-End ML Fairness.” arXiv preprint arXiv:2410.02777 (2024).

[2] Kang et al. “Scaling up Trustless DNN Inference with Zero-Knowledge Proofs.” arXiv preprint arXiv:2210.08674 (2022).

We hope that our rebuttal has addressed the reviewer’s questions/concerns and are happy to further engage with them during the discussion period.

We thank the reviewer for their strongly positive assessment of our paper and discuss their questions below:

What are some other ways of denial/delay/abstention?

Indeed, our work considers abstention using model confidence, i.e. thresholding the maximum softmax score (as we describe in the Abstain Option paragraph in Section 3, see Equation (2)). While this is the predominant default abstention option due to its ease of use and broad applicability, other ways of uncertainty quantification / abstention exist. We highlight some representative instances in the Abstention mechanisms in ML paragraph in Section 2. After having established the foundations of artificial uncertainty induction in this paper, future work should consider the impacts of Mirage and the effectiveness of Confidential Guardian as part of other contemporary uncertainty quantification approaches.

What are some other ways of attacking with uncertainty?

As the reviewer correctly points out, there are multiple ways that an uncertainty attack could be realized. While we envision many such alternate approaches, we think the most interesting alternate formulation relies on modifying the training data instead of the loss function. In our formulation, Mirage is designed to reduce confidence as part of the loss computation which allows for precise control of a targeted confidence level by specifying a desired $\varepsilon$. Alternatively, it would be possible to introduce uncertainty by flipping labels, injecting noise into the features, or deliberately undersampling certain parts of the dataset, and then training a predictive model with a standard cross-entropy loss on this modified dataset. This approach would work similarly to data poisoning but with the distinct goal of reducing confidence, not (just) predictive accuracy. While such an approach would likely be effective at decreasing model confidence (and other derived uncertainty metrics), we are confident that such an approach would be harder to tune to a particular targeted confidence level. Absent any loss modifications, the effectiveness of such an attack would depend on the degree of confidence the model already has on neighboring data points which might differ widely on a per-batch basis, leading to instability during training. Moreover, any non-traditional loss function adaptations would likely require a change in the amount of noise or data under-sampling which makes such an approach less flexible to use. Finally, it is unclear if such an approach would allow us to only decrease model confidence or whether it would lead to a decrease in both model confidence and model accuracy at the same time. Similarly to the point that the reviewer raised above, we think that this could be an exciting avenue for future work!

We hope that our rebuttal has addressed the reviewer’s questions and concerns and are happy to further engage with them during the discussion period.