Machine Unlearning for Contrastive Learning under Auditing

We thank the reviewer for the detailed and informative review. Here we address your concerns:

[W1(a): Motivation of positive alignment calibration]:

We clarify that FS may not be a reliable metric for black-box auditing, particularly when evaluating small unlearning sets (e.g., for individual data removal requests) due to high variance, as shown in Section 3.4. However, FS remains valuable as a white-box evaluation metric when analyzing the effectiveness of unlearning algorithms.

[W1(b): Optimization is in the proposed method's favor]:

We would like to clarify that in terms of choosing an optimal unlearning algorithm for the model owner, the auditing tool is not used as a white-box evaluation. Thus, our optimization is not in the proposed method's favor.

[W2: Theoretical Results support our design]:

• Our method's design is fundamentally motivated by the theoretical analysis of contrastive learning presented in [1], which decomposes the loss into two key components;

• As shown in Equation 3 of our paper, the contrastive learning objective consists of : (1) maximizing alignment between positive samples and; (2) ensuring uniformity of normalized features on the hypersphere

• Our method systematically addresses both components for unlearning:

  (1) Negative alignment calibration: Reduces uniformity within the unlearn set, directly impacting unlearn performance (UA) as visualized in AM

  (2) Positive alignment calibration: Weakens alignment between positive pairs, affecting UA as demonstrated in AGM

  (3) Performance preserving term: Maintains the overall distributional uniformity on the hypersphere, as validated in Section 5.5

[1] Wang and Isola, Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML 2020

[W3: AM is a pair-wise version of FS]:

• We would like to clarify that AM and AGM are not simply decomposed versions of the FS score, as they additionally measure similarity between negative pairs within the unlearn set. These metrics were motivated by the need to provide data owners with comprehensive visualization tools for understanding the unlearning effect.
• AM and AGM correlate well with other unlearning metrics to help the data owner understand the unlearning effect. Specifically, the diagonal elements demonstrate reduced alignment of positive pairs, while the off-diagonal elements show reduced uniformity of the unlearn set. Together, these contribute to the reduced Unlearn Accuracy (UA), a white-box evaluation provided by the model owner.

[Suggestions in writing:] Thank you for your constructive feedback. All suggested revisions have been incorporated into the modified draft (changes marked in blue).

Finally, thank you for your attention to our responses. We hope we have adequately addressed your concerns and welcome any further discussion.

We thank the reviewer for acknowledging our contribution and the insightful feedback. Here we address your concerns:

[W1(a): Motivation of FS]:

We emphasize that we do not claim FS is a reasonable nor complete measure of unlearning auditing. As we demonstrated in Section 3.4, we indeed find FS to be insufficient and unreliable for black-box auditing for a small subset (e.g., 8 images), matching the reviewer's intuition. However, we would like to point out FS remains valuable as a white-box evaluation metric when analyzing the effectiveness of unlearning algorithms.

[W1(b): Motivation of AM and AGM]:

(1) We would like to clarify that AM and AGM are not simply decomposed versions of the FS score, as they additionally measure similarity between negative pairs within the unlearn set. These metrics were motivated by the need to provide data owners with comprehensive visualization tools for understanding the unlearning effect.

(2) We agree that AM and AGM correlate well with other unlearning metrics. Specifically, the diagonal elements demonstrate reduced alignment of positive pairs, while the off-diagonal elements show reduced uniformity of the unlearn set. Together, these contribute to the reduced Unlearn Accuracy (UA), a white-box evaluation provided by the model owner.

(3) Regarding the difference between negative and positive alignment calibration: $p_u^{\times}$ represents negative pairs within the negative set, which influences the off-diagonal elements in AM and AGM matrices. We have added relevant definitions in the modified paper (marked in blue).

[W2: Experimental Results]:

(1) Standard Errors: We report standard deviations across 5 random trials for all experimental results (Tables 1-3) in Appendix B.6 and Table 11 in our modified paper (marked in blue). Our evaluation metrics align with established practices in machine learning unlearning literature (e.g., [1][2][3]). The comprehensiveness of our evaluation in contrastive learning ensures unbiased assessment of our method's performance.

(2) RA, UA, TA: Our choice of retraining as the primary baseline is justified by:

• Retraining serves as the gold standard for unlearning, representing the ideal outcome we aim to approximate.
• Higher RA does not necessarily indicate better unlearning performance, as our objective is to match retraining behavior rather than maximize accuracy on the retained dataset.
• The use of absolute differences in our evaluation is methodologically sound, as our goal is to measure deviation from retraining results rather than directional changes in EMIA.
• Consequently, we employ absolute gaps to quantify how closely our method approximates retraining outcomes, without preferential treatment to either positive or negative deviations.
• Same or similar evaluations have been commonly used as baseline measures of approximating unlearning in [1]-[5].

[1] Fan et al., SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. ICLR 2024

[2] Jia et al., Model sparsity can simplify machine unlearning. NeurIPS 2023

[3] Golatkar et al., Eternal sunshine of the spotless net: Selective forgetting in deep networks. CVPR 2020

[4] Shen et al., Label-agnostic forgetting: A supervision-free unlearning in deep models. ICLR 2024

[5] Chen et al., Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. CVPR 2023

Finally, thank you for your attention to our responses. We hope we have adequately addressed your concerns and welcome any further discussion.

Thank you for the detailed responses. After going through the other reviewers' reviews as well as your rebuttals and the additions to the paper I have decided to keep my score.

• Auditing: I echo the concern of reviewer Ua6w in that I still don't find the suggested auditing metrics convincing. A data owner might see that there is a change in the proposed matrices from before to after unlearning but how does this translate into any assurance that the data was unlearned?
• Reporting: I think standard errors should be reported in the tables in the main paper as they cast doubts into the conclusions one can draw from these results. The authors report a gap w.r.t. the retrain baseline that is redundant information instead of standard errors.
• Methodoloy: I understand that this is how results are reported in prior works but this might still be inappropriate in this particular application to contrastive learning. I am particularly concerned that this approach doesn't lend itself well to scenarios where the results are high variance. Take an hypothetical scenario where a method is the same as the retrain baseline in expectation. Taking the average of a (signed) gap over 5 trials, the expected value is 0 and the variance is 5 times smaller. Taking the average absolute gap over 5 trials the expected value is not zero and scales with the variance. The variance in the numbers reported seems too high for this kind of approach to be reasonable and the results are overall not very convincing to me. It seems that slight changes in methodology or even just changing the random seed might change the conclusions.

Thank you for your response.

I had already read the reply in question but without any ground truth, I still don't know how to assess this contribution. As an example, I am not sure how to interpret Figures 6 and 7. The proposed method (AC) does not seem to produce gap matrices that are qualitatively closer to the retrain one which would seem desirable when the goal is to approximate retraining. It rather seems the opposite is true, but without a single numerical measure even this is hard to assess.

Regarding the metrics used to evaluate AC, I was merely observing that measuring absolute values of a gap effectivelly penalizes variance "both ways" and thus might not be the best choice in this scenario where results seem rather random. Increasing the number of trials would indeed help with this but, as it stands, the results do not seem convincing to me in the face of the standard errors (and not just for EMIA and CMIA but also the other metrics).

We thank the reviewer for the detailed and informative feedback, here we address your concerns:

[W1: Time Efficiency]:

We emphasize two key points:

(1) Training on retain set is standard practice

• All major unlearning algorithms (except gradient ascent) incorporate retain set training
• This is a well-established approach in the field

(2) Our method is computationally efficient

• Requires only 10 epochs for unlearning
• Runtime: 1.87 minutes (ours) vs. 109.47 minutes (retraining) in Table 1
• Tables 1-3 demonstrate efficient unlearning while maintaining unlearning performance comparable to retraining

[W2: Evaluation Metrics are not convincing]:

(1) Theoretical Guarantees: Only complete retraining (with randomly reinitialized parameters) can provide a strict certificate of removal, which, as noted, is computationally prohibitive. Therefore, we explore more practical unlearning methods to approximate retraining effects.

(2) Verification Challenge: Even when approximate unlearning achieves results comparable to retraining (as shown in Table 1), these claims by model owners cannot be independently verified by data owners, as discussed in Section 3.4. This necessitates dedicated auditing mechanisms.

(3) Auditing Need: Given data owners' limited model access, they require assurance that their unlearning requests are genuinely implemented rather than circumvented through inference-time tricks (e.g., conditional zeroing of outputs for unlearned data). Our AM and AGM metrics provide transparent visual evidence of how unlearning affects model behavior.

(4) Interpretation of Results: While unlearning a small dataset may not dramatically alter model weights, the consistent presence of non-zero elements across AGM matrices demonstrates genuine model recalibration rather than superficial modifications.

In summary, approximate unlearning presents a tradeoff between complete certification and computational efficiency. Our work provides both algorithms and validation tools to support model owners and data owners in implementing and verifying the unlearning process.

[W3: Unlearning Setup]:

(1) Unlearning success $\neq$ lower test accuracy:

• The goal of machine unlearning is to approximate retraining after data removal; not necessarily to achieve specific model performance metrics
• For data owners requesting unlearning, the primary concern is removing their data's influence for privacy protection, not the model's eventual accuracy
• Therefore, unlearning success should be measured by comparing our method's outputs to full retraining results, rather than focusing on absolute model performance.

(2) Random selection as a valid evaluation strategy:

• We employ multiple random trials (n=5) for unlearning set selection to ensure robust evaluation and prevent selection bias;
• Random selection is a well-established evaluation protocol in the unlearning literature, as demonstrated in some example works [1][2][3].

(3) Few-shot learning is a downstream task while we consider pretraining:

• We address contrastive learning pretraining with large datasets
• Unlike few-shot learning scenarios, removing 10% of data during pretraining has minimal impact on downstream performance
• Reviewer's concern about the few-shot learning impact doesn't apply to our pretraining context

[1] Fan et al., SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. ICLR 2024

[2] Jia et al., Model sparsity can simplify machine unlearning. NeurIPS 2023

[3] Golatkar et al., Eternal sunshine of the spotless net: Selective forgetting in deep networks. CVPR 2020

Q1: Systematic benchmark: While studying strategic data selection for unlearning is an interesting research direction, to the best of our knowledge, there is no standardized benchmark that systematically evaluates unlearning methods based on specific data selection strategies.

Q2: Theoretical research: While prior work such as [4] provides theoretical guarantees for exact unlearning through retraining, these analyses do not extend to approximate unlearning methods, particularly in the context of contrastive learning which is the focus of our work.

[4] Bourtoule et al., Machine Unlearning. IEEE S&P 2021.

Finally, thank you for your attention to our responses. We hope we have adequately addressed your concerns and welcome any further discussion.

We thank Reviewer T8fZ for the insightful feedback. Here are our responses:

[W1: More intuitions about the proposed method]:

Our method provides two benefits:

(1) More effective unlearning for model owners

(2) More reliable auditing for data owners

We propose AM and AGM as stronger auditing metrics based on:

• Motivation: Our analysis in Section 3.4 reveals that comparing only positive alignment scores (FS) before and after unlearning can be unreliable for individual data owners.
• Technical Contribution: AM and AGM utilize comprehensive information obtained by data owners, visualizing element-wise similarity for both positive and negative pairs within the unlearning subset.
• Key Findings: The metrics AGM reveal:
  • Diagonal elements (positive pairs) demonstrate calibrated lower alignment;
  • Off-diagonal elements (negative pairs) show reduced uniformity within the unlearn set；
  • Together, these patterns confirm diminished training effects on the unlearn set.

In summary, AM and AGM provide evidence to data owners that their requests are not circumvented by simple inference-time conditions, but have been properly implemented in the updated model.

[W2: Insights on maintaning the negative pairs]:

We assume the reviewer was referring to the performance preserving term in our unlearning objective. This term is used to preserve uniformity, which is a key property for achieving good performance in contrastive learning [1]. In Section 5.5, we demonstrate that this term indeed preserves uniformity of the learned representation after unlearning.

[1] Wang and Isola, Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML 2020

[W3: UA, TA, RA]:

The relationship between these metrics follows a clear pattern from retraining:

• UA ≈ TA: Indicates successful unlearning, as unlearned data is treated like unseen test data
• RA > UA/TA: Expected behavior, as training accuracy is naturally higher

This pattern (UA ≈ TA < RA) serves as a practical validation of successful unlearning, i.e., an approximate way of retraining.

[W4: Datasets]:

According to your suggestion, we have added additional experiments on the SVHN dataset (Appendix B.5 and Table 10) to further validate our Alignment Calibration algorithm's superiority. Please refer to the modified draft for the complete experimental results (marked in blue).

[W5: hyperparameters]:

We have indeed included training hyperparameters in Appendix A.2.

Finally, thank you for your attention to our responses. We hope we have adequately addressed your concerns and welcome any further discussion.