Towards Effective Evaluations and Comparisons for LLM Unlearning Methods

Many thanks for your constructive comments and suggestions! Please see our responses below.

Q1. For the metrics used for machine unlearning, The author suggests using Extraction Strength as the metrics since it exhibits linear relationship before and after different attacks. However, the evaluation is still weak to me. First of all, the attacks used in the paper are not the strongest jailbreaking attacks (like GCG attacks) and the authors also do not consider using an ensemble of strong attacks to elicit model’s knowledge, making the evaluation of the paper weak. Also, showing linear relationship before and after attack might not be a good way to measure machine unlearning before and after attack, especially if the model is backdoored or poisoned (e.g., a poisoned and backdoor model can hide their knowledge in their model parameters except when a trigger is presented or obfuscate the knowledge so that the metrics cannot used to detect unlearning). In short, there are no rigorous guarantees that using extraction strength is a valid metric under an adversarial scenario.

A1. Many thanks for your comments! First, we would like to clarify why linear relationships (or PCC) is appropriate for assessing the metric robustness. As detailed in Appendix A, a proper metric that reflects the extent of knowledge parameterization should not be affected after attacking. In other words, the scores values before and after attacks should remain the same. However, it is also worth noting that, by the causal graph in Fig 4, this conclusion holds only if the attacking methods change only superficial model behaviors while preserving the inherent knowledge. For our considered 4 distinct types of red teaming scenarios, jail-breaking and probing strictly satisfy this criterion. Therefore, when considering only jail-breaking and probing, we can ensure PCC approaches 1 and the slope of the linear fit is close to 1, such that the scores before and after attacks lie in the identity line. As observed, in these criteria, ES is overall better than other metrics, especially for embed probing.

However, relearning and input noising are also of practical interests, but they involve fine-tuning and in-context changes that may modify the inherent knowledge within the models. Such modifications are likely to shift the distribution of scores, especially for relearning. Therefore, to broaden the feasible range of applying attacking methods, we further loosen the above condition to test only the PPC scores. We apologize again for any confusion and will include the related discussion in our revision.

Moreover, we follow your suggestions to incorporate the GCG attacks, the results are summarized as follows:

PPL: 0.83 / 0.50; ES: 0.94 / 0.91; EM: 0.92 / 0.91.

We also report the results that combine relearning and probing, which are two strongest attacking methods, the results are summarized as follows:

PPL: 0.54 / 0.03; ES: 0.73 / 0.61; EM: 0.55 / 0.29.

The results still show that ES is the most robust metric. We are always open to exploring additional attacking strategies. Many thanks for your wonderful suggestions!

Q2. The contribution is incremental. To me, the most novel part of the paper is that the authors use model mixing (aka model merging/souping) to control the trade-offs between retention and unlearning. All the other parts seem incremental and do not convey interesting empirical findings.

A2. We sincerely appreciate your comments, but we respectfully disagree with your assessment regarding the lack of contributions of our work. We have succinctly outlined the key contributions of our work in the abstract, further distilled into two primary aspects—reliable metrics and reliable comparisons—in the General Response. Specifically, we challenge the reliability of commonly accepted metrics in the literature, such as those based on PPL and ROUGE scores, arguing that they are less effective than previously believed. Instead, we advocate for the ES as a more accurate measure of knowledge parameterization, supported by formal justifications. Additionally, we propose a calibration framework designed to simplify the challenges associated with comparing unlearning efficacy when dealing with dual objectives. This framework has spurred a series of experimental findings detailed in Section 6, which introduce new insights not previously discussed.

To the best of our knowledge, the aforementioned factors have not been discussed in prior research. These insights contribute significantly to the community by enabling a more effective assessment and comparison of various unlearning methods. Kindly please correct us if there are any inaccuracies in our statements and we are really happy to further discuss with you about the contributions of our works. We will further highlight our contributions in our revision.

Q3. The writing of the paper needs to be improved. A lot of useful details help to digest the results are shown in the appendix instead of the main text (e.g., attack methods) and the main text does not give enough concise summary to understand them. For example, I still cannot understand how we use Token Noising as an attack method (even the appendix does not give a good description of it). Also, the organization of the paper could be improved (e.g., lines 264-265 ask the reader to first read Section 6 for the experiment).

A3. We sincerely appreciate your suggestion! Regrettably, due to space constraints and our primary focus on discussing both appropriate evaluation metrics and calibration strategies, we have to place some of the important details to the appendix. We commit to dedicating more time to refining the organization and further emphasizing our key contributions in the main text. Additionally, we will allocate more space for discussions on experimental setups and results in the revised document.

For the reason why token noising can be viewed as an attacking method, it is often the cases where overfitting occurs during unlearning. Thereby, token noise can lead to certain distribution shift, potentially causing the unlearning processes fail. Such vulnerabilities have been shown to lead to some successful attacks, such as that for EM. We will add the related discussions in our revision.

Dear Reviewer JEXC,

Thank you for your great efforts in reviewing our work and for your insightful questions. We hope that our answers and general responses have helped to clarify the points discussed. Please let us know if there is anything else you require further information on or if there are any additional concerns you might have.

Best regards,

Authors of # 2049

Dear Reviewer JEXC,

Many thanks for your great efforts in reviewing our works. Following your suggestions, we have conducted more experiments and further elaborated on our main contributions. We really hope that our answers can help to clarify.

Since the discussion due is approaching, please let us know if there is any further comments or concerns you might have. We are also open to further discussions with you.

Thank you very much!

Best regards,

Authors of # 2049

Dear Reviewer JEXC,

We sincerely thank you for your efforts in reviewing our work! We believe we have addressed your initial concerns. As the deadline for discussion approaches, please let us know if you require any further clarifications. We are eager to continue the discussions with you.

Looking forward to your response!

Best regards,

Authors of # 2049

Thank you for your response. I still have some concerns over the attack included to to elicit model’s knowledge. A concurrent work (I know the work is too new so that it is unlikely you can compare with) has demonstrated fine-tuning attacks, orthogonalization, and some adaptive attacks like Enhanced GCG tailored for unlearning could easily recover the knowledge stored in the model. Thus, I tend to keep my score.

Q3. Although UWC seems like a method to manage the trade-off between unlearn and retain performance, it is not clear how it fits into unlearning evaluation pipeline.

A3. The primary motivation behind UWC is driven by the dual objectives of unlearning: We should ensure effective removal of targeted knowledge and reliable retention of other, non-targeted knowledge, simultaneously. However, achieving both goals often involves the trade-off between removal and retention. For example, as illustrated in Fig 1, GA outperforms NPO in terms of removal, whereas NPO maintains better retention than GA. In this case, we cannot tell which method is overall better. These challenges motivate us to explore the concept of calibration, which aims to align overall performance across different models (i.e., ensuring retention). By ensuring consistent retention levels across different models / methods, we can then clearly compare their overall unlearning efficacy by focusing solely in comparing their behaviors regarding the removal of specific knowledge.

The benefits of calibration are substantial and twofold.

1. Fair comparisons: Calibration can largely mitigate the difficulties of comparing unlearning efficacy across different methods, serving as the foundation for our experiments detailed in Sec 6. Therein, we have discovered new insights that were not mentioned previously. For example, after proper hyper-parameter tuning, GA-based methods have shown to be more effective than NPO; adapting batch sizes and implementing likelihood capping can further improve the overall efficacy of unlearning.

2. Practical applications: Beyond evaluations and comparisons, calibration also has benefits from an application perspective. It can be employed to recover the performance of non-targeted data for over-unlearned models. This is crucial for practical LLM applications, where unlearning must be conducted without impairing the normal model functionality. Given the current limited progress in retention, calibration can be employed post-unlearning to ensure the unlearned LLMs still useful.

We will add the related discussions in our revision.

Q4. The statistic in figure 2 is confusing. How does model have lower score (which means better unlearning from the paper) after attacking than before?

A4. Many thanks for your suggestion! As mentioned in the caption of Fig 2, the x- and y-axes denote the metric scores before and after attacks, respectively. The blue and orange dots / lines represent scores for retention and removal, respectively. As observed in all the attacking cases, the scores after attacking will either arise or remain unchanged, except for EM noising (indicating that EM is not a proper metric). We believe that your confusion may stem from a misunderstanding of the colors of the lines as indicating scores before and after attacks. We will clarify these descriptions in our revised manuscript to prevent any further misunderstanding.

Many thanks for your constructive comments and suggestions! Please see our responses below.

Q1. There is a lack of justification for selecting the metric: Why does the PCC measure the metrics' robustness again attacks? In Figure 2, the plot is characterized by the test static before and after the attack for different methods, models, and forget set ratio. Why should we assume there is a linear correlation among them? In addition, the paper uses TOFU as unlearning dataset/task, but does not survey the metric used in the TOFU paper (truth ratio).

A1. We apologize for any confusion that may arise. Here, we would like to further explain why PPC is a proper criterion for assessing the metric robustness.

In Appendix A, we argue that an optimal metric should accurately reflect the extent of knowledge parameterization and not be affected by attacks. In other words, scores values before and after attacks should remain consistent. However, we want to further emphasize that, by the causal graph in Fig 4, this criterion holds only if attack methods affect only superficial model behaviors while preserving the inherent knowledge intact. For our considered 4 distinct types of red teaming scenarios, jail-breaking and probing critically satisfy this condition.

Hence, when only focusing on jail-breaking and embed probing, we need to ensure 1) PCC approaches 1 and 2) the slope of the linear fit should be close to 1, thereby the scores before and after attacks align closely with the identity line. As observed, ES is overall better than other metrics, especially for embed probing. However, relearning and input noising also hold practical significance, but they will alter the inherent knowledge within models via fine-tuning and in-context changes. These two attacking processes can naturally lead to the shifts in the distributions of scores (especially for relearning). Therefore, we relax the conditions and focus solely on evaluating the PPC scores to include all four attacking scenarios of our interest. We will refine the related discussion in our revision.

Now, we explain the reason why we did not explicitly include the truth ratio in our analysis. Overall, we want to extract a series of fundamental metrics from existing scores, rather than continually adopting to the numerous new metrics that are proposed daily, which can be tedious and redundant (e.g., many proposed scores are based on ROUGE and PPL). The truth ratio can be derived from PPL, thereby having been covered in our analysis. However, we also agree with your opinion that the truth ratio has its unique characteristics, and we further conduct a series of experiments to assess its robustness. The PPC values obtained are as following:

jail-breaking: 0.97 / 0.99; relearn: 0.56 / 0.09; probing: 0.83 / 0.02; noising: 0.99 / 0.99.

As observed, similar to PPL, the truth ratio shows robustness against jail-breaking and noising. However, it does not outperform some of the more robust metrics when facing relearning attacks and embedding probing. We will incorporate the related discussion in our revision. Sincere thanks for your suggestions!

Q2. Weak/unclear attack methods: it is unclear the what dataset the relearn attack in figure 2 is based on (from other section I would think it is TOFU, but it is not presented until late in the paper). The setup for relearning it not clear.

A2. Many thanks for your comment! Due to space limit, the detailed descriptions of the adopted attacking strategies have been allocated in the Appendix D. We adopt the original TOFU dataset for fine-tuning one epoch, using the negative log-likelihood as the objective. The Adam optimizer is adopted with the same learning rates as original fine-tuning. We will make the descriptions about the attacking setups clearer in our revision.

Dear Reviewer fSyL,

Thank you for your great efforts in reviewing our work and for your insightful questions. We hope that our answers and general responses have helped to clarify the points discussed. Please let us know if there is anything else you require further information on or if there are any additional concerns you might have.

Best regards,

Authors of # 2049

Dear Reviewer fSyL,

Many thanks for your great efforts in reviewing our works. Following your suggestions, we have conducted more experiments to strengthen our findings and clarified many technical details to enhance clarity. We really hope that our answers can help to clarify.

Since the discussion due is approaching, please let us know if there is any further comments or concerns you might have. We are also open to further discussions with you.

Thank you very much!

Best regards,

Authors of # 2049

Dear Reviewer fSyL,

We sincerely thank you for your efforts in reviewing our work! We believe we have addressed your initial concerns. As the deadline for discussion approaches, please let us know if you require any further clarifications. We are eager to continue the discussions with you.

Looking forward to your response!

Best regards,

Authors of # 2049

Dear Reviewer fSyL,

As the deadline for discussion approaches, we are eager to know if there are any further questions or clarifications you require. We look forward to your response.

Best regards,

Authors of #2049

Dear Reviewer fSyL,

Thank you for your follow-up! We are eager to know if our clarifications have addressed your concerns and look forward to further discussions.

Best regards,

Authors of #2049

Many thanks for your great support and constructive comments! Please see our responses below.

Q1. In the paragraph “Is MM Proper for Calibration” what seems missing is discussion comparing this to a simpler/baseline approach, such as just tuning the original hyperparameters of the unlearning method to calibrate.

A1. Many thanks for your wonderful suggestions! We had not considered this aspect previously, but we totally agree with your view that hyper-parameter tuning can indeed be considered as a form of calibration. To highlight the superiority of MM, we would like to emphasize that a proper calibration method should ensure the control is applied in a noticeable yet smooth manner. However, as observed in Appendix H regarding hyper-parameter tuning, the model behaviors are quite sensitive to the choices of hyper-parameters, and we do not achieve the desired level of recovery in overall performance for many cases of hyper-parameter search. In contrast, in Fig 3, the control exerted by MM over model behaviors is notably smooth.

Additionally, conducting calibration through hyper-parameter tuning is too method-specific, which is not general for constructing a reliable evaluation framework. The computational costs associated with tuning hyper-parameters are also prohibitively high. By contrast, MM can be applied post-unlearning across various methods without incurring the additional costs associated with re-unlearning. Therefore, we believe MM is more general, reliable, flexible, and efficient than hyper-parameter tuning in calibration. We will add the related discussions in our revision.

Q2. Related to the above, in the “Hyper-parameter configurations” paragraph in the experiments, am I correct in understanding you do the hyperparameter sweep by evaluating the performance of the hyperparameters after also applying model mixing to calibrate the performance (this seems consistent with the appendix tables)? If so, you might consider adding somewhere in the paper that this hyperparameter sweep necessarily improves over the calibrated performance of not doing model mixing; this is as not doing model mixing is captured by just picking the best hyperparameters amongst those with \alpha=1.

A2. Yes, your understanding is accurate. Moreover, we would like to highlight that calibration eases the comparison of overall efficacy across different methods, especially when some methods excel at data removal while others outperform in data retention (cf. Fig 1). In line with your suggestions and to show that these scenarios are common in practice, we report the forget05 results before calibration in the following as a case study. As observed, the removal efficacy of GD exceeds that of NPO, while its retention performance is inferior. Since both removal and retention are crucial to us, it is hard to conclude which method is superior overall. We will add more results in our revision, many thanks for your suggestion!

Beyond evaluations, calibration also has benefits from the application perspective, which can be employed to recover the common performance for unlearned models. This is crucial for practical LLM applications, where unlearning must occur without compromising the normal model functionality. Given the current limited progress in retention, model mixing (or calibration) also holds practical significance. We will add the related discussions in our revision.

Q3. On the discussion of why NPO and other methods do no work in the experiments section, do you have results for what $\tau$ values (if at all) other methods start performing well? I’m imagining a figure where the x-axis is calibrated $\tau$ level, and the y-axis is forget performance, and a line for each method such that when the line is the lowest explains for what $\tau$ each method is best. Alternatively, some more detailed/explicit discussion for what $\tau$ levels other methods might perform better could help give a more complete picture for when other methods work well. I agree the high $\tau$ range seems most practical, but I think the above can be done easily given results in the appendix and can add to the empirical study.

A3. Many thanks for your suggestions! Please see below for the calibrated results under 5% unlearning setups across different values of $\tau$.

As we can see, GA-based methods can still beat NPO and other methods. We thank for your wonderful suggestion and we will add the related discussion in our revision.

Q4. (minor) A more general weakness, not specific to just this paper, is whether these performative unlearning goals for LLMs should also be generalization questions; if the goal is to not predict accurately for a set of samples, and those samples come from larger distribution, should we not be evaluating a “test” performance on the larger distribution? This is generally called concept unlearning in the literature. The field may still be far away from evaluating/studying such questions, but I raise it to clarify the setup studied in this paper is not the only setting for unlearning in LLMs. The authors might consider having more discussion on different formulations of unlearning before picking the setting they study and survey.

A4. We totally agree with your opinion. Your mentioned perspective is also supported by previous works that discuss about the scope of unlearning. Therein, they mentioned that, depending on the application scenarios, the unlearning goal may involve either removing a specific set of data or more general concepts/knowledge. While the TOFU benchmark offers a set of rephrased data for testing generalization (adopted in our evaluations as ES-perturb), it does not fully enable us to ablate or explore how to control the scope of unlearning. It is a fascinating yet challenging direction, of which the studies should involve collaboration on new unlearning benchmarks, metrics, and methods. We will certainly include a detailed discussion about it in our revised manuscript. Many thanks for your wonderful comments!

[1] Sijia et al. Rethinking Machine Unlearning for Large Language Models.

Q5. (minor) model mixing seems very related to the literature on model merging; you might consider surveying the literature more broadly as a reference for the method, and when/why it works.

A5. We fully concur with your perspective that our model mixing is closely related to model merging, a concept previously explored in the robust fine-tuning, model editing, and federated learning literature. Prior empirical studies have demonstrated that averaging over a set of models tends to preserve the properties of each [2,3]. Additionally, theoretical research [4] has suggested that the property of weight disentanglement, which posits that averaging weights is akin to an ensemble of models, is crucial for the success of model mixing (or model merging). We will add the related discussion in our revision.

[2] Wortsman et al. Robust Fine-tuning of Zero-shot Models.

[3] Ilharco et al. Editing Models with Task Arithmetic.

[4] Ortiz-Jimenez et al. Task Arithmetic in the Tangent Space.

Q6. (minor) If computationally feasible, could error bars/standard deviation be reported for some of the main tables; I am less concerned given the many settings tested, but this could add to the rigor of the empirical findings.

A6. Thank you for your suggestion! We report the mean and standard deviation (mean + std) results under the 5% forget setup as follows. All experiments were conducted five times using different random seeds. Our observations indicate that GA-based methods continue to outperform NPO, consistent with our original findings in Table 1. We will include additional results and analysis in our revision.

I thank the authors for their response! I believe the paper will have an impact on the field, and given the clarification to my understanding, I am raising both my score and confidence. In particular, I’m stating confidence 5 as to normalize with the other reviewers given my understanding of their reviews. I elaborate below.

To elaborate on my stance, I have also read the other reviews, and disagree with their complaints about novelty and impact. A core issue in comparing unlearning methods is that unlearning methods do not equally weigh retain and forget performance (even if more compute is spent into calibration), and this method resolves this simply with seemingly great success. It seems the other reviewers do not believe the method is novel because of its simplicity, but such an argument seems ill-grounded given the method's effectiveness and lack of previous literature on the problem. Furthermore, as now pointed out (which I hope will be further elaborated on in the final draft), the obvious approach of just hyperparameter tuning for calibration does not work and is also more computationally expensive.

Another core issue in the field is that there are many metrics for unlearning, and this paper gives empirical evidence towards working with a single metric. Complaints were raised on rigor, but the claim in the paper is to have a metric that best preserves ranking, which is what the correlation metric does (though spearman may be better than pearson, but given the plots linear seems appropriate). Similar methodology was already used in published empirical studies in security conferences (with the difference being a focus on unlearning for privacy legislation) [1]. Another point was raised that poisoning and backdooring attacks would not fit such tests, but it is already known such attacks are computationally undetectable [2] and hence out-of-scope for the threat model of all current unlearning methodologies (which is implicitly assuming computational feasibility of unlearning). Thinking of the broader field, tests will need to continue to be made as more attacks are known, but this paper presents a case study and methodology showing great promise on how to go about these comparisons.

[1] “Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P)”

[2] “Goldwasser, Shafi, et al. "Planting undetectable backdoors in machine learning models." 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS). IEEE, 2022.”

Many thanks for your constructive comments and suggestions! Please see our responses below.

Q1. Lack of technical contribution: I think most people working in this area would agree we need more metrics and benchmark datasets. However, this paper though goes into that direction, does not really provide enough meaningful and technical contribution in my view. The paper basically tried 4 popular unlearning methods on the TOFU datasets while proposing a calibration framework (See W2). This can mostly be done in leaderboard or in a measurement paper rather than a technical paper. And findings on metric tradeoffs are mostly not surprising.

A1. We sincerely appreciate your comments, but we respectfully disagree with your assessment regarding the lack of contributions of our work. Your view that this manuscript is not a technical paper outlooks our systematical comparisons of various unlearning metrics and methods (cf. Sec 3-5) as well as our justification for the rationale behind them (cf. Appendix A, Sec 4, and Appendix F-G). Additionally, our experiments cover 6 (instead of 4) representative baselines as well as a set of practical tricks that can further improve the effectiveness of existing works.

We have succinctly outlined the key contributions of our work in the abstract, further distilled into two primary aspects—reliable metrics and reliable comparisons—in the General Response. Specifically, we challenge the reliability of commonly accepted metrics in the literature, such as those based on PPL and ROUGE scores, arguing that they are less effective than previously believed. Instead, we advocate for the ES as a more accurate measure of knowledge parameterization, supported by formal justifications. Additionally, we propose a calibration framework designed to simplify the challenges associated with comparing unlearning efficacy when dealing with dual objectives. This framework has spurred a series of experimental findings detailed in Section 6, which introduce new insights not previously discussed.

These factors contribute significantly to the community by enabling a more effective assessment and comparison of various unlearning methods. Kindly please correct us if there are any inaccuracies in our statements and we are really happy to further discuss with you about the contributions of our works. Additionally, we will follow your suggestions (especially from Q3) to refine our paper to better highlight our contributions.

Q2. Lack of justification of the calibration framework: The calibration metric lacks justification. Figure 3 seems to only tell us ES is monotonically increasing with mixing factor. However, the key question in proposing a metric is to ask: What does this metric measure? Does it measure the right thing? In this case, I cannot see clearly the impact of mixing model weights on the overall unlearning effectiveness. The only justification I can find is (1) being inspired by the literature on parameter disentanglement, but why is it related to unlearning? (2) The vague observation in Figure 3. Can authors explain why mixing model weights can help calibrate the unlearning metric other than just the empirical observation between ES and in Figure 3? After all, this can well be a spurious correlation. In other words, why would the community trust this calibration in evaluation?

A2. We apologize for any confusion that may arise. To begin with, we would like to highlight the motivation for the concept of calibration. For a practical unlearning method, we should pursue both effective removal and reliable retention. However, there is typically a trade-off between these two objectives. It complicates comparisons among unlearning methods, as one method might excel in retention while another is better at removal. Consequently, it becomes challenging to determine which method is superior overall. To address this, we propose a novel strategy: calibration. By aligning the retention performance across different methods, we can focus on comparison solely on their removal efficacy, overcoming the notorious trade-off issues and ensuring a more fair comparison.

The question now is what ensures an effective calibration approach. Ideally, it should manage model performance smoothly, displaying monotonic and stable changes in terms of both retention and removal. By achieving this, we can align different models in a stable manner (aligning retention performance) meanwhile ensure a fair comparison of removal efficacy (as the drop in removal efficacy is also fairly controllable).

The primary purpose of Fig 3 is to demonstrate that model mixing can achieve the goal of smooth control. The horizonal and vertical axes represent the controlling parameter $\alpha$ within model mixing and the value of ES score (cf. the caption of Fig 3), respectively. ES scores are used to measure the performance with respect to both retention (blue) and removal (orange). We observe that across different values of $\alpha$, the performance for both retention and removal changes in a noticeable yet smooth manner. Thereby, we conclude that model mixing is appropriate for calibration purposes. It appears that your misunderstanding may arise because we placed our descriptions of the definitions for the axes in the caption. We will clarify Fig 3 further in our revision.

We would like to further highlight that Fig 3 is our justification for the reliability of our evaluation framework. There are no spurious factors within our calibration method, as it relies solely on the controlling parameter $\alpha$. We are looking forwards to further discussions on the rationale and contributions of model mixing and calibration. We also welcome any additional concerns or question you may have!!

Q3. Lack of organizing clarify: The paper spends the first 6 pages motivating the calibration framework, and only 1 page introducing it and 1 page for experiments. I think people who in this area already know most of the points in the first 6 pages, it'd better to shorten it and get to the point more directly and spent more content on introducing the key idea of calibration and include more justification.

A3. Many thanks for your suggestions. As aforementioned in A1, our contributions are twofold. In the first 6 pages, we allocate 2 pages to the abstract and introduction, 1 pages to the necessary background, 2 pages to our first contribution on finding a reliable unlearning metric, and 1 pages for our second contribution on proposing the calibration framework. Therefore, the first 6 pages encompass a lot of new discussions. We sincerely appreciate your feedback and agree that there is still a large room to polish the organization of our manuscript. Following your suggestions, we will condense the introduction and preliminary sections, and allocate more space to elaborate on the calibration process and the experimental results. Thanks again for your suggestions!

Dear Reviewer taYS,

Thank you for your great efforts in reviewing our work and for your insightful questions. We really hope that our answer and the general responses can help to clarify. Please let us know if there is anything else you need further information on or any additional concerns you might have.

Best regards,

Authors of # 2049

Dear Reviewer taYS,

Thank you for your great efforts in reviewing our work and for your insightful questions. Following your suggestions, we have elaborated on our main contributions and clarified technical details for enhanced clarity. We really hope that our answers can help to clarify.

Since the discussion due is approaching, please let us know if there is anything else you need further information on or any additional concerns you might have.

Best regards,

Authors of # 2049

Dear Reviewer taYS,

We sincerely thank you for your efforts in reviewing our work! We believe we have addressed your initial concerns. As the deadline for discussion approaches, please let us know if you require any further clarifications. We are eager to continue the discussions with you.

Looking forward to your response!

Best regards,

Authors of # 2049

Dear Reviewer taYS,

As the deadline for discussion approaches, we are eager to know if there are any further questions or clarifications you require. We look forward to your response.

Best regards,

Authors of #2049

Dear Reviewer taYS,

Kindly please take a moment to review our feedback at your convenience. We will always await your responses and are looking forward to further discussions!

Best regards,

Authors of #2049

Dear Reviewer taYS,

We are still awaiting your further feedback and would appreciate any additional comments or questions that you might have.

Best regards,

Authors of #2049