Bootstrapping Language Models with DPO Implicit Rewards

Q1. Is foregoing the partition function Z principled in this setting?

It depends on how we use the DPO implicit reward. In our case, the absolute values of the implicit rewards are not important, because we only need the ranking of two responses. This is done by taking the difference between the implicit rewards of two responses, where the partition function simply cancels because it depends on $x$ only. Therefore, we argue foregoing the partition function $Z$ in our case is proper.

Q2 & Q3. IRPO (Pang et al, 2024) should be mentioned. Authors could draw a parallel between the replay buffer approach and the learning from demonstrations literature (e.g. DQfD, Hester et al, 2017)

We have addressed both in the revised paper (Line 292).

Q4. What is the granularity of the search for \\alpha\^\\star? What is the cost of conducting such experiments?

We adopt a simple grid search with step size being 0.001. It runs for ~33 seconds on a CPU-only laptop.

Q5. Self-rewarding DPO (Yuan et al, 2024) [3] seems very much connected: is it a baseline here? It should be.

Yes, we have included self-rewarding DPO in our comparison. Specifically, we implement their core idea, which is to use LLM-as-a-judge to score individual responses so as to construct a preference dataset without the need for external reward models, and conduct apple-to-apple comparison with our method. Self-rewarding DPO corresponds to the "LLM-as-a-Judge" baseline method in $\\textrm{\\color{blue}Table. 1}$.

W5. Writing & Clarity.

Thank you for your careful reading and feedback! We have thoroughly revised our paper to address typos and improve overall clarity.

Section 4.1 as a whole is really confusing in the current state.

We reorganized the content by moving Section 4.1 in the main text (from the original submission) to Appendix A, for better logical flow and readability.

citation missing for RLHF (Christiano et al) l. 28

We have added it.

when mentioning the length exploitation issue, authors could quickly explain why the models exploit length (which is because longer responses are correlated with higher probability of being preferred by human or machine-based preferences)

We have revised the paper (Line 220).

no citation for Gemini Pro! also the version used should be stated (1.0 vs 1.5 vs 1.5 002)

It should be Gemini Pro 1.0, whose result is available on the AlpacaEval 2 leaderboard. We have revised the paper to make it clear.

“Offline DPO w/ new ref: similar to offline DPO but we assign the current policy as the new reference model, while we use a fixed reference model in Offline DPO. This corresponds to γ = 1” -> misleading as my understanding is that the reference is fixed (it is just the end policy of the first round of DPO)

Yes, the reference policy should be fixed during a single round of DPO training. Here, after each round of iteration, we update the reference policy with the latest trained model, and proceed with another round of training. We have revised the paper to improve the clarity.

W6. Experiments/Ablation that can be added.

What about using all the K samples with a Plackett-Luce model? Does this bring any benefit?

Incorporating more samples in preference tuning with a Plackett-Luce model could potentially enhance performance, as these samples provide more fine-grained information. Prior work by Song et al. has explored this direction and demonstrated promising results. Due to the time limit during rebuttal, we leave a thorough investigation of combining K samples with DICE as future work.

Comparing replaying human preference data with KL regularization to the previous policy.

We acknowledge that KL regularization targets the same purpose as experience replay - avoiding forgetting the knowledge inbuilt in the initial policy. This can be done by incorporating one additional KL term in the RLHF objective:

Following the derivation of DPO, a similar contrastive loss can be derived. Yet, this approach faces practical issues: one extra policy model $\\pi\_\\theta\^{(0)}$ needs to be loaded on GPUs, substantially increasing the memory overhead during training. Compared with this technique, experience replay is a more lightweight and stable alternative.

Baseline missing: offline AI feedback annotations instead of implicit RM.

In our experiment, we included a baseline that utilized AI feedback via LLM-as-a-judge prompting, referred to as the "LLM-as-a-Judge" baseline method in $\\textrm{\\color{blue}Table. 1}$.

[song et al.] Preference ranking optimization for human alignment. AAAI-24.

Thank you for your valuable review and suggestions. Below we respond to the comments in weaknesses (W) and questions (Q).

Note: We've marked "3Zep" next to relevant changes in the revised paper to make it easier for you to navigate.

W1. Clarify the connection to prior work.

Thank you for raising this concern. We appreciate the opportunity to clarify the connection to Park et al., 2024, and acknowledge their significant contributions. Following your suggestion, we have revised our paper to ensure proper attribution, with all updates clearly highlighted in blue.

Both our work and Park et al. share a common idea of mitigating length bias by penalizing verbose responses through regularization—a concept previously explored in the classical RLHF pipeline [Singhal et al.]. Building on this, Park et al. integrate length regularization into the training objectives of direct alignment algorithms such as DPO, while our work applies it at the data construction stage in the iterative DPO framework. This distinction offers us an advantage in efficiency. Determining the best regularization coefficient in training objectives often requires multiple training runs, whereas our method can adaptively determine an optimal value for $\\alpha$ in a minute, as acknowledged in the Strengths section of reviewer 3RHj.

[Singhal et al.] A Long Way to Go: Investigating Length Correlations in RLHF

W2. Include regularized DPO as a baseline.

Following your suggestion, we conducted a comparison between the regularized DPO and our proposed approach, with results presented in $\\textrm{\\color{blue}Table. 7}$. The findings indicate that our approach achieves better performance while effectively mitigating the length exploitation. In contrast, regularized DPO exhibits a quality-length trade-off, where larger regularization coefficients successfully control response length but lead to a decline in response quality.

W3. Further discussion on the results that improvement does not keep over 2 iterations.

To investigate model performance over three iterations, we compared our approach with the LLM-as-a-Judge baseline (the strongest baseline) on the AlpacaEval 2 benchmark. The results are presented in $\\textrm{\\color{blue}Table. 6}$ of the paper revision.

The table reveals a decline in LC win rates at the third iteration for both our approach and the LLM-as-a-Judge baseline. This aligns with a known challenge in the literature: both self-rewarding language models and SPPO demonstrate consistent improvement only up to three iterations, which corresponds to two iterations in our approach, as our method begins after one initial iteration of DPO training.

We agree with the reviewer's comment that the performance drop in the third iteration may result from the policy that is becoming deterministic over iterations - reduced diversity in preference tuning is a known issue [Kirk et al.]. This reduction in response diversity has negative effects on DPO training [Pal et al.]. It would be an interesting and valuable direction for future work that studies if sampling new responses with a very high temperature can lead to improvement over 3 iterations.

[Kirk et al.] Understanding the Effects of RLHF on LLM Generalisation and Diversity

[Pal et al.] Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

W4. Mathematical notations and statements.

I do not understand why using $\\hat{r}$ instead of $r\^*$ (this is the optimal reward!) l. 34

We agree that $r\^*$ is a more standard notation for the (optimal) reward oracle, and we have updated it accordingly.

“For a prompt x, we denote its optimal response as $y\^⋆$” --> in general there can be multiple optimal responses (as there can be multiple optimal policies), please adapt the statement & but here authors mean preferred response? this really needs clarification

We would like to clarify that y\^\\star denotes the optimal response, not the preferred response. The optimality is with respect to the optimal reward r\^\\star, such that y\^\\star \\in \\arg\\max\_{y} r\^\\star(y,x). For simplicity, we assume that there is a single optimal response with respect to the optimal reward, i.e., y\^\\star = \\arg\\max\_{y} r\^\\star(y,x). Note that the optimal policy is uniquely determined by the optimal reward r\^\\star (up to an additive constant), and their relationship is specified by the equation in Line 34 of the Introduction section. We hope these clarifications could address your question and we have updated accordingly in our revised paper.

W3. Line-by-line comments

Algorithm 1: worth stating that the dataset is created by taking the best and worst samples according to the (length-penalized) reward

We have updated accordingly.

Algorithm 1: I believe there is a typo where it says to use "pi_ref^(t-1) as reference policy", and it should be pi_ref^(t)

This is the last comment after line 4 in Algorithm 1, which specifies the target ($\\pi\_\\theta$) and reference ($\\pi\_{\\mathrm{ref}}$) policies which we use to evaluate the length-regularized reward: $r\_{\\mathrm{LR}}(x,y;\\alpha) = \\beta \\log \\frac{\\pi\_\\theta(y|x)}{\\pi\_{\\mathrm{ref}}(y|x)} - \\alpha |y|$. Suppose we are constructing the dataset for round $t$, we need to ensure that $\\pi\_\\theta$ is the trained model after round $t-1$, and that $\\pi\_{\\mathrm{ref}}$ is the reference model used in round $t-1$. This is to ensure that the DPO training in round $t-1$ makes the first term in $r\_{\\mathrm{LR}}$ a valid implicit reward to be used in round $t$.

Therefore, we would like to clarify that this is not a typo. In Algorithm 1, taking $t=1$ as an example, $r\_{\\mathrm{LR}}$ is obtained with $\\pi\_{\\theta\_0}$ and $\\pi\_{\\mathrm{ref}}\^{0}$, which are exactly the initial DPO-tuned policy and the initial reference policy (used to tune $\\pi\_{\\theta\_0}$) given in the algorithm input (as well as in $\\textrm{\\color{blue}line 201}$).

Suggestions to modify the abstract

Thanks for this suggestion and we have updated it accordingly.

Line 216: might be good to clarify where does pi_mu comes from in practice

$\\pi\_\\mu$ is the behavior policy that collects the offline dataset, which can be a mixture of existing LLMs or human experts. We updated with a footnote in the revised paper (the original Section 4.1 is moved to Appendix A for better logical flow and readability).

Line 227: "without minimizing the likelihood of other suboptimal responses" - I didn't understand what point you were trying to make, since the loss doesn't depend on these other responses by definition?

Yes, given the offline dataset, the DPO loss can be minimized to zero by only minimizing the likelihood of the losing response appearing in the dataset, while other suboptimal responses out of the dataset can still remain high likelihood.

Equation (9) - It would seem more natural to put the absolute value outside the expectation instead of inside

Thank you for this insightful point. We apologize that the original Eq.9 (now Eq. 6 in the revised paper) contains a typo. We have carefully checked our implementation and confirmed that we actually took the absolute value outside the expectation. So all our experiments use the version that you have kindly described. Running the alternative gives 0.021, which is close to 0.023. We have fixed the typo in Eq.6.

Q1. Are the base models both trained on the same UltraFeedback dataset from which the offline dataset is sampled?

Yes. We have clarified this in Section 5.1. Thank you for your suggestion.

Q2. What is the length bias like for LLM-as-a-Judge?

We refer to $\\textrm{\\color{blue}Figure. 4 in Appendix C}$ for the average absolute length difference with both our implicit reward model and the LLM-as-a-judge reward model. The figures plot the optimization landscapes of Eq. 6, where we observe that the optimal $\\alpha$ is $0$ for LLM-as-a-judge, indicating that it seems not to cause length bias. Therefore, we do not apply a similar technique to post-processing the preference dataset for LLM-as-a-Judge baseline.

Thank you for your encouraging review, valuable suggestions, and additional supportive comments. Below we address the points raised under Weaknesses (W) and Questions (Q).

Note: We've marked "3RHj" next to relevant changes in the revised paper to make it easier for you to navigate.

W1. Clarification on hyperparameter tuning.

Thank you for pointing this out. In our experiments, the hyperparameters, including $\\gamma$ and $\\beta$, were selected based on the model performance on AlpacaEval 2 (AE2). These parameters were tuned for each method and model separately (with $\\gamma$ specific to our approach) but remained fixed across iterations. Specifically, $\\beta$ was tuned during the first iteration from the set {0.01, 0.1}, while $\\gamma$ was tuned using the ranges {0.0, 0.25, 0.50, 0.75, 1.0} for the Zephyr backbone and {0.0, 0.10, 0.25, 0.50, 0.75, 1.0} for the Llama-3 backbone. The parameter $K$ was fixed at 16 for simplicity in implementation - we could generate all the responses in 2 shots on an 8-GPU server using data parallelism. The best hyperparameter setting identified in AlpacaEval 2 was directly applied in Arena-Hard (AH) for all methods and models without any further tuning.

We agree with the reviewer that using a validation set for hyperparameter tuning would be more rigorous. Following the best practice, hyperparameters should be optimized on a validation set, and the model's performance should be subsequently evaluated on a separate test set or benchmark. As it would take a long time to construct a validation set/benchmark and conduct the whole pipeline with thoroughly tuned hyperparameters, we try to address this concern by treating the two benchmarks (AE2 and AH) as proxies for validation and test sets. Specifically, we validate on AE2 while testing on AH, and then reverse the roles of the two benchmarks. 

Below we show the results in $\\textrm{\\color{blue}Table. R2.1}$ and $\\textrm{\\color{blue}Table. R2.2}$: 

$\\textrm{\\color{blue}Table. R2.1}$ Model performance on AlpacaEval 2 (AE2) and Arena-Hard (AH) across different $\\gamma$ for the Zephyr Backbone. LC denotes the length-controlled win rate. CI denotes confidence interval. 

$\\textrm{\\color{blue}Table. R2.2}$ Model performance on AlpacaEval 2 (AE2) and Arena-Hard (AH) across different $\\gamma$ for the Llama-3 Backbone. LC denotes the length-controlled win rate. CI denotes confidence interval. 

The two benchmarks and two backbone models constitute four validation-test instances. We observe that the $\\gamma$ selected from the validation set/benchmark consistently achieves the best performance on the test set across all instances, demonstrating the optimal value of $\\gamma$ is robust to the choice of the validation set/benchmark. Meanwhile, we do observe that the optimal $\\gamma$ for different backbone models is varied, which is expected, as different backbones produce responses of varying quality. Higher-quality generated data reduces the need for replayed experience, aligning with our observations for the Llama-3 backbone.

W2. Comparison to reward model-based approach like PPO.

Thank you for suggesting this relevant comparison. It would be interesting to explore how the implicit reward model interacts with PPO. We are actively working on these comparisons and will provide a follow-up comment once the experiments are completed.

Great, I think these changes improve the paper. I am happy to see that other reviewers have increased their scores, and will continue to advocate for acceptance of the paper. I will keep my score as is since it was already in anticipation of such improvements.

W2. Limited conceptual novelty.

Although the implicit reward has been acknowledged in prior works, our study investigates the underexplored question of whether the implicit reward model can be effectively utilized to further improve the LLM itself, and we answer the question in the affirmative. We believe our findings offer a fresh perspective to the self-alignment literature and, more importantly, it provides special advantages in the scenario where the general-purpose scalar reward model fails to generalize (see discussion in W1). 

Regarding the length regularization, we would like to clarify that while prior works [Meng et al., Park et al.] in LLM alignment have investigated variants of length regularization, these approaches typically incorporate the regularization term into the training loss function. However, such approaches often require extensive hypertuning (e.g., involving multiple training runs that may take hours). In contrast, our approach applies reward shaping before DPO training to construct the preference data, which can be completed within a minute.

[Meng et al.] Simpo: Simple preference optimization with a reference-free reward.

[Park et al.] Disentangling length from quality in direct preference optimization.

W3. Clarification on section 4.1.

Following your suggestion, we have moved the original Section 4.1 to $\\textrm{\\color{blue}Appendix. A}$ to improve the flow of the revised paper. Below, we provide clarifications on your questions.

Equation (6) seems wrong to me unless you mean S to be ALL possible completions besides the "optimal" one

Yes, $\\mathcal{S}$ refers to the set of all suboptimal responses. The main purpose of this analysis is to give intuitions on why on-policy sampling could benefit iterative DPO training, which we employ to build our self-alignment pipeline. Here, we assume the "optimality" of responses is measured according to the oracle reward, instead of the learned DPO implicit reward, so there should not be over-optimization or OOD issues. Nevertheless, we admit this section may cause misunderstanding, especially when put in our method section. We have moved it to $\\textrm{\\color{blue}Appendix. A}$ and added necessary footnotes.

W4. Minor concerns

I would call the length penalty reward specification/augmentation/engineering rather than reward shaping, which in my experience is reserved for rewards that are meant to improve the learning process but maintain the same desired policy.

Thank you for your insightful comment. While we used the term "reward shaping" to describe our method of adaptively adjusting the reward based on response length, we acknowledge its more specific usage in the literature. To avoid confusion during the rebuttal, we will retain the term for now but will consider adopting a more precise term, such as "reward specification" suggested by the reviewer, in a future revision.

Tortured acronym that is already used elsewhere. Consider something more unique that actually maps to the method name.

Thank you for highlighting the overlap and suggesting alternatives. We will explore a more unique and descriptive acronym for a future revision.

Thank you for your valuable review and suggestions. We appreciate the recognition of iterative training with self-supervised data, also named self-alignment, as an important research direction. Below we respond to the comments in weaknesses (W) and questions (Q).

Note: We've marked "A3Uc" next to relevant changes in the revised paper to make it easier for you to navigate.

W1 & Q1. Experiments lack of comparison with other external reward model.

Thank you for raising this question. We would like to clarify that our work adheres to the self-alignment setup, which focuses on improving LLMs without external supervision [Yuan et al., Sun et al, Gao et al., Zhang et al.] (e.g., powerful reward models from RewardBench). This setup is particularly important because, in some scenarios, obtaining an external reward model is challenging such as for models tailored to specialized domains (e.g., medical or legal language), where a general-purpose reward model may not be suitable [Colombo et al.]. Therefore, investigating methods for improving language models without external reward models remains valuable, as also highlighted in this official comment of Reviewer 3RHj.

Since our method relies on an initial round of DPO with human preference data (i.e., the seed data), we acknowledge that a relevant baseline would involve first training an internal reward model (IntlRM) using the same seed data and then leveraging this IntlRM for iterative improvement. This comparison allows us to evaluate the effectiveness of our DPO implicit reward against a dedicated reward model within the self-alignment setup.

The IntlRM training follows the standard setting of OpenRLHF [Hu et al.], and we have included the training scripts in the updated Supplementary Material for reproducibility. The following table compares the performance of our DPO implicit reward and IntlRM:

The experiment results indicate that the DPO implicit reward achieves higher accuracy than the IntlRM trained on the same preference data. Furthermore, it surpasses the performance of an external reward model, ERM-555k, trained on substantially more data. This implies that the implicit reward model offers distinct advantages when evaluating its own generated data, though the scalar reward models in general excel on a wider range of tasks when the preference data is abundant, as demonstrated by ArmoRM-Llama3-8B-v0.1, which was trained on 1M preference data. Based on these findings, we argue that the implicit reward is a competitive option in the self-alignment setting. For further details, refer to $\\textrm{\\color{blue}Section 5.4}$ and $\\textrm{\\color{blue}Table. 5}$ in the paper revision.

[Yuan et al.] Self-Rewarding Language Models

[Sun et al.] SALMON: Self-Alignment with Instructable Reward Models

[Gao et al.] Human-Instruction-Free LLM Self-Alignment with Limited Samples

[Zhang et al.] Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation

[Colombo et al.] Saullm-7b: A pioneering large language model for law.

[Hu et al.] OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Dear Reviewer A3Uc,

Thank you once again for your constructive feedback. We would like to kindly remind you that we have included comparisons with other reward models in $\\textrm{\\color{blue}Section 5. 4}$ and $\\textrm{\\color{blue}Table. 5}$ of the paper revision. Additionally, we have clarified our contributions and improved the overall writing of the paper.

As the discussion period is coming to a close in two days, we look forward to your response and would be happy to address any further comments or questions you may have.

Best,

The Authors

W3. Length-regularized reward shaping is a widely adopted technique.

While prior works in LLM alignment have explored variants of length regularization, these approaches typically integrate the regularization term into the training objectives. In contrast, our approach applies it at the data construction stage in the iterative DPO framework. This distinction offers us an advantage in efficiency. Determining the best regularization coefficient in training objectives often requires multiple training runs, whereas our method can adaptively determine the optimal value for $\\alpha$ in a minute, as acknowledged in the Strengths section of Reviewer 3RHj.

[Meng et al.] Simpo: Simple preference optimization with a reference-free reward.

[Park et al.] Disentangling length from quality in direct preference optimization.

W4. Theoretical explanation of why repeated use of the implicit reward model can enhance performance.

We hypothesize that the performance gains of our approach arise from the iterative DPO process combined with on-policy sampling. The key intuition behind on-policy sampling is that it allows the LLM to continually generate preference pairs, reducing the likelihood of negative examples over time. This is in contrast to repeatedly optimizing over a fixed set of negative examples, which may already have a low likelihood and thus contribute minimally to further improvements.

In Section 4.1 of the original paper (now moved to $\\textrm{\\color{blue}Appendix A}$ for better logical flow and readability), we provide a theoretical analysis highlighting the advantages of learning with on-policy samples. Furthermore, we empirically validate the effectiveness of the implicit reward in our response to W5.

W5. Advantages of implicit reward model compared with the external reward model.

As discussed in our response to W1, this paper aims to answer whether a DPO-tuned LLM can be further improved without relying on external models, such as a stronger LLM or an external reward model. Therefore, comparisons with external reward models are outside the scope of this work. 

To ensure a fair comparison within our framework, we trained a scalar reward model using the same offline preference dataset employed for DPO training. We refer to this model as the internal reward model (IntlRM). The IntlRM training follows the standard setting of OpenRLHF [Hu et al.], and we have included the training scripts in the updated Supplementary Material for reproducibility.

We compared the IntlRM with the DPO implicit reward model in $\\textrm{\\color{blue}Table. 5}$ of the revised paper. The results indicate that the DPO implicit reward achieves higher accuracy than the IntlRM trained on the same dataset. Furthermore, it surpasses the performance of an external reward model, ERM-555k, trained on substantially more data. This implies that the implicit reward model offers advantages when evaluating its own generated data, though the scalar reward models in general excel on a wider range of tasks when the preference data is abundant, as demonstrated by ArmoRM-Llama3-8B-v0.1, which was trained on 1M preference data. Based on these findings, we argue that the implicit reward is a competitive option and its superiority can potentially lead to better performance in iterative training.

[Hu et al.] OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Thank you for your valuable review and suggestions. We appreciate that you find our results provide useful insights. Below, we address the comments on Weaknesses (W) with empirical evidence. 

Note: We've marked "6mfr" next to relevant changes in the revised paper to make it easier for you to navigate.

W1. Proposed approach lacks significant novelty.

Thank you for raising this concern. We would like to highlight the key contributions that distinguish our work from prior research:

1. Our work proposes to use a principled alternative reward model, specifically the implicit reward derived from DPO training, in the self-alignment (self-improving fine-tuning) literature. In contrast, prior approaches primarily rely on prompting the language model for rewards [Yuan et al., Sun et al.] using the LLM-as-a-judge technique;
2. In addition, we propose two techniques, length-regularized reward shaping and experience replay, that are shown to be essential for achieving optimal performance with our approach.

Although the implicit reward has been acknowledged in prior works, our study investigates the underexplored question of whether the implicit reward model can be effectively utilized to further improve the LLM itself, and we answer the question in the affirmative. We believe our findings offer a fresh perspective to the self-alignment literature and also provide a practical solution for practitioners seeking improvements over the vanilla DPO, as also highlighted in this official comment of Reviewer 3RHj.

[Yuan et al.] Self-Rewarding Language Models

[Sun et al.] SALMON: Self-Alignment with Instructable Reward Models

W2. Improvement of the proposed approach is marginal.

To address your concerns about the extent of improvement, we present both the absolute and relative improvements in LC (length-controlled win rate) on AlpacaEval 2 (AE2) and WR (win rate) on Arena-Hard (AH). These improvements are summarized in the tables below:

$\\textrm{\\color{blue}Table. R1.1}$ Absolute and relative improvements of LC on AE2 and WR on AH for the Zephyr Backbone.

$\\textrm{\\color{blue}Table. R1.2}$ Absolute and relative improvements of LC on AE2 and WR on AH for the Llama-3 Backbone.

The values are taken or derived from $\\textrm{\\color{blue}Table. 1}$ in the paper. Absolute improvement refers to the direct difference in performance metrics between any evaluated method (e.g., our approach or a baseline) and the base model. Relative improvement is calculated as the absolute improvement divided by the base model's score. The Highest Baseline refers to the best score across all baselines in $\\textrm{\\color{blue}Table. 1}$. Across all benchmarks and backbones, our approach demonstrates substantial improvements (as also acknowledged by all the other three reviewers), with minimal relative gains of 51.37% on AE2 and 68.69% on AH. Notably, our method consistently outperforms the best baseline by a significant margin.

Dear Reviewer 6mfr,

Thank you once again for your constructive feedback. We would like to kindly remind you that we have clarified our contributions, highlighted our method's significant performance improvements, and included comparisons with additional reward models in $\\textrm{\\color{blue}Section 5. 4}$ and $\\textrm{\\color{blue}Table. 5}$ of the paper revision.

As the discussion period is coming to a close in two days, we look forward to your response and would be happy to address any further comments or questions you may have.

Best,

The Authors