$\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$

Dear Reviewer,

Thanks for your kind review. We are glad that you found our paper meaningful and easy to follow. We provide detailed answers to your comments below.

Q1: It is hard to evaluate the performances on SOTA (relatively large) models, e.g., 7B or 8B models.

A1: Thank you for raising this concern. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models SimPO[3]. We perform $\beta$-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For comparison with baselines, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1]. Please refer to Table below:

Table: AlpacaEval 2 results under the Mistral-Instruct (7B) and Llama3-Instruct (8B). LC and WR denote length-controlled and raw win rate, respectively. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.

Q2: This paper fails to show how the dynamic terms are derived.

A2: Thank you for your suggestion. Our work primarily identifies an empirical relationship between $\beta$ and data quality, validated across various datasets and model architectures. While we recognize that the theoretical foundations warrant further exploration, the proposed dynamic strategy is nonetheless notable, providing a new and effective paradigm for fine-tuning large models and studying data quality.

Q3: Is it harder to tune compared to the vanilla DPO with only one hyperparameter?

A3: Our proposed $\beta$-DPO is a straightforward, efficient, and easily transferable strategy. Firstly, the selection of $\beta\_0$ in all instances of $\beta$-DPO is consistent with the $\beta$ used in DPO, with a default choice of 0.1 (0.01 for UltraChat-200k), eliminating the need for extensive hyperparameter tuning for $\beta$. As for the uniquely introduced $\alpha$, we find:

In most scenarios, setting $\alpha = \frac{2}{M_0}$ yields stable performance improvements, where $M_0$ can be estimated using a moving average updating scheme (refer to Equation 7). This is informed by the formula $\beta\_{\text{batch}} = [1 + \alpha(\mathbb{E}\_{i \sim \text{batch}}[M\_i] - M\_0)]\beta\_0$, resulting in an overall change range of $[\frac{2\mathbb{E}\_{i \sim \text{batch}}[M\_i] - M\_0}{M\_0}]\beta\_0$, which normalizes based on $M\_0$ over the foundation of $\beta\_0$.

To substantiate this perspective, we present performance in the above table, demonstrating that our setting achieves significant enhancements across various datasets and models compared to DPO, without imposing additional pressure for hyperparameter searches. We appreciate your concern; while we believe that further theoretical consolidation is a meaningful future endeavor, we maintain that the $\beta$-DPO approach remains valuable, offering a straightforward (not overly reliant on hyperparameter tuning) and effective (stable performance enhancements) new paradigm for fine-tuning large models and studying data quality.

Q4: How does the algorithm capture the global information through batch-level data?

A4: We employed a moving average estimator to estimate the global reward margin, as referenced in Equation 7. This technique has been commonly used in deep learning [1, 2].

Q5: The winning rates are only evaluated by GPT-4 instead of real human groups.

A5: Thank you for your suggestion. As it stands, the current research landscape indicates that GPT-4 provides the most common strategy for validation. Moreover, the alignment between GPT-4 assessments and human evaluations has been sufficiently established in existing benchmarks (AlpacaEval 2).

[1] Yuan et. al. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. ICML2022.

[2] ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION. ICLR 2015.

[3] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng, Mengzhou Xia, and Danqi Chen.

Thank you for your response. Regarding the question about batch size and calibrating the learning rate, we provide the following elaboration:

Q1: As the calibration is better at the batch level than the instance level, do you have an ablation on batch size?

A1: We have experimented with different batch sizes for $\beta\_{\text{Batch}}$ on the Pythia-410M model, and the results are as follows:

Batch Size	WR (%)
128	30.18
64	29.35
32	28.78

The above results demonstrate that larger batch sizes lead to better model performance on Pythia-410M model. Our intuitive understanding of this observation is:

• Larger batch sizes enable more accurate estimation of $\beta$.
• In the extreme case where batch size = 1, batch-level calibration degrades to instance-level, resulting in high instability in the estimation of $\beta\_{\text{Batch}}$.

We appreciate your suggestion. Due to the limited time for discussion, we will further investigate this conclusion's reliability by experimenting with other model sizes (Pythia 2.8B, Mistral-Instruct (7B) | Llama3-Instruct (8B)).

Q2: Is it equivalent to calibrating the learning rate instead of $\beta$?

A2: Dynamic $\beta$ is not equivalent to calibrating the learning rate. We can further analyze this issue from the perspective of gradients. $\nabla\_\theta \mathcal{L}\_\text{DPO}(\pi\_\theta;\pi\_{\text{ref}}) = -\beta\mathbb{E}\_{(x, y\_w, y\_l) \sim \mathcal{O}}[\sigma(\hat{r}\_\theta(x, y\_l) - \hat{r}\_\theta (x, y\_w)) (\nabla\_{\theta, y\_w} - \nabla\_{\theta, y\_l})]$ where $\sigma(\hat{r}\_\theta(x, y\_l) - \hat{r}\_\theta (x, y\_w)) =\sigma(\beta \log \frac{\pi\_{\theta}(y\_l|x)}{\pi\_{\text{ref}}(y\_l|x)} - \beta \log \frac{\pi\_{\theta}(y\_w|x)}{\pi\_{\text{ref}}(y\_w|x)})$.

Therefore, the gradients are correlated with $\beta \sigma(\beta [\log \frac{\pi\_{\theta}(y\_l|x)}{\pi\_{\text{ref}}(y\_l|x)} - \log \frac{\pi\_{\theta}(y\_w|x)}{\pi\_{\text{ref}}(y\_w|x)}] )$, but non-linearly correlated with $\beta$. Here, $[\log \frac{\pi\_{\theta}(y\_l|x)}{\pi\_{\text{ref}}(y\_l|x)} - \log \frac{\pi\_{\theta}(y\_w|x)}{\pi\_{\text{ref}}(y\_w|x)}] < 0$. Consequently, as $\beta$ increases, $\sigma(\beta [\log \frac{\pi\_{\theta}(y\_l|x)}{\pi\_{\text{ref}}(y\_l|x)} - \log \frac{\pi\_{\theta}(y\_w|x)}{\pi\_{\text{ref}}(y\_w|x)}] )$ approaches 0, and the entire gradient also approaches 0.

Considering the gradient update: params = old_params - lr * grad Directly calibrating the learning rate cannot achieve the same effect as calibrating $\beta$. However, if an appropriate construction of lr can be found (high gap --> small learning rate), the experience from $\beta$-DPO suggests that there may be some improvements.

We sincerely appreciate your valuable feedback and eagerly anticipate integrating these improvements into our manuscript. Please let us know if you have any further concerns, and we are encouraged to have a discussion.

Dear Reviewer,

Thanks for your kind review. We are glad that you found our paper meaningful and easy to follow. We provide detailed answers to your comments below.

Q1: It would be nice if the author can verify the results on another dataset with different domains.

A1: Thank you for raising this concern. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models SimPO[1]. We perform $\beta$-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For comparison with baselines, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1]. Please refer to Table below:

Table: AlpacaEval 2 results under the Mistral-Instruct (7B) and Llama3-Instruct (8B). LC and WR denote length-controlled and raw win rate, respectively. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.

Q2: What is a usual range of beta in experiment? Is it possible that goes to negative?

A2: Our initial $\beta\_0$ is set to 0.1, and the range of beta in experiments falls within [0.0, 0.4]. The specific range of variation is detailed in REBUTTAL Figure 2 Middle. Experimentally, $\beta\_{\text{batch}}$ can become negative. According to Equation 6, a negative value implies that the reward discrepancy is negative or $\mathbb{E}\_{i \sim \text{batch}}[M\_i]$ is significantly lower than the global mean of $M\_i$, indicating a high likelihood of outliers. Therefore, we apply a cutoff at 0 to ensure training stability, $\beta\_{\text{batch}} = \max(0.0, \beta\_{\text{batch}})$.

Q3: What if you sort the data in an increasing order and select top instances?

A3: Thank you for pointing this out. This method represents a hard selection approach, whereas our method utilizes soft selection. The hard selection concept aligns with what we have shown in Figure 5 Left (Filter Head & Tail), and we believe both are effective filtering methods. Both approaches validate that extreme $M\_i$ values are likely outliers that need filtering.

It is important to highlight that this work does not propose a novel filtering method, but we find that filtering enhances stability. As shown in Figure 5 Left, dynamic scheduling could also improve other filtering methods. We are confident that our dynamic schedule will continue to provide stable performance improvements as better LLM data filtering techniques emerge in the future.

[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng, Mengzhou Xia, and Danqi Chen.

Thanks for your kind review. We are glad that you found our paper meaningful and easy to follow. We provide detailed answers to your comments below.

Q1: The experiments are limited to specific datasets and model size ranges.

A1: Thank you for raising this concern. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models SimPO[1]. We perform $\beta$-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For comparison with baselines, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1]. Please refer to Table below:

Table: AlpacaEval 2 results under the Mistral-Instruct (7B) and Llama3-Instruct (8B). LC and WR denote length-controlled and raw win rate, respectively. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.

Q2: Comparison with gold reward.

A2: Thank you for pointing this out. In fact, computing the gold reward for the training set of 160k samples 1) greatly increases computational complexity and 2) makes the gold reward highly sensitive to the particular model used for annotation. Such an approach is difficult to generalize to arbitrary data scenarios. Additionally, to reduce the instability of "running" reward discrepancies, $\beta$-DPO utilizes batch-level calibration. Figure 5 (right) and Table 2 clearly demonstrate its superiority. This aligns with our original intent for this work: to develop a simple-to-implement, highly scalable dynamic $\beta$ strategy.

Q3: Including the $\beta$-guided filtering strategy.

A3: Thank you for this suggestion. We have included the complete comparison chart in REBUTTAL Figure 2 Left. It can be clearly observed that: 1) the dynamic $\beta$ strategy can adapt to various data filtering methods, and 2) the $\beta$-guided filtering proposed in this paper remains optimal.

[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng and Mengzhou Xia and Danqi Chen.

Dear Reviewer,

Thanks for your kind review. We are glad that you found our paper meaningful and easy to follow. We provide detailed answers to your comments below.

Q1: There are very few details about the reward models used in the paper

A1: Thank you for pointing this out. We directly use the implicit reward model induced by the policy trained by DPO, where the reward discrepancy in DPO is expressed as: $\beta \log (\frac{\pi\_\theta (y\_w \mid x) }{\pi\_{\text{ref}}(y\_w \mid x)}) - \beta \log (\frac{\pi\_\theta (y\_l \mid x) }{\pi\_{\text{ref}}(y\_l \mid x)})$.

Q2: Why are low-gap examples considered high-quality?

A2: We appreciate your concern regarding this matter. First, it is crucial to clarify that both the low-gap and high-gap examples are derived from data of a certain quality, rather than from extremely poor quality sources characterized by meaningless chatter. We posit that datasets of extremely poor quality would not be utilized for training at this stage. Furthermore, we can categorize the cases of $y\_w$ and $y\_l$ into the following four types:

To further elucidate, both preferred and dispreferred responses are low-quality, yet they serve a beneficial purpose. We follow the settings of controlled sentiment generation in DPO. With access to the ground-truth reward function (a sentiment classifier), we control the quality of $y\_w$ and $y\_l$, where high-quality response is derived from a fine-tuned GPT-2 large model and low-quality response from an unrefined GPT-2 large model. The following table, derived from the IMDB test dataset, illustrates the KL-divergence and the associated rewards:

The results indicate that low-quality responses can be even more meaningful for model improvement than high-gap examples.

Q3: Why is the Anthropic HH dataset considered low-gap? Are there better datasets available?

A3: The terms "low-gap" and "high-gap" are employed in a relative context. In comparison to the negative samples generated by SFT, the overall distribution of Anthropic HH exhibits smaller differences. To support this assertion, please refer to Appendix Figure 6, where positive samples originate solely from the HH dataset, while negative samples consist of a mix from SFT-generated and original negative samples from Anthropic HH. As the proportion of original Anthropic HH samples increases (i.e., the mixture ratio decreases), the distribution of reward margins becomes more concentrated.

Q4: How is beta utilized in data filtering?

A4: In this work, the reward margin informs both the choice of $\beta$ and the framework for data filtering, establishing a bridge between $\beta$ and data filtering. The specific expression is: $p(\beta\_i)= \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{(\beta\_i/\beta\_0-1)^2}{2\sigma^2\alpha^2})$

Q5: How is the selective filtering ablation conducted?

A5: We apologize for any misunderstanding in our expression. Your interpretation is correct; we operate based on the reward margin. We will revise this section for clarity. In the DPO formulation, the sample gradients are strictly negatively correlated with the reward margin, implying that a larger reward margin corresponds to a smaller gradient. Therefore, during loss computation for each batch, we sort samples based on their reward margins and conduct selection accordingly. In the experiment corresponding to Figure 5 (Left), "Filter Tail 20%" refers to the filtering of the 20% of samples with the largest gradients, which corresponds to those with the smallest 20% reward margins, and vice versa.

Q6: Is m (the momentum coefficient) also a hyperparameter?

A6: m (the momentum coefficient) is indeed a hyperparameter. However, in all our experiments, we consistently set it to 0.9 without any hyperparameter search. To further illustrate, we present additional comparisons in REBUTTAL Table 2, showing that this parameter has a minimal effect on model performance.

Q7: Current results are only on the Pythia class of models

A7: Thank you for raising this concern. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models [1]. We perform beta-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For baseline comparisons, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1]. Please refer to REBUTTAL Table 1. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.

[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng and Mengzhou Xia and Danqi Chen.

Dear Reviewer,

Thanks for your kind review. We are glad that you found our paper clear and easy to follow. We provide detailed answers to your comments below.

Q1: How is the reward model constructed and used?

A1: Thank you for pointing this out. We directly use the implicit reward model induced by the policy trained by DPO, where the reward discrepancy in DPO is expressed as: $\beta \log (\frac{\pi\_\theta (y\_w \mid x) }{\pi\_{\text{ref}}(y\_w \mid x)}) - \beta \log (\frac{\pi\_\theta (y\_l \mid x) }{\pi\_{\text{ref}}(y\_l \mid x)})$.

Q1.1: As a consequence, using intermediate rewards for additional online choices (such as tuning and filtering) may lack theoretical soundness.

A1.1: In the early stages of training, $\beta$-DPO behaves similarly to DPO (see REBUTTAL Figure 2 Middle). Due to the low accuracy of the intermediate implicit models, the margin between winners and losers is minor, resulting in $\beta\_{\text{batch}} \approx \beta\_0$. However, in the later stages of training, as the margin begins to exhibit significant differences, the advantages of a dynamic $\beta$ become apparent.

Q1.2: An implicit DPO reward model introduces greater non-stationarity into the process.

A1.2: This question raises a pertinent point. An implicit DPO reward model can lead to instability in the estimation of dynamic $\beta$, thus making the entire training process more non-stationary. To address this issue, we propose that batch-level calibration is essential. As demonstrated in Table 2 of the initial manuscript, instance-level calibration on dynamic $\beta$ results in a substantial performance drop. In contrast, batch-level calibration enhances stationarity and accentuates the effects of dynamic $\beta$. Additionally, Figure 5 (Right) illustrates that instance-level calibration exacerbates the influence of outliers.

Q2: It is unclear when only one or both techniques are necessary.

A2: As demonstrated in Table 1 of the original manuscript, we observe that in a smaller model (410M), the improvement of data filtering is more significant, while in a larger model (2.8B), the improvement of dynamic $\beta$ is more significant. Additionally, the dynamic schedule can also improve other filtering methods (Figure 5 Left). Since our filtering method relies only on implicit reward discrepancy, it also works with other PO methods (Figure 5 Middle).

Q3: Can you strengthen your support for the gap principle beyond the empirical observation of Figure 1?

A3: We also attempt the exact opposite (high gap -> low $\beta$; low gap -> high $\beta$). Refer to REBUTTAL Figure 1, REBUTTAL Figure 2 Right for the experimental results. We find that across multiple datasets and models, the current strategy remains optimal.

Q4: Outlier filtering is performed using only an unimodal 3-sigma principle

A4: Utilizing the 3-sigma principle is primarily to reduce the bias in $\beta$ estimation and thus enhance training stability. Intuitively, high-gap samples often carry low information content, whereas low-gap samples might still be noisy, indicating that winners and losers may not be well distinguished (label flipping). We hope this helps you gain an intuitive understanding. A more detailed analysis can be found in the original manuscript, Section 4.1 (lines 157-170).

Q5: Testing the approach beyond DPO

A5: Thank you for pointing this out. First, we demonstrate Dynamic $\beta$ Enhancement across IPO, KTO, and SPPO in Figure 5 (Middle) of the original paper. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models SimPO[1], which surpass c-DPO/RPO etc. We perform $\beta$-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For comparison with baselines, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1]. Please refer to REBUTTAL Table 1. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.

Q6: Automated parameter tuning.

A6: In most scenarios, setting $\alpha = \frac{2}{M_0}$ yields stable performance improvements, where $M_0$ can be estimated using a moving average updating scheme (refer to Equation 7). This is informed by the formula $\beta\_{\text{batch}} = [1 + \alpha(\mathbb{E}\_{i \sim \text{batch}}[M\_i] - M\_0)]\beta\_0$, resulting in an overall change range of $[\frac{2\mathbb{E}\_{i \sim \text{batch}}[M\_i] - M\_0}{M\_0}]\beta\_0$, which normalizes based on $M\_0$ over the foundation of $\beta\_0$.

To substantiate this perspective, we present performance in the above table, demonstrating that our setting achieves significant enhancements across various datasets and models compared to DPO, without imposing additional pressure for hyperparameter searches. We appreciate your concern; while we believe that further theoretical consolidation is a meaningful future endeavor, we maintain that the $\beta$-DPO approach remains valuable, offering a straightforward (not overly reliant on hyperparameter tuning) and effective (stable performance enhancements) new paradigm for fine-tuning large models and studying data quality.

[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng and Mengzhou Xia and Danqi Chen.

Dear reviewer,

We greatly appreciate your invaluable feedback. We now aim to provide a concise summary that carefully addresses your main concerns. We hope that this effort will be worthy of your support.

Q1: How is the reward model constructed and used?

A1: As you mentioned in your second point, we directly utilize the implicit reward model induced by the policy trained with DPO, where the reward discrepancy in DPO is expressed as: $\beta \log (\frac{\pi_\theta (y_w \mid x) }{\pi_{\text{ref}}(y_w \mid x)}) - \beta \log (\frac{\pi_\theta (y_l \mid x) }{\pi_{\text{ref}}(y_l \mid x)})$.

Q1.1: As a consequence, using intermediate rewards for additional online choices (such as tuning and filtering) may lack theoretical soundness.

A1.1: In the early stages of training, $\beta$-DPO exhibits similar behavior to DPO (REBUTTAL Figure 2 Middle). Due to the low accuracy of the intermediate implicit models, the margin between winners and losers is small, resulting in $\beta_{\text{batch}} \approx \beta_0$. However, as training progresses and the margin starts to show significant differences, the benefits of a dynamic beta become evident.

Q1.2: An implicit DPO reward model introduces greater non-stationarity into the process.

A1.2: An implicit DPO reward model can indeed lead to instability in the estimation of dynamic $\beta$, making the entire training process more non-stationary. To mitigate this issue, we propose that batch-level calibration is crucial. As shown in Table 2 of the original manuscript, instance-level calibration on dynamic $\beta$ leads to a significant performance drop. In contrast, batch-level calibration improves stationarity and emphasizes the effects of dynamic $\beta$.

Furthermore, the moving average updating scheme on $M_0$ (ref to Equation 7) helps alleviate the impact of （Q1.1） the poor reward model and （Q1.2） potential instability. Although the reward margin increases further ($M_i$ increases) as the model's capability improves, $\beta_{\text{Batch}}$ is positively correlated with $M_i - M_0$. Consequently, $\beta$-DPO focuses more on the dynamic reward of different datapoints relative to global datapoints, rather than their absolute values. This approach may help reduce the impact of the changing reward margins on the overall stability of the system. The evolution of $\beta_{\text{Batch}}$ over the course of training steps, as illustrated in REBUTTAL Figure 2 Middle, supports this perspective.

We were wondering if our responses have addressed your concerns since the discussion phase is coming to a close. We are also eager to know if you have any other concerns or suggestions. Thank you for your time and consideration!

First of all, we would like to express our sincere gratitude to all reviewers for their time and efforts in evaluating our submission. As the Author-Reviewer discussion period is drawing to a close, we would like to summarize the discussion results and the improvements made to our paper.

Below, we summarize our discussions and feedback with the reviewers:

• Reviewer $\color{red}{\text{fXTA}}$: We have provided a detailed explanation of the implicit RM expression used in this paper and how it mitigates the corresponding non-stationarity. We believe that the extensive comparative experiments are sufficient to validate the reliability of our method. Thank you for your constructive and valuable comments.

• Reviewer $\color{blue}{\text{3526}}$: We have conducted experiments on more diverse datasets and model sizes to further validate the effectiveness of $\beta$-DPO. Considering your positive rating, we believe we have addressed your concerns. Thank you for your positive feedback.

• Reviewer $\color{green}{\text{UzDb}}$: We have clarified the implementation of the reward model and further explained why low-gap examples are considered high-quality. Additionally, we have attempted to use an explicit RM to strengthen the claim that the method is "reward-model independent". We believe our rebuttal has addressed your concerns, considering that you have further improved your ratings of our work.

• Reviewer $\color{orange}{\text{uGzu}}$: We have conducted experiments on more diverse datasets and model sizes, and visualized the range of values for $\beta_{\text{Batch}}$. Considering your positive rating, we believe we have addressed your concerns. Thank you for your positive feedback.

• Reviewer $\color{black}{\text{4RgK}}$: We have further analyzed the impact of different model sizes, automated parameter tuning, batch sizes, and learning rates. Thank you for your constructive and valuable comments.

We are pleased to note that Reviewers $\color{blue}{\text{3526}}$, $\color{green}{\text{UzDb}}$, and $\color{orange}{\text{uGzu}}$ have provided positive ratings and their concerns have been addressed. For Reviewers $\color{red}{\text{fXTA}}$ and $\color{black}{\text{4RgK}}$, we understand that you may have been too busy to participate in the discussion phase. However, we kindly request that you re-evaluate our paper in the subsequent stages.

As the deadline for the author-reviewer discussion approaches, we sincerely hope that our efforts and improvements will be taken into consideration. Once again, we thank the reviewers for their efforts in helping to improve the quality of our paper.

您好，

感谢您的来信以及对我们的工作提出的宝贵建议。关于参数化 $\beta(x)$ 的方法，虽然确实是一种可行的解决方案，但在实际调节过程中，此类函数以及正则项仍存在诸多不确定性和不稳定性。例如，$\beta(x)$ 取值过大或过小可能导致效果不佳，因此仍需依赖较多手动调整。本研究的主要目标是提出动态化的设计思想，并展示其良好的泛化性能。更多的改进空间和优化方向，我们期待未来研究能够进一步探索！

祝好，
Junkang

Dear Heyang,

Thank you for your email and the valuable suggestions regarding our work. While parameterizing $\beta(x)$ is indeed a feasible approach, tuning such functions along with regularization terms still involves significant uncertainties and instabilities in practice. For instance, an excessively large or small $\beta(x)$ may lead to suboptimal results, necessitating considerable manual intervention. The primary goal of this study is to propose the dynamic design concept and demonstrate its robust generalization performance. We believe further improvements and refinements remain open questions for future research!

Best regards,
Junkang