In-Context Learning State Vector with Inner and Momentum Optimization

Dear Reviewer MvM2,

Thank you for your review. According to the feedback from you and other reviewers, we have conducted additional experiments and analysis. We have uploaded a Rebuttal PDF that contains new figures and tables. We provide the analysis in General Response (shown in a separate comment on this page).

We would like to address your concerns and questions in detail below.

---

Concern 1: Limitation in Model Size and Datasets

To address your concerns, we have conducted additional experiments using the larger Llama-2-70B model to evaluate the effectiveness of our proposed optimization method and the D&C aggregation. Detailed results and analysis can be found in the "Experiment on Llama-2-70B" in our General Response. Notably, the performance of ICL shows some improvement with the larger Llama2-70B model; our method continues to significantly enhance the state vector and surpass regular ICL in few-shot settings.

Regarding the dataset limitations, please refer to the response to "Question 2" where we present experiments on more complex alignment tasks.

We hope that these additional results satisfactorily address your concerns.

Question 1: Reason of Introducing Dummy Query

In our preliminary experiments, we observed a decline in performance in state vector extraction when we omitted the dummy query and simply added a separate token right after the demonstration. This finding aligns with earlier studies [1,2]. The issue likely arises because omitting the query alters the input format significantly from regular ICL. Therefore, we follow the previous settings by introducing a "dummy query" and extracting the state vector from the separator token that follows this dummy query.

Thank you for your insightful question. We appreciate your feedback and hope this clarifies our approach.

[1] In-context learning creates task vectors

[2] Function vectors in large language models

Question 2: Performance on Alignment Task

Thank you for your valuable question. We present the performance of the optimized state vector on the alignment task in a zero-shot setting. The results are shown in Table 8 in the Rebuttal PDF. Our findings indicate that although the state vector is slightly inferior to the regular ICL baseline, it still demonstrates significant potential as an effective alignment approach. By omitting the demonstration in the input, our method significantly reduces the time cost (e.g., by 5.4$\times$ on Llama-2-7B) while achieving 90% of the performance of regular ICL.

Moreover, compared to mainstream alignment-tuning methods such as instruction fine-tuning and reinforcement learning from human feedback, our method achieves an average of 85% of their performance without requiring any additional training. Notably, with models like Mistral-7B and Llama2-70B, our state vectors can achieve 80% of the alignment performance of GPT-4-0613. These results demonstrate that state vectors can enable efficient and effective model alignment, and that inner optimization is beneficial for complex alignment tasks.

We appreciate your feedback and hope this addresses your question comprehensively.

Question 3: Performance on Longer Model

Thank you for your insightful question. Switching to a long-context version of the model will result in a slightly different ICL state vector due to several factors:

1. Extended Token History: The state vector can access a more extended history of tokens, allowing it to retain more contextual information and improve performance on tasks with long-term dependencies.

2. Capturing Intricate Patterns: With a longer context, the state vector can capture more intricate patterns and relationships within the data, leading to more accurate and contextually appropriate responses.

3. Coherent Outputs: Long-context models generally produce more coherent outputs, as the state vector maintains a stable understanding over extended sequences, reducing the risk of losing context.

4. Enhanced Complexity Handling: The state vector in a long-context model can handle complex tasks more effectively, benefiting from the comprehensive integration of extensive contextual information.

We appreciate your feedback and hope this explanation clarifies the potential impacts of using a long-context model.

Discussion about information anchors

Thank you for your insightful comments. We are pleased to discuss the mechanism of information gathering in ICL.

Our research provides further evidence that separate tokens can progressively gather in-context information. In this paper, we investigate the impact of layer selection on state vector extraction in Section 6.2 Layer Selection. We observe that initially, increasing the number of layers for state vector extraction improves performance, and it reaches peak performance around the middle layers. This indicates that ICL information is progressively integrated into the separate token. In the work "Label Words are Anchors," it is found that information flows first from the example to its corresponding label words in the early layers, and then from the label words' representation to the final separate token representation in different layers. This also shows that ICL information progressively flows to the separate token and peaks around the middle layers. These results are consistent with our observations.

We hope this explanation addresses your concerns. If you have any further questions or need additional clarification, please let us know. We are more than happy to discuss this further.

[1] Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning.

---

Thank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion.

Dear Reviewer MvM2,

We hope you are doing well. As the discussion period is coming to an end (Aug 13), we wanted to reach out to see if you have any follow-up questions. If so, we would appreciate the opportunity to respond before the discussion period ends. We believe our above messages should have addressed your concerns, and therefore may warrant an increase in score if you find them helpful as well. Would you please let us know if you have any further questions or concerns? We are happy to continue the discussion.

Thank you very much again for your thoughtful review and help in improving the paper. We appreciate your time and consideration.

Best regards,

Paper15186 Authors

Dear Reviewer 4gb3,

Thank you for your review. According to the feedback from you and other reviewers, we have conducted additional experiments and analysis. We have uploaded a Rebuttal PDF that contains new figures and tables. We provide the analysis in General Response (shown in a separate comment on this page).

We would like to address your concerns and questions in detail below.

---

Concern 1: Method is too Simple

We respectfully disagree with the comments on the simplicity of our work. Before we dive into the concrete points, we would like to clarify our motivations and contributions:

• We have conducted a comprehensive study of the working mechanism of the compressed vector in ICL. We demonstrated that attention activations can be equivalent to updated parameters via gradient descent under specific conditions. Leveraging this understanding, we proposed the definition of the state vector. To the best of our knowledge, we are the first to analyze the compressed vector in ICL through the dual view of ICL and gradient descent.
• Inspired by the concepts of model soup and momentum gradient optimizer, we proposed the Inner Optimization and Momentum Optimization methods for the state vector. Extensive experiments on a range of tasks have shown that our methods significantly enhance the effectiveness and efficiency of the state vector.
• To address the example limitation caused by model input length constraints, we introduced the Divide-and-Conquer Aggregation method. Extensive experiments demonstrate that our approach effectively aggregates information from state vectors, successfully scaling up to a large number of examples.

Although our inner optimization can be viewed as a kind of uniform averaging, it is a simple yet effective method to enhance the state vector within a single forward pass. Additionally, our proposed momentum optimization further enhances the inner optimized state vector. Our work not only provides a novel theoretical perspective on ICL compressed vectors but also introduces practical applications for the study of ICL's working mechanism. In this way, our work, as the first to establish a bridge between the theoretical understanding of ICL's working mechanism and the application of ICL compressed vectors, offers a novel integration in the field.

We hope this explanation addresses your concerns. If you have any further questions, please feel free to let us know, we would be happy to discuss them with you.

Concern 2: Limitation in Theory Basis

We acknowledge that our theoretical development is still in the early stages. Due to the complexity of transformers, it is very challenging to rigorously prove the mechanisms of ICL is very challenging. However, our method demonstrates that under specific conditions, ICL can execute gradient descent algorithms. Previous works [1,2] have also built upon this understanding, conducting experiment to support it through attention pattern similarity.

While we cannot mathematically infer the generalizability and reliability of our method, we have extensively validated it across 12 datasets. The experiments demonstrate that our method is both generalized and robust. Additionally, we have provided results on more complex and realistic alignment tasks. Please refer to the"State Vector for Alignment" in our General Response for detailed experiments and analysis. Notably, our optimized state vectors can achieve 85% of the alignment performance of GPT-4 on Mistral-7B and Llama-2-70B.

We hope this explanation and the additional experiments address your concerns regarding the theoretical basis and practical applicability of our approach.

[1] Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers

[2] In-context Learning and Gradient Descent Revisited

Concern 3: Limitation in Model Size

Thank you for your insightful feedback and for pointing out the potential limitations concerning model sizes.

In response to your concerns, we have performed additional experiments using the larger Llama2-70B model to evaluate the effectiveness of our proposed optimization method and the D&C aggregation. Due to resource constraints, we were unable to extend our method to even larger models. Detailed results and analysis can be found in the "Experiment on Llama-2-70B" in our General Response. Importantly, although the performance of ICL exhibits some improvement with the larger Llama2-70B, our method continues to significantly enhance the state vector and surpass regular ICL in few-shot settings.

We hope that these additional results satisfactorily address your concerns.

---

Thank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.

Dear Reviewer 4gb3,

We hope you are doing well. As the discussion period is coming to an end (Aug 13), we wanted to reach out to see if you have any follow-up questions. If so, we would appreciate the opportunity to respond before the discussion period ends. We believe our above messages should have addressed your concerns, and therefore may warrant an increase in score if you find them helpful as well. Would you please let us know if you have any further questions or concerns? We are happy to continue the discussion.

Thank you very much again for your thoughtful review and help in improving the paper. We appreciate your time and consideration.

Best regards,

Paper15186 Authors

The Experiment result on Figure 10

We sincerely appreciate your insightful observations regarding the experiment in Figure 10. Thank you for pointing out the issue with the missing regular baseline. To address this and supplement our analysis, we have provided the original results of the Llama-2-70B aggregation experiment below. We will also revise our tables accordingly. Due to space constraints, we are only presenting the results for the 10 examples and 100 examples settings. We categorize the result into zero-shot and few-shot settings.

Zero-shot:

Few-shot:

As shown in the few-shot results, our D&C aggregation performs worse than the ICL baseline and average aggregation when the number of aggregated examples is small. The primary reason for this is that the conquer stage relies on a limited number of examples (e.g., only 1-shot in the conquer stage of D&C aggregation, compared to 10-shot for average aggregation when using a total of 10 examples). This limitation prevents the model's ability to effectively compress information from the group-specific state vectors into the final D&C aggregated state vector. However, as the number of examples increases, the performance of D&C aggregation improves significantly. At the 100-example setting, it outperforms the ICL baseline across four datasets (with improvements of 2.2 on Person-Occupation, 3.6 on Antonym, 1.7 on English-French, and 5.0 on Product-Company). Additionally, in the experiments shown in Figure 2, our method also achieves notable improvements (ranging from 1.4 to 11.3 on Llama-2-7B and from 1.3 to 10.1 on GPT-J). We achieved similar improvements on the larger Llama-2-70B model as we did on the smaller Llama-2-7B model, which demonstrates the effectiveness of our D&C aggregation method.

Regarding the average aggregation, we would like to emphasize that we included the average aggregation as a straightforward and intuitive baseline instead of the proposed method (it's not our main contributions). Consistent with your observations, we also found that the averaging aggregation does not yield significant results on Llama-2-70B. We attribute this to the fact that the simple average method, while providing some improvement, is insufficient for state vector aggregation. In contrast, our proposed D&C aggregation surpasses average aggregation under the 100-shot setting, which shows that our method is more effective and offers greater improvement over average aggregation.

---

Thank you very much again for your thoughtful review and feedback. Please let us know if you have any further questions, as we are happy to continue the discussion.

Dear Reviewer yzeU,

Thank you for your review. According to the feedback from other reviewers, we have conducted additional experiments and analysis. We have uploaded a Rebuttal PDF that contains new figures and tables. We provide the analysis in General Response (shown in a separate comment on this page).

We would like to address your concerns and questions in detail below.

---

Question 1: State Vector Usage in Zero-shot

We would like to clarify the input of our optimization method. When extracting the state vector, we use demonstrations and a dummy query as input. Then we extract attention activations from the forward pass to form the state vector. Thus, the process is the same for both the zero-shot and few-shot settings. Subsequently, we obtain the processed state vector through optimization or aggregation methods. The extraction and the following optimization or aggregation are performed only once since they are unrelated of the test data.

During testing, our experiments are divided into zero-shot and few-shot settings, which differ based on the input to the test data. In the zero-shot setting, we only input the query, whereas in the few-shot setting, we input both the demonstrations and the query. During the forward pass, we use the attention activations stored in the state vector to replace the ones computed in the transformer.

We hope this explanation addresses your question. Should you have any further inquiries, we are at your disposal for any additional discussions.

Question 2: Baseline Result In Figure 2

Thank you for your question.

In Figure 2, the ICL baseline denotes the 10-shot ICL result. Due to the input length limitations of the model, regular ICL cannot be directly used on multiple examples (e.g., the average input length of AG News in the 10-shot setting is 3872, while the maximum input length of Llama-2 is 4096). Therefore, we only use the 10-shot ICL as baseline, while we provide the averaging aggregation as a stronger baseline. We hope this explanation addresses your question.

Question 3: First-Order Gradient Optimizer Result

Here, we would like to provide a more detailed analysis. Firstly, we present the calculation formulas for all optimizers. In the following formulas, $g_t$ and $v_{t}$ denote the initial gradient and the optimized gradient at the $t$-th iteration, respectively. $\beta$ denotes the momentum coefficient.

1. Polyak Momentum Optimizer (mom.)

$v_{t}=\beta v_{t-1}+(1-\beta)g_t$

2. AdaGrad Optimizer (adag.)

$s_t=s_{t-1}+g_t \cdot g_t$

$v_{t}=\frac{1}{\sqrt{s_t+\epsilon}} \cdot g_t$

3. RMSprop Optimizer (rms.)

$s_t=\beta s_{t-1} + (1-\beta) g_t \cdot g_t$

$v_{t}=\frac{1}{\sqrt{s_t+\epsilon}} \cdot g_t$deacay

4. Adam Optimizer (adam.)

$s_t=\beta_1 s_{t-1} + (1-\beta_1) g_t \cdot g_t$

$h_{t}=\beta_2 h_{t-1}+(1-\beta_2) g_t$

$\hat{s}_t=\frac{s_t}{1-\beta_1^t}$

$\hat{h}_t=\frac{h_t}{1-\beta_2^t}$

$v_{t}=\frac{1}{\sqrt{\hat{s}_t+\epsilon}} \cdot \hat{h}_t$

We believe there are two main reasons why first-order gradient optimizers are not well-suited for state vector optimization:

• First-order gradient optimizers use adaptive learning rates, which require a significant amount of historical information. This strategy often shows instability and reduced effectiveness when data is limited.

• First-order gradient optimizers have more complex hyper-parameters than the Momentum Optimizer, making it challenging to find the optimal hyper-parameters. Experimental results indicate that directly using hyper-parameter settings commonly used in gradient descent results in poor performance.

We hope this explanation resolves your concerns. If you have any further questions, please feel free to let us know, and we would be happy to discuss them with you.

---

Thank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.

Dear Reviewer yzeU,

We hope you are doing well. As the discussion period is coming to an end (Aug 13), we wanted to reach out to see if you have any follow-up questions. If so, we would appreciate the opportunity to respond before the discussion period ends. We believe our above messages should have addressed your concerns, and therefore may warrant an increase in score if you find them helpful as well. Would you please let us know if you have any further questions or concerns? We are happy to continue the discussion.

Thank you very much again for your thoughtful review and help in improving the paper. We appreciate your time and consideration.

Best regards,

Paper15186 Authors

Dear Reviewer FZSw,

Thank you for your review. According to the feedback from other reviewers, we have conducted additional experiments and analysis. We have uploaded a Rebuttal PDF that contains new figures and tables. We provide the analysis in General Response (shown in a separate comment on this page).

We would like to address your concerns and questions in detail below.

---

Question 1: Difference with Task Vector

Thank you for your feedback. Since the task vector aims to extract the hidden state in the initial $L$layers for intervention, it is functionally equivalent to our state vector without optimization that extract attention activations in initial $L$layers. However, we must emphasize that the proposed state vector differs significantly in its integration into the model. In our theoretical framework, the state vector is defined based on attention activations of the separator token, which can be viewed as updated parameters. Therefore, even though it is mathematically equivalent to the task vector, the state vector possesses a stronger theoretical foundation and interpretability.

Question 2: Contribution Clarify

We would like to accurately describe our work and contributions. Previous work [1] proposed the dual view of ICL and gradient descent and provided evidence from the perspective of attention pattern similarity. Our work extends this understanding and applies it to the working mechanism of ICL compressed vectors. Based on our understanding and theoretical framework, we propose the definition of the state vector and introduce its optimization and aggregation methods. Our paper can be considered as primarily proposing a new compressed vector method, inspired by the similarity between ICL and gradient descent. Therefore, our work establishes a bridge between the study of ICL's working mechanism and the application of ICL compressed vectors.

[1] Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers

Question 3: Rationality of Optimization

Thank you for your question. In our perspective, we consider the state vector not simply as a compression of demonstration text information, but rather as a storage of the implicit ICL task function information contained within the demonstrations. Our optimization methods have enhanced the accuracy of this task information. The experiments in Section 6.1 Qualitative Study can intuitively demonstrate this point. As shown in Figure 4, a trend is observable in the movement of these clusters as the example position increases, indicating that as the number of demonstrations increases, the state vector becomes closer to the ideal task representation. Our momentum optimization effectively utilizes this trend and captures the underlying task dynamics within the demonstrations, thereby enhancing the state vector.

Question 4: Choice of Optimization Method

Thank you for your insightful question.

Based on the experimental results, we find that momentum optimization performs better on Knowledge tasks, the reason may be that more examples can effectively promote the model's recall of internally stored knowledge, while 10 examples may be insufficient. Therefore, our momentum optimization can enhance the accuracy of the knowledge-based task function stored in the state vector by extending its trend.

On the other hand, inner optimization shows better average performance on Linguistics and Translation tasks. This might be because, for these tasks, 10 examples are sufficient for the task, and momentum optimization could over-edit the state vector, thereby introducing noise.

In light of these findings, we suggest using momentum optimization when increasing the number of examples effectively enhances ICL performance. Conversely, when an increase in examples does not contribute much to performance improvement, using only inner optimization is recommended.

We hope this explanation addresses your question. Please let us know if you have any further questions; we would be happy to discuss them with you.

Question 5: Difference of Aggregation Result in 10-shot

Thank you for your question. We would like to elaborate on the differences between averaging aggregation and D&C aggregation in the 10-shot setting.

In the 10-shot setting, we have only one demonstration group. For averaging aggregation, we directly extract the group-specific state vector from this single demonstration group, which becomes the final aggregated state vector.

For D&C aggregation, the process is slightly different. In the divide stage, we extract the group-specific state vector from the group-specific dummy query. In the conquer stage, we first pair the group-specific dummy query with its label to form a one-shot demonstration. Then, we input a new dummy query with this one-shot demonstration and extract the aggregated state vector from the dummy query. Thus, the D&C aggregated state vector and the averaging aggregated state vector are not the same.

Our analysis suggests that the D&C aggregated state vector performs worse when the number of examples is small because the conquer stage relies on a limited number of examples. This limitation prevents the model from effectively compressing the information from the group-specific state vector into the final D&C aggregated state vector.

Typo in Table 5

We would like to clarify that the average result in Table 5 represents the average performance across 12 tasks, not just the 6 additional tasks. We appreciate your feedback and will revise our manuscript to ensure this is more clearly communicated.

---

Thank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion.

Dear Reviewer FZSw,

We hope you are doing well. As the discussion period is coming to an end (Aug 13), we wanted to reach out to see if you have any follow-up questions. If so, we would appreciate the opportunity to respond before the discussion period ends. We believe our above messages should have addressed your concerns, and therefore may warrant an increase in score if you find them helpful as well. Would you please let us know if you have any further questions or concerns? We are happy to continue the discussion.

Thank you very much again for your thoughtful review and help in improving the paper. We appreciate your time and consideration.

Best regards,

Paper15186 Authors

Dear Reviewer pJQh,

Thank you for your review. According to the feedback from you and other reviewers, we have conducted additional experiments and analysis. We have uploaded a Rebuttal PDF that contains new figures and tables. We provide the analysis in General Response (shown in a separate comment on this page).

We would like to address your concerns and questions in detail below.

---

Concern 1: Limitation of Model Size

Thank you for your valuable feedback and for highlighting the potential limitations related to the model sizes.

In response to your concern, we have conducted additional experiments using the larger Llama2-70B model to evaluate the effectiveness of the proposed optimization method and the D&C aggregation. Please refer to the "Experiment on Llama-2-70B" in our General Response for detailed experiments and analysis. Notably, while the performance of ICL improves with the larger Llama2-70B, our method still effectively enhances the state vector and outperforms regular ICL in a few-shot setting.

We appreciate your feedback and hope these additional results address your concerns.

Concern 2: Limitation of Datasets

Thank you for your feedback and for highlighting the potential limitations associated with the tasks.

To explore the effectiveness of state vectors on more complex tasks, we conducted experiments on the alignment task, which involves aligning LLMs with human preferences and is inherently more complex and diverse. Please refer to the "State Vector for Alignment" in our General Response for detailed experiments and analysis.

Notably, our state vectors can achieve 85% of the performance of mainstream alignment-tuning methods (i.e., instruction-finetuning and reinforcement learning from human feedback) without training. With Mistral-7B and Llama-2-70B, our state vectors can achieve 80% of the alignment performance of GPT-4-0613.

We hope these additional experiments provide a thorough understanding of our method's capabilities on complex tasks.

Question 1: Definition of Momentum Gradient Optimization

Thank you for your valuable feedback and for raising an important point regarding the strict definition of the momentum gradient optimization algorithm used in our study.

In this paper, we employed the Polyak Momentum Optimization algorithm. To ensure clarity and enhance the reader's understanding, we have provided the update rules for all optimization algorithms used in Table 2, corresponding to the $opt(*)$ function calculation method described in Equation 9 in the paper.

To maintain consistency with the notation used in the paper, in the following formulas, $E_i$ denotes the $i$-th ($1 \le i \leq N$) state vector (we ignore the hyper-parameter $L$ for simplicity), and $N$ is the number of state vectors. Additionally, let $v_{t}$ represent the update vector at $t$-th iteration, initialized as $v_0=\boldsymbol{0}$. Here, $\alpha$ denotes the learning rate, $\beta$ denotes the momentum coefficient, and $\epsilon$ is a very small constant for calculation stability. Below are the detailed update rules for each optimizer:

1. Polyak Momentum Optimizer (mom.)

$g_t = E_{t}-E_{t-1}$

$v_{t}=\beta v_{t-1}+(1-\beta)g_t$

2. AdaGrad Optimizer (adag.)

$g_t = E_{t}-E_{t-1}$

$s_t=s_{t-1}+g_t \cdot g_t$

$v_{t}=\frac{1}{\sqrt{s_t+\epsilon}} \cdot g_t$

3. RMSprop Optimizer (rms.)

$g_t = E_{t}-E_{t-1}$

$s_t=\beta s_{t-1} + (1-\beta) g_t \cdot g_t$

$v_{t}=\frac{1}{\sqrt{s_t+\epsilon}} \cdot g_t$

4. Adam Optimizer (adam.)

$g_t = E_{t}-E_{t-1}$

$s_t=\beta_1 s_{t-1} + (1-\beta_1) g_t \cdot g_t$

$h_{t}=\beta_2 h_{t-1}+(1-\beta_2) g_t$

$\hat{s}_t=\frac{s_t}{1-\beta_1^t}$

$\hat{h}_t=\frac{h_t}{1-\beta_2^t}$

$v_{t}=\frac{1}{\sqrt{\hat{s}_t+\epsilon}} \cdot \hat{h}_t$

After $N$ iterations, we use the final update vector $v_{N}$ to optimize the state vector:

$opt([E_i]^N_{i=1})=\alpha * v_{N}$

We hope this clarifies the strict definition and update rules for the momentum gradient optimization algorithm used in our study. Thank you once again for your insightful question.

Question 2: Comparison between D&C and Averaging Aggregation

In our aggregation experiment, we compared our D&C aggregation with the average aggregation method. The average aggregation method involves averaging the group-specific state vectors. As shown in Figures 2 and 6, although average aggregation benefits from an increasing number of examples, D&C aggregation outperforms average aggregation when the number of examples is the same. This is primarily due to the fact that D&C aggregation better captures the information in the group-specific state vector, leading to more robust and better performance.

Thank you for your question. We hope this comparison adequately addresses your question.

Question 3: Optimization in Code

In the code, FixOptimizer denotes the inner optimization. OneStepOptimizer denotes the direct momentum optimization on the original state vector, which we did not apply. FixOneStepOptimizer denotes the momentum optimization on the inner optimized state vector , and this represents the "momentum optimization" mentioned in the paper.

Suggestion of Presentation

We will revise Figure 4 to include a color legend for different positions to enhance its clarity. For the specific parameter used in the paper, we provided them in the "implementation details" in Appendix A. We appreciate your suggestions and hope these could address your concerns.

---

Thank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.