LOGO --- Long cOntext aliGnment via efficient preference Optimization

Thank you for acknowledging the novelty and efficiency of our work. We appreciate your concerns and suggestions, and we will address them one by one below.

Weakness 1 & Question 3: Controlled Study of LOGO

You are correct in suggesting that a controlled study of LOGO is necessary to fully understand its effectiveness. While our original paper included such studies, the results were scattered across Table 1 and Figure 6 (ablation study). For your convenience in the rebuttal, we refer you to the detailed comparison of the Llama-3 model in the General Response, as well as the detailed comparison of Llama-2 in our response to Reviewer XuY2. Below, we excerpt the conclusions for a clearer presentation:

It is evident that LOGO significantly enhances performance, while the improvements from SFT and SimPO have a clear upper limit. Simply combining SFT + SimPO (or other forms of DPO methods) has its limitations. The main reason is the absence of a suitable evaluation model to assess the quality of samples during the construction of preference data (which we also discussed with Reviewer XuY2). Additionally, in the long-context model generation process, various types of misalignments are difficult to judge with an evaluation model.

By expanding the sampling space of negative samples, we can not only alleviate this issue but also allow the model to avoid more erroneous patterns. This is where LOGO differs significantly from DPO methods, as it addresses the challenge of constructing preference data in long-context scenarios where verification of results is extremely difficult.

Question 1: About Training Data

We understand your concerns and hope the following explanations can resolve your confusion:

1. Use of Model πθ for Data Construction: The primary goal of our paper is to leverage the model's inherent critical information retrieval capability[3] to enhance its generative abilities. Therefore, we construct our training data starting from the model itself. The purpose of simulating critical/non-critical segments within the long context, as mentioned in our paper, is to simulate responses based on critical information vs. responses without using critical information. This is a progressive enhancement strategy that allows LCM to avoid potential errors it might make. Using other models to construct data would mean teaching πθ to learn from other models' potential errors, which deviates from our original intention. Therefore, we rely entirely on πθ to construct training data.

2. Non-use of long-llm-data's Golden Data: This is a critical point. The long-llm-data, originating from work [4], constructs golden data by providing GPT-4 with the long context. However, how can we ensure that GPT-4's generated data is correct (e.g., free from hallucinations)? While this work is promising, upon manual checking of some cases, we found significant issues. Below are our manual check results for 50 cases from each subset:

   Source	Context has corresponding entities	Answer is correct
   S-DocQA	33/50	29/50
   M-DocQA	30/50	29/50
   Book	38/50	29/50

We can discover that even GPT-4 makes numerous errors and exhibits significant hallucination, producing content not present in the original context. In contrast, we need to construct data using the LOGO approach and we only provide critical/non-critical segments from the context to ensure the data quality. That's why we use πθ to regenerate golden data.

Thank you for acknowledging the contribution of our work in addressing a gap in current Long Context Model (LCM) training methods. Below are our responses:

Question 1: Lack of Rigorous Evaluation Methods for Detecting Misaligned Outputs and Hallucination

You have raised a very astute observation regarding the evaluation of misaligned outputs and hallucinations in our dataset. We appreciate your insight and believe some misunderstandings need to be clarified by starting a discussion about ''the challenges we faced in constructing our dataset and evaluating these misaligned outputs''.

As you rightly pointed out, we have written in our paper (Lines 188-189) that '' there is a lack of effective strategies (or models) to detect these misaligned outputs''. While it is theoretically feasible to evaluate certain types of misalignment, such as using manual evaluation or leveraging GPT-4 to assess the quality of synthetic data, the variety of misalignment types greatly complicates the evaluation process. Therefore, we aimed to minimize our dependence on evaluation models when constructing our data.

LOGO addresses this issue from three aspects:

1. Data Construction: During data construction, we utilized non-critical segments within the context to guide the model to generate completely incorrect answers (dis-preference sample) and critical segments to guide the model to generate correct answers (preference sample). Here, the accuracy of preference samples is significantly higher than that of dis-preference samples since the responses are generated from the critical segments. Additionally, since we employed the RL method for training, our goal was to keep the model away from incorrect samples, rather than fitting to correct samples. Therefore, ensuring that the preference samples are acceptable and of higher quality than the dis-preference samples was sufficient, and we clearly achieved this.

2. Objective Function of LOGO: We expanded the space of negative samples in LOGO's objective function, allowing the model to avoid not just one type of error. This means that as long as the negative samples are generally incorrect, the training is effective.

3. Utilizing the intrinsic knowledge of the model to serve our objectives.: LOGO's goal is to leverage the long-context model's inherent critical information retrieval capabilities[1] (as mentioned in the Introduction section) and use the retrieval key information to generate a response. Therefore, regardless of whether there is an evaluation function, constructing preference data based on critical/non-critical segments within the context is reasonable because we aim to stimulate stronger generation capability based upon the inherent retrieval capability. It is worth noting that you may notice that stronger models (e.g., Llama-3) benefit more from LOGO training. This is because, during data construction, stronger LCMs can create higher-quality preference data (because those models have stronger capabilities), thereby further enhancing the effectiveness of training with synthetic data.

From a deployability and scalability perspective, LOGO's data construction method and training objective function are not only straightforward but also highly effective.

We acknowledge that the evaluation model is a crucial component in the synthesis of long-text data. However, any research is progressive, and LOGO has already demonstrated the significant role of negative(dis-preference) samples in preference alignment within the long-context field. We believe this is not just an issue for us alone, but rather a challenge that the entire long-context community should address collectively.

Question 2: Comparison with more baselines

We appreciate your suggestions and recognize the value of the references you provided, which primarily focus on training longer and better LCMs. It seems that in the early stages of LCM development, there was not a clear distinction between Context Window Scaling and Long-context Alignment, which is why we did not compare LOGO with these works directly due to the different settings.

You also mentioned a pipeline approach, which we understand is similar to our setup: first, using a method (e.g., LongRoPE) to extend the model's context window size, and then using another strategy (e.g., SimPO) for long-context alignment. In fact, the experiments in the "Results on LCMs" group of Table 1 in our paper employ the same method. Therefore, the comparison that needs to be added is between LOGO and SimPO on a long-context model.

Question 2: Comparison with more baselines

Below are the model results on the LongBench testing set. Considering that the baselines you mentioned are mainly conducted on the Llama-2 model, we chose Llama2 as the backbone model for additional experiments in this rebuttal. Specifically, we utilize Data Engineering [2] to scale the context window size. During the evaluation stage, we truncate the evaluation data length to the model's maximum context window size. Note that to ensure a fair comparison, we have listed the type and purpose of the dataset (Symbols: Context Window Scaling -> CWS, and Long-context Alignment -> LA, *denotes training from a long-context model [2]).

We can observe that under different settings, LOGO consistently performs the best. We have also studied the related work you provided, among which LongAlign[4] and Refined-Alpaca-1k[5] both directly construct high-quality long-context data, achieving both context window scaling and alignment.

Compared to these studies, LOGO differs significantly in terms of training data scale and experimental setups (one starting from short-context models, the other from long-context models). We believe that this comparison may not be entirely fair. However, we have included this type of work in the above table for your reference, and you will find that LOGO still performs better.

For more experimental results on Llama-3 and other models, you can refer to the results shown in the General Response and Table 1 in our paper.

Question 3: More Theoretical Analysis such as Error Bounds for LOGO and Convergence Properties

Thank you for your inquiry regarding the theoretical underpinnings of LOGO, while I don't come from a traditional Machine Learning background, I have endeavored to provide as detailed a theoretical analysis as possible in the revised version of the manuscript based on my understanding, specifically in Appendix F and G of the revised manuscript.

Since the numerous mathematical formulas make it impractical to present them fully in OpenReview, we put our conclusion here:

From a theoretical standpoint, the error bound of LOGO depends on two key factors:

• the volume of training data should sufficient (we have 0.3B tokens during the training stage);
• the model probabilities πθ(y|x) not assign an unduly low probability. This aligns with our data construction process, which is entirely model-dependent (i.e., the training data is generated by the model itself), ensuring that the predicted probabilities are not set too low.

Reference:

[1] Wu, Wenhao, et al. "Retrieval head mechanistically explains long-context factuality." arXiv preprint arXiv:2404.15574 (2024).

[2] Fu, Yao, et al. "Data engineering for scaling language models to 128k context." arXiv preprint arXiv:2402.10171 (2024).

[3] https://huggingface.co/yaofu/llama-2-7b-80k

[4] Bai, Yushi, et al. "Longalign: A recipe for long context alignment of large language models." arXiv preprint arXiv:2401.18058 (2024).

[5] Zhao, Hao, et al. "Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning." arXiv preprint arXiv:2402.04833 (2024).

We hope our responses and perspectives have addressed your concerns and misunderstandings. If you have any additional concerns or questions, please don't hesitate to let us know. We believe LOGO represents a significant advancement in the long-context alignment field due to its effectiveness and efficiency. We hope you can reconsider your evaluation of our work and an improvement in your score would be greatly appreciated, thank you.

#### Context ####
[... context ...]
AEM outpoerforms both LEM and DPEMM by 6.5 and 1.7 respectively in F-measure on the FSD dataset, and 4.4 and 3.7 in F-measure on the Twitter dataset. 
[... context ...]
We can also observe that apart from K-means, all the approaches perform worse on the Twitter dataset compared to FSD, possibly due to the limited size of the Twitter dataset. 
[... context ...]
#### Question ####
What baseline approaches does this approach out-bperform?

#### Answer ####
| Model            | Answer                                                |
|------------------|-------------------------------------------------------|
| Ground Truth     | K-means, LEM, DPEMM.                                  |
| LOGO (Ours)      | AEM outperforms both LEM and DPEMM.                   |
| LongAlign        | The proposed approach outperforms the baseline        |
|                  | approaches on all three datasets.                     |
| PoSE-YaRN-96k    | LEM and DPEMM.                                        |

In these two cases where the context contains interfering information, we find that the answers generated by LOGO are consistent with the Ground Truth, indicating that LOGO can accurately retrieve relevant information in the context and utilize it for response. However, the other two methods (LongAlign and PoSE) are affected by irrelevant information within the context, leading to outputs that contain hallucinated or distorted information.

Reference

[1] Bai, Yushi, et al. "Longalign: A recipe for long context alignment of large language models." arXiv preprint arXiv:2401.18058 (2024).

[2] Gao, Chaochen, et al. "Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model." arXiv preprint arXiv:2405.19846 (2024).

[3] Wu, Wenhao, et al. "Retrieval head mechanistically explains long-context factuality." arXiv preprint arXiv:2404.15574 (2024).

[4] Zhu, Dawei, et al. "PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training." The Twelfth International Conference on Learning Representations.

[5] Wu, Wenhao, et al. "Long context alignment with short instructions and synthesized positions." arXiv preprint arXiv:2405.03939 (2024).

We hope that our responses and perspectives can address your concerns and misunderstandings. If you have more concerns or questions, we would also appreciate hearing from you. We believe that LOGO represents a significant advancement in the field of long-context alignment due to its effectiveness and efficiency. Additionally, we hope you can reassess our work and an improvement in your score would be greatly appreciated, thank you.

Question 2: Theoretical Guarantees for Position Synthesis Method

We understand that the question of the theoretical guarantees for the position synthesis method is equivalent to understanding why Relative Position Encodings (RPE) are effective. For the effectiveness and theoretical proof of RPE, you can refer to the paper "Self-Attention with Relative Position Representations" available at https://arxiv.org/pdf/1803.02155. Other works such as [4][5] also use position synthesis to improve training efficiency.

Simply put, for the Transformer model, the model learns a function $f(i, j)$ to understand the relative position information between any two tokens. The extrapolation ability of position encoding can be understood as the model's desire to learn information about larger $(i-j)$ values. Referring to the mathematical formula of RoPE, the attention formula can be written as: $Att_{i,j} = \frac{(QK^{T} + f(i, j))V}{d^{1/2}}$ where $QK^{T}$ and $V$ are the query, key, and value, respectively. To perform length extrapolation, one only needs to consider how to make $(i-j)$ larger. The conventional approach is to increase the sequence length to enlarge the value of $(i-j)$, with each token corresponding to an absolute $i$ and $j$ value. As for position synthesis, it only needs to consider changing the values of $i$ and $j$ without altering the actual sequence length.

A potential issue is that many position indices may be missing in the positional synthesis method, and we illustrate how we compensate for the missing relative positions in Appendix D in our paper.

Question 3: How LOGO Works When Increasing Context Lengths Beyond 32K

Indeed, as sequence lengths increase, the primary consideration is how to use position encoding synthesis to cover more positions, which requires more short-context data to fill in the gaps. However, if the sequence length truly exceeds 32K, strategies like Ring Attention are necessary, utilizing multiple GPUs to share the training memory load. Below is the evaluation results when increasing context length to 32K:

Specifically, based upon the original 8K context, we extended the context up to 32K by filling the context with irrelevant text (sampling from pre-trained corpus), allowing the model to further learn how to utilize key information from the context to generate responses. This approach not only tests the model's ability to handle longer contexts but also its capability to focus on relevant information amidst potentially distracting content. We observe that LOGO continues to demonstrate potential for further performance improvements (LOGO-8K improves from 42.3 to 47.0, while LOGO-32K improves from 42.3 to 48.6), which can be attributed to both more comprehensive positional encoding coverage and the benefits gained from increased computational resources (such as Ring attention).

Question 4: Detailed Analysis of Failure Cases

This is a valuable suggestion. Of course, we provide two failure cases here, and we have also supplemented the corresponding results in Appendix H of the revised paper (Figure 15 ~ Figure 19).

Below, we present two error cases:

#### Context ####
Thaddeus P. Mott ... 
[... context ...]
At the time of his death, he was also the last surviving son of the eminent surgeon Valentine Mott...Upon his death in 1865, Mott was interred at Green-Wood Cemetery in Brooklyn, New York.
[... context ...]
Beaulieu-sur-Loire (French pronunciation: literally Beaulieu on Loire) is a commune in the Loiret department in north-central France.
Two days later, Anthony Roberts was on the scene with a detachment of Philadelphia police.
[... context ...]

#### Question ####
Where was the place of death of Thaddeus P. Mott’s father?

#### Answer ####
| Model            | Answer                |
|------------------|-----------------------|
| Ground Truth     | New York.             |
| LOGO (Ours)      | New York.             |
| LongAlign        | Beaulieu-sur-Loire.   |
| PoSE-YaRN-96k    | Anthony Roberts.      |

Thank you for your feedback and for acknowledging the computational efficiency of our method. Regarding the concerns and issues you raised, we would like to address and clarify these points in detail below.

Weakness 1 & 2: Paper's Main Contribution Appears Incremental Rather Than Transformative & Fail to Address Fundamental Challenges of Long-Context Understanding

We appreciate your feedback. However, we contend that the core argument of our paper centers on utilizing the inherent critical information retrieval capabilities of long-context models to improve their alignment effectiveness. We identify three primary challenges for long-context models: 1) the expansion of the context window, which is a key focus of current research; 2) the capability to retrieve essential information; and 3) the ability to process and generate responses based on this retrieved key information. Much of the existing research on aligning long-context models [1][2] is dedicated to the creation of higher-quality long-context data for fine-tuning (SFT) the model. Our perspective, however, highlights that models already possess robust capabilities for retrieving key information within a long context, a point indirectly confirmed by work [3]. This paper primarily explores how to leverage this retrieval ability to address misalignment issues in LCMs, such as hallucinations, which are vital for comprehension tasks.

The methodologies we have adopted are not arbitrary combinations but rather carefully selected approaches designed to harness the model's inherent capabilities. For example, SimPO, when compared to traditional SFT, not only prevents misalignment but also demonstrates particular effectiveness in generation tasks. Similarly, PoSE (validated by studies [4][5]) presents a training strategy that is especially accessible to academic researchers who often face resource constraints, such as limited GPU availability.

We firmly believe that the primary contribution of this work extends beyond merely combining existing methods. Rather, it lies in identifying an elegant yet powerful approach to activate the model's latent capabilities. Indeed, we would argue that the innovative application of simple and effective existing methods to address challenges in the long-context domain represents a significant contribution to the field. After all, isn't the clever utilization of established methods to solve complex problems a valuable advancement in itself?

Weakness 3: Experimental Limitations:

We acknowledge your concerns and would like to address them by providing additional experimental results to demonstrate the effectiveness of our approach:

• We would respectfully direct your attention to the Llama-3 results presented in the General Response, as well as the specific findings related to the Llama-2 model detailed in our response to Reviewer XuY2. These comprehensive results offer valuable insights into our method's performance across various settings and in comparison with different baselines.

• We recognize that direct comparisons may not be completely equivalent due to variations in datasets and training resources (as different baselines serve distinct purposes, which we elaborated on in our General Response - various methods enhance different aspects of LCMs, while our primary focus is on improving long-context alignment capabilities). Nevertheless, these additional comparisons provide substantial evidence supporting the effectiveness of our methodology.

Question 1: How Does LOGO Fundamentally Differ from DPO?

In our paper, we explicitly mention in lines 192-193 that ''we design the loss function based on SimPO''. Additionally, lines 149-150 clarify that ''SimPO is a variant of DPO''. Therefore, both LOGO and DPO aim to address the lack of dis-preference data during the SFT process. From this perspective, most current work essentially aligns with the core principles of DPO.

However, the key distinction between LOGO and DPO (which was previously primarily suited for short-context tasks) lies in the complexity of constructing preference data and their application scenarios, particularly as evaluating model outputs in long-context scenarios presents significant challenges. To address these challenges, we have expanded the space for dis-preference samples in the LOGO objective function, rather than directly applying SimPO's loss function. Furthermore, we have incorporated an SFT regularization term to maintain the model's language modeling capabilities.

Those modifications represent the fundamental difference from DPO.

Thank you for your recognition of our work. We are pleased that you highly regard the contributions and soundness of our work. Below are our responses and supplementary experimental results addressing your concerns.

Question 1: Analysis of GPU memory, training, and accuracy between preference optimization (LOGO) and traditional SFT

We have conducted a detailed analysis and comparison of different training strategies, as follows:

Note: The longer training time for SFT + Ring Attention is attributed to our PCIE network infrastructure.

• The GPU performance analysis related to LOGO's hyperparameters is presented in Figure 6(c) in the paper.
• Performance comparisons can be found in Table 1, Figure 4, Figure 5, and Figure 7, as well as the additional experimental results in General Response.
• Ring attention configuration involves context parallel communication between every two GPUs.
• All experiments were conducted with the same number of training steps (2000).

Question 2: Conducting experiments with longer context lengths, such as 128K and 256K

You are correct in stating that 128K is now considered the threshold for long-context models. For context lengths of 128K, we can directly use YaRN's method to expand from 80K to 128K, and the model performance impact is not significant. For even longer context lengths, we need to extend the position synthesis method to a greater range and use more data to compensate for the context length gaps. We used 16K actual input (we still build data upon 8K critical/non-critical segments, and then fill the context with irrelevant text upon reaching the 16K context length) and simulated a 256K context length using position encoding synthesis, starting training from the Llama-3-8B-Instruct-LOGO-80K model. Here are the results on the LongBench testing set:

It can be observed that LOGO performs well under the 128K and 256K settings, with the 256K training scenario significantly outperforming the Yi-200K model. This improvement may be due to the introduction of noise in our training dataset (filling the length to 16K with irrelevant context sampled from pre-trained corpus), which further enhances the model's ability to locate information and utilize key information for responses. This also demonstrates the scalability of the LOGO method, which can be extended to even longer lengths.

3. Discussion: Scalability of LOGO to models trained on diverse, multi-modal data

Thank you for your thought-provoking question about the potential for scaling LOGO to models trained on diverse, multi-modal data, such as long video VLMs. It's an exciting area of research, and coincidentally, I have recently been working on video generation. Here, I can provide an example from video generation to illustrate.

Controllability in Long Video Generation: As you may have noticed, there are some significant controllability issues in long video generation, akin to the hallucination problem in text generation. Models may start following a prompt accurately but eventually deviate, introducing concepts and entities not present in the original prompt.

Keyframes and Controllability: In the video domain, keyframes serve as pivotal points that define the structure and content of a video sequence. These keyframes can be thought of as the video equivalents of critical information within the context as mentioned in our paper. The concept of keyframes offers a promising avenue for enhancing controllability in video generation.

Harnessing Keyframes for Control: By constructing keyframes that act as anchors for the video content, we can potentially guide the model to generate content that remains aligned with the initial prompt. This approach is analogous to how LOGO uses key information within a text context to generate responses. By creating hallucination keyframes, we can train the model to recognize and avoid deviating into unrelated content, similar to how LOGO steers the model away from non-preferred outputs.

Training Approach Inspired by LOGO: Inspired by LOGO, we could train the model to generate videos that are more controllable and less prone to hallucination. This could involve:

1. Preference Optimization for Videos: Extending the preference optimization approach to video generation, where the model learns to differentiate between preferred (on-topic, coherent) and non-preferred (off-topic, incoherent) video content based on keyframes.

2. Relative Position Representations for Video Context: Adapting the relative position representations used in LOGO to understand the temporal relationships between keyframes and other video segments, helping the model to maintain coherence over longer sequences.

We appreciate your interest in the potential applications of LOGO beyond text, and we believe that the future community will not only focus on whether we can create long-video (or other modalities) models but also on making that generation more controllable.

We hope our responses have addressed your concerns. If you have more questions, please feel free to ask. We believe that LOGO will benefit the field and community of long-context alignment because it is a simple, scalable, and high-performing method. If possible, we kindly ask for a reassessment of our work, and an improvement in your score would be greatly appreciated.

3. Manuscript Revision (Important)

We have submitted a revised version of the manuscript to address reviewers' concerns. All changes made in the manuscript are highlighted in blue font, and the rest of the manuscript remains unchanged.

The modification includes:

1. Reorganize the results and add one baseline in Table 1 for a clearer demonstration of our experimental settings;
2. Appendix F: Error Bound Analysis;
3. Appendix E: Convergence Property from a Gradient Analysis Perspective;
4. Appendix H: Error Analysis

We kindly ask all reviewers to give special consideration to these revisions.

Reference

[1] Wu, Wenhao, et al. "Retrieval head mechanistically explains long-context factuality." arXiv preprint arXiv:2404.15574 (2024).

[2] Bai, Yushi, et al. "Longalign: A recipe for long context alignment of large language models." arXiv preprint arXiv:2401.18058 (2024).

[3] Zhu, Dawei, et al. "PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training." The Twelfth International Conference on Learning Representations.

[4] Meng, Yu, Mengzhou Xia, and Danqi Chen. "Simpo: Simple preference optimization with a reference-free reward." arXiv preprint arXiv:2405.14734 (2024).