LOGO --- Long cOntext aliGnment via efficient preference Optimization

Dear Reviewer 2obo, we sincerely appreciate your thorough review of our work and the detailed feedback provided!

---

[Concern 1] A fairer comparison of YaRN

[Re] We have conducted the experiment and found that incorporating YaRN indeed leads to further performance improvements:

---

[Concern 2] Adding evaluations with full-context retention

[Re] We have conducted the experiments and provided the evaluation results on Ruler below.

---

[Concern 3] Lack of training process and evaluation details

[Re] Below, we provide more details:

• Fairness of Comparisons in Training Competitor Models We acknowledge the token count difference between Data-Engineering-80K (5B tokens) and LOGO. For evaluation, we used the open-source pretrained checkpoint of the Data-Engineering-80K model, but the token format varies from LOGO, complicating direct comparison. Nonetheless, LOGO outperforms Data-Engineering-80K despite using a smaller dataset (0.3B tokens), showing its efficiency.
• Hyperparameters and Negative Samples We used a fixed learning rate of 5e-5, with a cosine decay scheduler (100 steps warmup from 1e-8). The training setup included 8× A800 GPUs, global batch size of 64, microbatch size of 4 per GPU, and gradient accumulation of 2. The impact of negative prompt numbers is discussed in Section 5.1, with additional details in Appendix F.
• Reliability of Perplexity Perplexity may not fully reflect long-context performance. We placed the perplexity results in Appendix G, using the PG-19 dataset as a test set, not for validation during training.
• Inverse Trend of Perplexity and Performance We respectfully disagree with the concern on perplexity and performance trends. Our analysis shows that minor perplexity differences have little impact on model results. The primary goal was to show that LOGO maintains its language modeling ability, and performance should be assessed using benchmarks like LongBench, not just perplexity.

---

[Question] Lack of breakdown of evaluation details

[Re] To provide clarity, we first present the partial context length distribution of LongBench. We can find that LongBench has a tightly concentrated length distribution, making it difficult to observe clear performance improvements across different length ranges.

We encourage the reviewer to refer to our additional experiments on LongBench V2 (Reviewer icM3 Concern 2) and Ruler. These benchmarks feature a broader and more diverse context length distribution. The results demonstrate that LOGO consistently improves the backbone model’s performance across all context lengths, with particularly significant gains at longer context lengths (128K), further validating LOGO's effectiveness for long context.

---

We hope our responses have adequately addressed your concerns. If you have any further questions, please don’t hesitate to ask.

Dear Reviewer 6VnN, thanks for your insightful comments and suggestions. Below is our detailed response.

---

[Question 1] Reasoning Capability Evaluation

[Re] Thanks for raising this important point. However, our primary objective is not to enhance reasoning per se, but rather to mitigate misalignment issues in long-context responses.

Nevertheless, we have extended our experiments with LongBench V2 in this rebuttal, which is specifically designed for real-world long-context reasoning (Reviewer icM3’s Concern 2). Our results indicate significant performance improvements even in reasoning tasks, particularly in settings ''without chain-of-thought (CoT) prompting''. Notably, Llama3-8B-80K achieves an 8-point average improvement, and Qwen2.5-7B-Instruct yields a 2.6 average point improvement, suggesting that LOGO training may implicitly enhance reasoning capabilities—potentially due to the presence of multi-hop data in our synthesis preference data.

To achieve reasoning performance on par with OpenAI’s O1 or DeepSeek’s R1, stronger RL supervision signals, such as Long-CoT, would likely be required. Interestingly, we have recently been investigating this direction: by enabling LCM to actively search for and integrate key information from long-context inputs before generating responses, we observe substantial performance gains. However, this exploration falls beyond the scope of LOGO itself and is an avenue for future work.

---

[Question 2] Efficiency Gains vs. Traditional RLHF Approaches

[Re] A key challenge in applying traditional RLHF methods (e.g., PPO, DPO) to long-context modeling (LCM) is their heavy reliance on reward models, critical models, etc. For instance, PPO requires a reward model, a reference model, and a value model, while DPO depends on a reference model.

However, in the long-context domain, there are no publicly available reward models or value models to facilitate such training. To address this, LOGO introduces an alternative: importance scoring and a novel training objective to replace the need for an offline reward model. This significantly improves training efficiency and eliminates the dependency on additional models.

Besides, among open-source works, we have found only one study similar to ours: LongPO, which follows a traditional DPO-based approach and requires an additional reference model for training. To provide a clearer comparison, we present the following table highlighting the key differences between SFT, LOGO and LongPO:

We observed that without employing the Ring Attention strategy or a CP-parallel approach, LongPO cannot be deployed on one 80GB GPU due to the combination of long input sequences and the requirement for an additional reference model. Even with the Ring Attention strategy—where every 2 GPUs process a segment of the sequence in parallel, the training time for 2,000 steps exceeds 24 hours. Additionally, LongPO's performance is inferior to LOGO, as it only applies a simplistic preference data processing method, where shorter responses are treated as preferred replies.

In short, even in settings with ample computational resources, LOGO may still remain beneficial due to its scalability and reduced model dependency, making it a more practical and efficient alternative to traditional RLHF methods for long-context alignment.

---

We hope our responses have adequately addressed your concerns. If you have any further questions, please don’t hesitate to ask.

Dear Reviewer icM3, thanks for your insightful comments and suggestions. Below is our detailed response:

---

[Concern 1] Rigor of the main claim that increasing training data generally improves model effectiveness

[Re] We acknowledge that increasing high-quality training data can improve model performance, but training efficiency is a critical factor. Our work focuses on the efficient long-context alignment algorithm in LOGO rather than pure data scaling. Indiscriminate pure data scaling offers limited returns on long-context tasks. As shown in Table 1, SFT with 5B tokens (Data-Engineering-80K) improves LongBench performance by only +1.3 points, whereas LOGO achieves +2.5 points using just 0.3B tokens (16× less data). Additionally, Table 3 shows that LOGO improves performance (40.7 → 47.0), while SFT obtains a smaller gain (40.7 → 43.2) with a similar amount of data. Moreover, scaling data size in SFT requires careful balancing, as improper ratios (Figure 10) can hinder alignment. We also compare LOGO with other long-context DPO method, please refer to the second point of Reviewer 6VnN.

---

[Concern 2] Evaluation results on LongBench-V2

[Re] We have included the results in LongBench-v2 below.

We observed that Llama-3-8B-Instruct-80K-QLoRA-Merged performed significantly worse than expected, with results even lower than the 25% random guessing baseline, while LOGO can improve by an average of 8 points. We have carefully verified this outcome using the official code and double-checked our implementation to ensure its correctness. After applying LOGO training, both models exhibit substantial performance gains, further validating the effectiveness of our approach.
Additionally, based on the suggestions from Reviewer 2obo, we have also provided the evaluation results on **Ruler** in the second point in the rebuttal box for Reviewer 2obo.

---

[Concern 3] Innovative Contributions

[Re] Long-context capabilities have become indispensable, making effective long-context alignment more critical than ever. To our knowledge, LOGO is among the first open-source RL methods tailored for long-context alignment during the time of this paper submission, whereas prior works primarily achieve alignment with SFT. As demonstrated earlier, SFT alone yields limited benefits for long-context tasks, and while RLHF is a promising alternative, existing RL methods (e.g., DPO) are not well-suited for long-context alignment due to deployment challenges and inefficiencies.

While SimPO is effective for short-context generation, it lacks adaptations for long-context tasks. LOGO introduces: 1) Positional synthesis, enabling efficient handling of long sequences. 2) A novel preference data construction method, along with a novel preference strategy, to address the lack of long-context evaluation models.

---

[Question 1] Entity Extraction and Importance Scoring

[Re]

1. Entity Extraction: As noted in Section 4.1 (Lines 261–262), we use spaCy to extract entities (e.g., person names, locations) from questions and chunks.
2. Importance Metric: In long-context QA tasks, questions often target specific snippets within the long context. Overlapping entities between a chunk and a question indicate higher relevance (e.g., a chunk mentioning "Berlin" is likely critical for a question about "Germany’s capital"). We empirically found that selecting top-K chunks (e.g., K=16 for 512-token chunks) suffices to cover salient information, balancing efficiency and accuracy. Therefore, we use the number of overlapping entities as a key metric to measure importance, and the importance score reflects the relevance of each chunk and the question.

---

We hope our responses have adequately addressed your concerns. If you have any further questions, please don’t hesitate to ask.