LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.

Q1: "For query generation failure cases (Sec 2.2.1), how frequently do smaller models output non-query text? "

A1: When using the smaller Qwen2.5-7B-Instruct model to generate data, the success rate is 77.2%. This means that 22.8% of the generated texts are non-query text.

Q2: "Could a classifier improve filtering beyond rule/length heuristics?"

A2: To compare the performance of a classifier and rule/length heuristics, we sample 1,000 outputs and use the Qwen2.5-72B-Instruct model as the classifier for comparison with rule/length heuristics. The prompt provided to Qwen2.5-72B-Instruct is as follows:

Classify the text below strictly as either QUERY or NON-QUERY.
QUERY: Represents a direct question or explicit search intent (e.g., "What is...", "How to...", "Find restaurants near...").
NON-QUERY: Represents statements, descriptions, continuations, or any non-interrogative content.

Ultimately, only 48 outputs show inconsistent classifications between the two methods, yielding an agreement rate of 95.2%. This suggests that the differences between the two approaches have minimal impact.

Q3: "Beyond QA, can LongMaggie support tasks requiring long-form outputs by modifying the generation protocol?"

A3: The core functionality of LongMagpie lies in its ability to generate diverse, document-adaptive queries. We categorize the generated queries into the following types:

These queries support a wide range of downstream tasks, particularly those requiring long-form responses. For example, Planning or Reasoning tasks involving multi-step analysis can be addressed by models such as DeepSeek-R1, while Creative Tasks are well-suited for models optimized for long-form content generation.

Q4: "Architectural Bias: Synthesized data inherits limitations of source LLMs (Qwen/Llama), risking error propagation."

A4: We plan to adopt the following strategies to detect and mitigate biases inherited from the source LLMs:

• Cross-model verification: Inspired by the multi-agent debate framework proposed by Google [1], we will generate QA pairs for each document using multiple LLMs, then conduct a debate among them. A separate judge model is used to evaluate the debated outputs and select the most appropriate responses. This process helps reduce the likelihood of propagating biases or errors from any single model.

• Quality filtering: We employ FsfairX-Llama3-RM-v0.1 to score and filter out low-quality generated data. As shown in Figure 2, this filtering step effectively improves the overall quality of the dataset.

• Safety filtering: In addition, we perform safety checks using Llama-Guard-3-8B. Our analysis shows that 99.86% of the data passes the safety criteria. Detailed safety statistics are provided in A7.

Q5: "Narrow Output Scope: Emphasis on document-query pairs overlooks tasks needing extended reasoning (e.g., multi-step analysis)."

A5: We categorize the query tasks in A3, and our dataset already includes instances that involve extended reasoning (Planning or Reasoning), such as multi-step analysis.

Q6: "Computational Cost: Heavy reliance on 70B models for high-quality data generation raises accessibility barriers."

A6: Considering the computational cost, we will release our data in the revised version to make it more accessible to the community. We hope this will further promote the development of long-context models in the field.

Q7: "Ethical Gaps: No discussion of misuse potential (e.g., generating deceptive long-context content) or fairness audits."

A7: We performed a safety analysis using the Llama-Guard-3-8B model to classify the generated content. The results are summarized below:

Thanks to the safety alignment of the Qwen model used in generation, the resulting dataset is overwhelmingly safe, with 99.86% of samples categorized as safe. This suggests that our pipeline can produce high-quality instructional data with minimal safety concerns.

[1] Du, Yilun, et al. Improving factuality and reasoning in language models through multiagent debate. ICML 2023.

We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.

Q1: "...While p-Mix clearly shifts the trade-off between long and short performance, it is not evident that it offers a meaningful balance overall."

A1: The p-Mix strategy offers a broader option that caters to scenarios where stronger performance on short-text inputs is required. As shown in Table 3, among all mix approaches, p-Mix achieves the best short-text performance while incurring the least degradation in long-context performance. For use cases that prioritize long-context understanding, we recommend directly employing our best-performing model trained solely on LongMagpie data.

Q2: "What model was used to generate the instructions for the (short-context) Magpie dataset?"

A2: The instructions for the Magpie dataset were also generated using Qwen2.5-72B-Instruct, which is the same model we used in our work.

Q3: "Table 4 highlights that the size of the dataset has a measurable effect on performance. However, the paper does not clearly specify how many data instances are present in the other evaluated datasets. How do their sizes compare to LongMagpie? A clearer and more detailed description of the experimental setup would improve the paper’s transparency and reproducibility."

A3: To ensure a sample-count-controlled comparison, we train on 190k samples across different methods. For ChatQA, we use its original 190k dataset; For LongAlign, we follow its original construction strategy to generate a 190k version. Results are shown below:

We further scale up the LongAlign dataset to 450k samples to compare its scalability. Results are as follows:

As shown, LongMagpie consistently outperforms LongAlign on average, especially as the data scale increases. We attribute this to its ability to generate more diverse and higher-quality questions (as shown in Figure 2) through adaptive query generation, rather than relying on fixed prompt templates or seed questions.

In addition, prior methods often depend on domain-specific long-context data or long-context-capable LLMs, which hinders their scalability. For example, ChatQA synthesizes data using NarrativeQA and needs to be combined with LongAlpaca12k and OpenOrca to reach 190k samples. LongAlign requires long documents and long-context models for data synthesis, and also needs to be mixed with short-text instruction data. In contrast, LongMagpie uses only general short-document datasets (~1.6k tokens average, as shown in Figure 3) and a simple, scalable method, enabling efficient synthesis at scale without external instruction data.

Q4: "In the data mixing experiments, why was the combination of LongMagpie + Magpie not explored? "

A4: We conduct experiments using the combination of LongMagpie and Magpie. The results were roughly comparable to the LongMagpie + Ultrachat setting. Specifically, we observed a slight improvement in long-context performance, while the performance on short-context tasks decreased slightly.

Q5: "...This requirement may restrict the applicability of the approach in low-resource languages or specialized domains where such document collections are scarce or difficult to obtain."

A5: For low-resource languages or specialized domains where document collections are scarce or hard to obtain, one potential solution is first to use large language models (LLMs) to synthesize domain-relevant documents, followed by employing LongMagpie to generate QA datasets. This approach can partially alleviate the issue. We believe that exploring the synthesis of documents remains an exciting and valuable direction for future research.

We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article.

Q1: "some of the improvements over strong baselines (LongMagpie + UltraChat vs. LongAlign + UltraChat) on benchmarks like RULER and HELMET can appear somewhat marginal (e.g., gains of 1 points)..."

A1: We would like to clarify that the best long-context performance is achieved using only LongMagpie data. The p-Mix strategy offers a broader option that caters to scenarios where stronger performance on short-text inputs is required. This design choice, to some extent, comes at the cost of slightly reduced performance on long-context benchmarks. However, for long-context tasks specifically, using only the LongMagpie-generated data yields the best results. Compared to other methods, we obtain the highest scores on HELMET (62.10), RULER (91.17), and LongBench V2 (34.4).

Q2: "...Could authors elaborate on the practical significance of these gains, perhaps highlighting scenarios or metrics where the advantage is more pronounced?"

A2: Following the partitioning method in [1], we provide metrics for different sub-tasks.

HELMET and RULER Results:

LongBench V2 Results:

These results highlight that LongMagpie consistently achieves either the best or second-best performance across the majority of subtasks, leading to its overall superior performance. In particular, it demonstrates significant improvements in Recall, ICL on the HELMET benchmark, as well as in QA, Code, and Long-Dialogue History Understanding tasks on LongBench V2.

Q3: "...Is it possible to do further scaling to observe more significant gains?"

A3: Due to our limited computational resources and time, we are unable to scale to larger data volumes within the current rebuttal period. However, as we discussed in A5, we compared our method with LongAlign and demonstrated that LongMagpie exhibits stronger scalability and more consistent performance improvements as the data scale increases.

Q4: "...This would clarify effectiveness at truly extreme versus moderately long contexts, and the effectivenes across tasks?"

A4: As outlined in A2, we provide a task-based categorization, and the reported metrics there are averaged over context lengths of 8k, 16k, 32k, 64k, and 128k. To further clarify effectiveness under truly extreme context lengths, we present the detailed results for the 128k context length:

These results show that LongMagpie continues to deliver leading or second-best performance even at extreme context lengths. We will provide a breakdown of results across different context lengths from 8k to 128k in the revised version.

Q5: "...could authors clarify this comparison or discuss if a sample-count-controlled comparison was considered?"

A5: To ensure a sample-count-controlled comparison, we train on 190k samples across different methods. For ChatQA, we use its original 190k dataset; For LongAlign, we follow its original construction strategy to generate a 190k version. Results are shown below:

We further scale up the LongAlign dataset to 450k samples to compare its scalability. Results are as follows:

As shown, LongMagpie consistently outperforms LongAlign on average, especially as the data scale increases. We attribute this to its ability to generate more diverse and higher-quality questions (as shown in Figure 2) through adaptive query generation, rather than relying on fixed prompt templates or seed questions.

In addition, prior methods often depend on domain-specific long-context data or long-context-capable LLMs, which hinders their scalability. For example, ChatQA synthesizes data using NarrativeQA and needs to be combined with LongAlpaca12k and OpenOrca to reach 190k samples. LongAlign requires long documents and long-context models for data synthesis, and also needs to be mixed with short-text instruction data. In contrast, LongMagpie uses only general short-document datasets (~1.6k tokens average, as shown in Figure 3) and a simple, scalable method, enabling efficient synthesis at scale without external instruction data.

Q6: "How do authors plan to detect and mitigate biases inherited from the source LLMs in the generated data? "

A6: We plan to adopt the following strategies to detect and mitigate biases inherited from the source LLMs:

• Cross-model verification: Inspired by the multi-agent debate framework proposed by Google [2], we will generate QA pairs for each document using multiple LLMs, then conduct a debate among them. A separate judge model is used to evaluate the debated outputs and select the most appropriate responses. This process helps reduce the likelihood of propagating biases or errors from any single model.

• Quality filtering: We employ FsfairX-Llama3-RM-v0.1 to score and filter out low-quality generated data (As shown in Figure 2).

• Safety filtering: In addition, we perform safety checks using Llama-Guard-3-8B. Our analysis shows that 99.86% of the data passes the safety criteria. Detailed safety statistics are provided in our response to Reviewer KGZd (as we discussed in A7).

Q7: "...Additionally, can the framework be extended beyond document-query relationships to cover other critical long-context tasks (e.g., summarization, complex structured extraction, multi-turn dialogue)?"

A7: We categorize the generated QA pairs into various task types, as shown below:

As shown above, our framework already generates a substantial number of instances beyond traditional document-query pairs, including tasks related to summarization and complex structured extraction.

In addition, the generated task types in LongMagpie often align closely with document content—e.g., code documents yield code-related queries, and structured texts lead to extraction tasks. We will expand to more diverse domains and formats to broaden task coverage, with concrete examples to be included in the revised version.

While we do not explicitly construct multi-turn datasets, our method improves performance on LongBench V2's Long-dialogue History Understanding task (see A2), indicating implicit benefits for multi-turn dialogue.

Q8: "Any safely analysis of the generated dataset?"

A8: Our analysis shows that 99.86% of the data passes the safety criteria. Detailed safety statistics are provided in our response to Reviewer KGZd (as we discussed in A7).

[1] Gao C, Wu X, Lin Z, et al. Nextlong: Toward effective long-context training without long documents. ICML 2025.

[2] Du, Yilun, et al. Improving factuality and reasoning in language models through multiagent debate. ICML 2023.

We sincerely thank you once again for your thoughtful and constructive feedback. We are glad to hear that your concerns have been addressed. All results and discussions from the rebuttal period will be incorporated into the final version of the paper.

Thanks for the further experiments and analysis. I believe the authors have addressed most of my concerns. I have decided to raise my rating and recommend that the authors include the new results and discussions in the final version of the paper.