What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices

Question 1: Discussion about the novelty of this article.

Answer 1: Thanks for your thoughtful advice. We would like to make it clear to you that our innovation is not focused on engineering optimization. In fact, we have brought a lot of insights that have not been discovered before. For example, the large model is not a good long context annotator but a better long context selector. In addition, we also analyze the shortcomings of quality classification on long contexts and also analyze the corresponding long context biases.

Moreover, we first pay attention to the importance of multi-hop long context synthesis data for long context training, and provided a set of general long text data synthesis frameworks and strategies available to the academic community to increase the long text performance of the model.

Question 2: Automated scoring and classification of LLMs may introduce bias, especially in complex, nuanced situations where human judgment may differ, thereby affecting the interpretability and reliability of the data.

Answer 2: Thank you for your detailed feedback. In fact, this is the flaw of all LLM-based automated processes. To this end, we have chosen models and strategies that are as consistent with humans as possible, which precision score reaches 96.43%.

Question 3: This paper points out that it significantly increases token consumption, raising concerns about its efficiency and scalability in resource-constrained environments.

Answer 3: Thanks for your constructive comments. The core consumption of tokens for long text data synthesis is still long context input, and the output of CoT actually only accounts for 1/10 of the total tokens. We don’t think this will bring too much additional resource consumption.

Question 4: The study focuses primarily on the MIMG framework without conducting a thorough comparison with other data synthesis or augmentation frameworks.

Answer 4: Thanks for your valuable suggestion. In fact, our focus is on the long text field, and the current data in this field are based on Self-Instruct. There is not even a systematic framework. In fact, some existing long text work is basically a subset of our MIMG. For example, the prompt of Self-Instruction is almost the same as that of Single-hop Question Generation Agent. Self-Instruction + LLM recheck is also consistent with the structure in which we add Quality Verification Agent. Hence, we think we have a thorough comparison to a certain extent.

Question 5: An analysis of failure cases or limitations in specific scenarios needed to be considered to provide a more balanced view of the usefulness of the framework.

Answer 5: Thank you for your constructive comments. We found that a smaller part of the results will lead to over-reasoning, and it may be necessary to combine other single-hop reasoning situations for more practical expansion in the future. We will conduct more case analysis and discussion in the next version.

Question 6: The performance gains in this paper appear to be comparable to those achieved by simply using a more powerful LLM. This inconsistency raises questions about the practical benefits of the proposed approach, especially given its added complexity.

Answer 6: Thanks for your thoughtful suggestions. We want to clarify that the conclusion linking superior models to enhanced performance is primarily supported by the application of our MIMG framework, rather than by the mere comparison between baseline models and MIMG. Furthermore, as shown in Figure 12, the quality of the data synthesized using Qwen2 within the MIMG framework significantly exceeds that generated by GPT-4 using the Self-Instruct strategy. This illustrates the effectiveness of our framework, even more effective than backbone replacement. Furthermore, as illustrated in Table 1, our approach consistently outperforms prior efforts utilizing GPT-3.5, GPT-4, and even human-created data, which demonstrates and further proves the effectiveness of our framework. In addition, GPT4 is often impractical for training data synthesis. It is also important to note that the large-scale training data, distilled from only 4 billion tokens using GPT-4, incurs a cost exceeding $10,000, rendering such an approach impractical for many applications.

Question 7: Suggestions for improving the readability of related work.

Answer 7: Thank you for your detailed reply. I will adjust the content of the related work part in detail in the next version.

Dear Reviewer LHkC,

As the discussion deadline is extended, we are actively looking forward to your further feedback. Thanks for your effort and understanding!

Kindest regards,

Authors of ICLR Submission 8639

Question 1: Some minor annotations and grammatical errors in this paper.

Answer 1: Thank you for your detailed suggestions, we will correct the corresponding minor errors point by point.

Question 2: Ablation studies between agents are lacking, so it is unclear which agent contributes the most.

Answer 2: Thank you for your insightful suggestions. To address your concerns, we analyze the contributions of various components of the agent on human-annotated data quality. As summarized in Table 1, each component positively contributes to the overall quality. Notably, the Multi-hop Question Merger Agent significantly enhances the multi-hop quality of the dataset, while the Multiple Question Sampling mechanism effectively increases the diversity of the generated data. Furthermore, the Single-hop Question Generation Agent plays a crucial role in simultaneously improving both the quality and diversity of the data. Additionally, the Quality Verification Agent serves as a critical safeguard, maintaining a lower bound to ensure model quality and thereby further enhancing the data's overall integrity.

Table 1: Results of the ablation study on MIMG, reporting the data quality score as the average of three independent manual evaluations.

Question 3: These experiments were only evaluated on InternLM, while widely used models such as the LLaMA series is necessary to be evaluated.

Answer 3: Thank you for your insightful advice. We want to claim that our work does not only conduct experiments on InternLM. All our analytical experiments are validated on multiple LLMs, including Qwen2, InternLM2, Gemini-1.5, GPT-4o-mini and GPT-4o. In addition, in terms of model training, as shown in Table 1, we apply the experiments on both InternLM2 and LLAMA-3.

Question 4: It is unclear whether the authors aimed to present a SOTA dataset or a framework for generating the data.

Answer 4: Thanks for your constructive response. In fact, both these two points are our contribution. The logic of our paper is to explore a solution with a higher quality-cost ratio, and then scale a better dataset based on this solution to optimize the model. Therefore, we discuss the optimal solution for each part and gradually optimize it as the best solution for the next step. So, at each step we discussed the quality of data generated by different models.

Furthermore, our research primarily focuses on long-context tasks, an area where the synthesis of multi-hop data has not been systematically explored for long-text applications. To the best of our knowledge, we are the first to introduce this consideration in such tasks. Furthermore, as outlined in Table 1 of the paper, the datasets we compared with were predominantly synthesized using advanced models, including GPT-3.5-Turbo, GPT-4, and even human annotations, which much surpass the capabilities of our data synthesis backbone. Despite this, our synthesized datasets demonstrate superior performance compared to these benchmarks.

We will add more discussions in the next version.

Question 5: The evaluation criteria for data need to be further clarified, especially indicators such as diversity.

Answer 5: Thank you for your constructive comments. I greatly agree with your point of view. Diversity refers to whether an annotator can find data with the same semantic meaning in previously annotated samples as a proportion of all data. We will discuss each indicator in detail in the next version.

Question 6: Details of LongMIT data, including number of samples and number of tokens.

Answer 6: Thanks for your valuable feedback. To address the problem, we conduct a detailed statistical analysis of sample size and token consumption across various datasets. As shown in Table 2, while our token count is comparable to that of NQ, our dataset contains significantly fewer samples and outperforms NQ by more than 10% in terms of performance. Interestingly, although NQ includes a much larger number of samples than most other datasets, its performance is inferior even to datasets with smaller sample sizes, such as LongAlpaca and LongAlign.

Table 2: The volume statistics results of different datasets.

Question 1: Analyzing these will help demonstrate the independent contribution and practical necessity of each module.

Answer 1: Thanks for your thoughtful suggestions. To address your concerns, we analyzed and evaluated individual components of the agent architecture on human-annotated data quality. As illustrated in Table 1, each component contributes positively to overall data quality. Specifically, the Multi-hop Question Merger Agent significantly enhances the multi-hop quality of the dataset, while the Multiple Question Sampling mechanism improves the diversity of generated data. Moreover, the Single-hop Question Generation Agent plays a pivotal role in balancing both the quality and diversity of the data. Finally, the Quality Verification Agent serves as a foundational mechanism, ensuring a lower bound for quality and further enhancing the dataset’s overall standard.

Table 1: Results of the ablation study on MIMG, reporting the data quality score as the average of three independent manual evaluations.

Question 2: Figure 12 shows that the cost token of LongMIT-GPT4o is more than four times that of Self-Instruct GPT4o, which may affect the feasibility of practical applications.

Answer 2: Thank you for your constructive comments. I think there is a misunderstanding here. First of all, the annotation of LongMIT-GPT4o in our figure is not accurate, which means that MIMG are equipped with all strategies that consume tokens but have higher data quality. Additionally, as shown in Figure 12, we would like to clarify that assuming that an equal amount of high-quality data is to be generated, the average token consumption of our method is about 5.3k, while the token consumption of using GPT-4o+Self-intstruct is close to 6.6k.

Question 3: Can you give a more detailed description and implementation method for indicators such as retention rate and average score to help readers better understand?

Answer 3: Thank you for your constructive comments. I greatly agree with your point of view. Our retention rate refers to the proportion of data that will be retained after being filtered by an LLM through a certain threshold. This is based on our observation that LLMs are better selectors rather than better annotators. LLMs have higher retention rates and are more consistent with humans.

Additionally, the average score refers to the average score of the model. Statistically speaking, the higher the score, the higher the quality.

Thank you for your response, but I still have the following questions:

1. Regarding Question 1, I still have some concerns. For the three metrics - high-quality, diversity, and multi-hop, there doesn't seem to be clear evidence in the paper demonstrating their impact on instruction-tuning performance. Could the authors provide evidence of how each module in MIMG affects final instruction-tuning performance, or provide direct evidence (including the magnitude) of how these three metrics impact instruction-tuning performance?
2. Regarding Question 2, I still don't understand your explanation. Could you provide accurate annotations on the figure regarding LongMIT-GPT4o and Self-Instruct-GPT4o?
3. Regarding Question 3, Thank you for your explanation, so does your retention rate refer to the data filtered by the Quality Verification Agent? Could you provide specific details such as the threshold value and the criteria for determining the threshold? Also, could you provide specific details about how the model determines the "average score"?