Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model

Q1: The paragraph lines 93-105 attempts to make some links with search engines, but the motivation could benefit from being further clarified and strengthened.

A1: Thank you for your valuable suggestion. We will further clarify and strengthen our motivation in the latest version. Our primary goal is to collect enormous queries and cluster articles related to each query, mimicking the functionality of search engines. However, collecting many queries is very time-consuming and cannot guarantee diversity. Thus, we employ the doc2query model to generate queries for each document in the corpus. This way, we solve the problem of cheaply acquiring many diverse queries.

Q2: Why isn't the Rake score used here to decide which keyword to retain? Or even simply retain the keyword with the largest Rake score?

A2: In fact, the RAKE score is used to filter out meaningful keyword sets based on our statistical analysis, indicating that keywords with scores greater than 3.0 are meaningful ( As we mentioned in line 213). Then, to further ensure the diversity of documents related to each keyword and mitigate any bias introduced by RAKE, we randomly select from the remaining keywords after filtering.

The rationale behind randomly selecting RAKE keywords is to maintain context diversity. For example, consider two similar documents, document A and document B, which may share keywords like "sofa" and "table." We aim to prevent these documents from being consistently associated with the keyword that holds the higher RAKE score. Instead, we randomly assign each document to one of the keywords, assigning document A to "sofa" and document B to "table." This random assignment ensures a greater diversity of documents linked to each keyword.

Q3: Was there any evaluation conducted to validate the quality of the keywords identified for the documents?

A3: We first scored the extracted keywords using human annotation and GPT-4o on 100 samples, ranging from 1 to 5. The Pearson correlation coefficient between human annotations and GPT-4o scores was calculated as 0.7986, indicating a strong correlation between the two methods.

Subsequently, We employed GPT-4 to score 1000 samples. The results are presented in the following table:

The evaluation results demonstrate the quality of the Quest keywords' strong overall performance.

Q4: How is the oversampling ... Adding the mathematical formula for this would be helpful for the reader's understanding.

A4: Essentially, oversampling redistributes the sampling probability to prioritize $I_s$, ensuring it gets a larger share of the total samples. We provide some explanations and mathematical formulas below:

Assuming $I_s$ contains $n_s$ training samples, $I_l$ contains $n_l$ training samples, $p$ is the oversampling probability, and the total number of training samples to be used is $N$. The number of samples drawn from $I_s$ can be calculated as:

$I_s = \lceil \left( \frac{n_s}{n_s + n_l} + p \right) \cdot N \rceil$

The number of samples drawn from $I_l$ can be calculated as:

$I_l = N - I_s$

Q5: What is the motivation for splitting into only two sets and instead of having a more fine-grained approach?

A5: As we mentioned in lines 222-224: ... the number of documents associated with different keywords varies significantly...we implement oversampling for documents with less frequent keywords." We also see the potential for Quest to combine with adaptive and more fine-grained approaches, and we will explore them in the future.

Q6: Figure 3 contains several typos: "query genaration" -> "query generation", "random smaple" -> "random sample"

A6: We apologize for the typos; we corrected them in the latest vision. Thank you for pointing it out.

Q7: In the results on Longbench in Table 1... More detailed explanations would be appreciated.

A7: Please refer to the General Author Response. We conducted a more comprehensive evaluation of QUEST using the latest long-context evaluation benchmark, HELMET. The performance across a broader range of subtasks statistically demonstrates that QUEST is effective.

Q8: It is surprising that Quest achieves 100% accuracy ... Are there any intuitive explanations for this phenomenon?

A8: We mentioned it in lines 371-376: Figure 1 is "retrieving a text sentence," and Figure 4 is "retrieving a numeric string." In addition, we use italics to highlight the difference in Figure 4's caption: "Unlike Figure 1 ... numeric string."

We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.

Q1: While I could understand the motivation... still seems fairly artificial and unjustified in comparison to using any summarization technique.

A1: We utilize the recently introduced Llama-3.2-1B model to generate summaries for 1,000 randomly sampled articles to analyze the differences between summarization and query generation. The results are as follows:

• Computational Cost

On average, each summary contains 147 characters, compared to 49 characters per query. The length gap indicates that generating summarizations requires roughly three times the computational resources needed to generate queries, negatively impacting this approach's scalability.

• Performance

Following A3, we use GPT-4o to evaluate the keywords extracted from queries and summarizations. The results are presented below:

The results show that summaries' keywords are of less quality than those derived from queries.

These results demonstrate that generating summaries incurs higher computational costs, resulting in lower-quality keyword extraction. We attribute this to the broader semantic space of summaries compared to queries, which interfere with effective keyword extraction.

Q2: ... It might have still been interesting to check the results when the keyword with the highest RAKE score is used ...

A2: Through entropy analysis, We evaluate the diversity of two strategies using the highest RAKE score and random sampling (scores greater than 3). The results below show that random sampling led to greater diversity.

We further construct a training dataset based on the highest RAKE score, and the results are presented in the table below:

These results demonstrate that random sampling enhances diversity and improves model performance.

Q7: ... although my question was more related to the positive impact of the Levenshtein distance metric on Quest.

A7: We apologize for misunderstanding your question.

We observe that Levenshtein distance provides a more precise reflection of changes in long-context modeling capabilities compared to metrics like ROUGE. As a fine-grained metric, Levenshtein distance measures character-level differences more continuously, allowing for smoother tracking of improvements in handling long texts.

Similar observations were mentioned in the recent HELMET long text evaluation paper and are supported by previous studies [1][2][3] that highlighted the noise issues of metrics such as ROUGE. HELMET [4] advocates for more precise metrics, such as exact matching and model-based evaluation, to assess long-context modeling performance.

Combining our paper's experimental results with the HELMET results, it becomes clear that Quest not only achieves strong performance on the Levenshtein distance metric but also shows improvement on other evaluation metrics. This demonstrates that Quest effectively enhances long-context modeling capabilities.

[1]Goyal T, Li J J, Durrett G. News summarization and evaluation in the era of gpt-3.

[2]Deutsch D, Dror R, Roth D. Re-examining system-level correlations of automatic summarization evaluation metrics.

[3]Chang Y, Lo K, Goyal T, et al. Booookscore: A systematic exploration of book-length summarization in the era of llms.

[4]Yen H, Gao T, Hou M, et al. Helmet: How to evaluate long-context language models effectively and thoroughly.

Q3: ...Please make sure to provide the details of this study (prompt for GPT-4o, information and instructions to the human participants, etc.) in the final version of the paper.

A3: We hired two PhD students to perform the annotation, along with GPT-4o, all following the scoring criteria:

Scoring Criteria

• Score 1: The keyword has little to no relevance to the article and may have been extracted incorrectly or is significantly off-topic.
• Score 2: The keyword has some relevance to the article but is of low importance or overly generalized. It is not a core theme or focus of the article.
• Score 3: The keyword has a clear connection to the article but is not one of its primary concepts or focal points.
• Score 4: The keyword is strongly related to the article and represents an important concept or secondary theme.
• Score 5: The keyword perfectly reflects the article's central theme, main concepts, or key discussion points.

Score distribution

Q1: The author used a query generation model to generate different queries for different documents, which would make the quality of query generation affect the final performance of the model. The author should discuss in more detail the impact of query quality or different query generation methods on model performance.

A1: This is a very good suggestion. We attempted to generate queries using LLMs, which intuitively could produce better queries. However, the enormous computational cost exceeded what we could afford. To explore the quality of query generation within a manageable range, we randomly replaced keywords in the queries with a certain probability to simulate the creation of low-quality queries. The experimental results are as follows:

The results show that the model's performance gradually declines as the replacement ratio increases, indicating that improving the quality of the query may further enhance performance.

Q2: The model approach is more like an integration of existing tools, without any contribution to methodological or theoretical innovation. For example, the query generation and selection process directly employ the off-the-shelf tools (doc2query, RAKE) without significant enhancement.

A2: Since the pre-training corpus is very large, to ensure the scalability of the Quest method, we mainly selected some lightweight components that had been verified to be reliable. The experimental results also confirm that our choice ensures both effectiveness and scalability. Although we have also explored some resource-intensive methods, such as using LLM to generate better queries, these methods are not practical at present because of their large resource consumption and inability to scale.

Q1: The paper does not motivate why these steps are necessary.

A1: As we mentioned in line 095: "Our inspiration stems from the observation that similar queries can aggregate semantic relevant but low-redundancy documents via search engines ...we predict potential queries using a generative model for each document in the training dataset." Our method, therefore, consists of three main steps: a. Query generation; b. keyword extraction; and c. index building:

a. Query generation

Our primary goal was to collect enormous queries and cluster articles related to each query, mimicking the functionality of search engines. However, collecting many queries is very time-consuming and cannot guarantee diversity. Thus, we employed the doc2query model to generate queries for each document in the corpus. This way, we solve the problem of cheaply acquiring many diverse queries.

b. Keyword extraction

However, we noticed that the generated queries were overly fine-grained, resulting in retrieved documents with insufficient diversity. To address this, we adopted a coarser-grained keyword-based approach for indexing, which not only improved diversity but also reduced redundancy.

c. Index building

By building an inverted index, we implemented an efficient aggregation pipeline, also the basis for scalable long-data synthesis.

Q2: One could imagine that much simpler methods would be able to lead to synthetic data that are to some extent similar ... Could this simpler method lead to similar results?

A2: This is a valuable hypothesis, and we designed three strategies to achieve the goal of "selecting documents to be concatenated that are similar to some extent."

• Strategy 1: Documents with mid-ranking similarity were selected and concatenated.
• Strategy 2: Documents were concatenated by random sampling.
• Strategy 3: Documents were concatenated in reverse order of similarity.

As shown in the table, for the KNN method, reducing the similarity of the selected documents results in a slight performance improvement, followed by a decline. Furthermore, these methods consistently perform worse than the Quest method.

Additionally, while the "mid-ranking" approach exhibits a similarity extent close to the Quest method, its performance remains far inferior. It indicates that document aggregation has a greater impact on overall performance than the extent of similarity, highlighting it as the key factor driving Quest's superior results.

Q3: The paper tends to over-claim the advantage of the proposed method. Globally, if one consider all the datasets, the demonstration that the proposed method is better is very weak.

A3: Please refer to the General Author Response. We comprehensively evaluated QUEST using the latest long-context evaluation benchmark, HELMET. The results, covering up to 17 subtasks, statistically confirm QUEST's high effectiveness.

Q4: Table 3 shows that Pythia obtains the best performance on 4 datasets, while Quest only on 2.

A4: Table 3 demonstrates that training on long texts synthesized by Quest does not adversely affect the performance of short text tasks, aligning with the objectives of previous studies [1][2].

While the performance of Quest on the four datasets is slightly lower than that of Pythia without long text training, the difference falls within the margin of evaluation fluctuations. Moreover, the average performance of Quest across multiple datasets is comparable to that of Pythia without long-text training, further supporting the conclusion that training on long texts synthesized by Quest does not compromise the performance of short-text tasks.

[1] Dubey A, Jauhri A, Pandey A, et al. The llama 3 herd of models.

[2] Fu Y, Panda R, Niu X, et al. Data engineering for scaling language models to 128k context.

Q5: Some of the figures are unclear.

A5: Thanks for pointing this out. We will improve and refine our figures in future versions.

Q6: Fig. 5 compares the results using the documents ... Do these data have equivalent sizes?

A6: Yes, we strictly controlled the amount of data in both the synthetic data and the long documents to be identical. We will provide more detailed explanations in the latest versions.

I hope my responses answer your questions and increase your confidence in our paper. Thank you!

Dear Reviewer ki5C,

We sincerely appreciate the insightful feedback you provided, which has been instrumental in enhancing the quality of our work. We have prepared detailed responses for your review.

Please let us know if we need to provide any further clarifications or feedback.

Best regards,

The Authors

Q1: Although the proposed method is simple, it largely follows previous approaches but with a different indexing and sampling method.

A1: Since the pre-training corpus is very large, to ensure the scalability of the Quest method, we mainly selected some lightweight components that had been verified to be reliable. The experimental results also confirm that our choice ensures both effectiveness and scalability. Although we have also explored some resource-intensive methods, such as using LLM to generate better queries, these methods are not practical at present because of their large resource consumption and inability to scale.

Q2: In addition, no ablation is provided so we are not sure which part of the QUEST is the most helpful (query-based indexing or sampling etc.).

A2:

a. query-based indexing

We employed GPT-4 to evaluate the quality of keywords with and without query-based indexing across 1000 samples, using a scoring system ranging from 1 to 5. The results are presented in the following tables:

The results indicate that query-based indexing significantly improves the quality of the extracted keywords. This is evident from the higher average score (3.574 vs. 2.634) and the increased proportion of keywords scoring 3 or above (74.3% vs. 49.3%). Furthermore, the percentage of keywords scoring 4 or 5 is notably higher with query-based indexing (63% vs. 31.2%).

b.sampling

In Appendix A.4, we examine the impact of the split ratio, which controls the proportion of oversampled keyword-based inverted indexes. As stated in line 790, "It suggests that moderate oversampling of indexes with fewer documents benefits long-context modelling, highlighting the importance of balanced data distribution for long-context tasks."

Q3: ... THis raises questions on the real performance of the proposed method.

A3: Please refer to the General Author Response. We conducted a more comprehensive evaluation of QUEST using the latest long-context evaluation benchmark, HELMET. The performance across a broader range of subtasks (as many as 17) statistically demonstrates that QUEST is sufficiently effective.

Q4: ...For instance, the examples in 5.2 show that most baseline methods also have good clustering quality.

A4: The validity of QUEST derives from the fact that Quest balances context diversity and semantic correlation. We demonstrate that the methods overemphasizing semantic correlation (good clustering quality) hurt the performance. As we mentioned in lines 88-92: "The results show that either prioritizing context diversity at the expense of semantic correlation (Standard) or overemphasizing semantic correlation while sacrificing context diversity (KNN and ICLM) leads to suboptimal performance... highlighting the need for a method to balance both aspects."

We also provide analysis in lines 474-478: "Figure 5 (left) shows that the Standard method forms the most dispersed clusters due to random aggregation, while KNN and ICLM yield tighter clusters in the 32k-context-length setting ... Figure 5 (right) illustrates document distribution for various methods under the 128k context length. Quest continues to aggregate relevant, lowredundancy documents."

Q5: In fig3, Llama 3 8B 128k has accuracy of 0.97, so why even longer context of 1M as in fig 1 would result in 1 accuracy?

A5: Fig 1 and 3 are two different Needle-in-a-Haystack settings. We mentioned it in lines 371-176: Figure 1 is "retrieving a text sentence," and Figure 4 is "retrieving a numeric string." In addition, we use italics to highlight the difference in Figure 4's caption: "Unlike Figure 1, task difficulty is increased within the 128k context length by retrieving a sentence rather than a random numeric string." They are both widely used, so we tested two datasets.

Q6: Why the performance in appendix from table 8 to table 10 are in consistent, I find it hard to inteprete these results and how do they differ from the tasks that are present in the main text?

A6: The Longbench benchmark aggregates many subtasks into different task categories. For example, the Single-Doc QA task category includes three subtasks: narrativeqa, qasper, and multifieldqa_en. Due to space limitations, we report the average performance of the aggregated task categories in the main experiment. In the appendix, we show the performance of each subtask.

Dear Reviewer 9HBG,

We sincerely appreciate your thoughtful feedback, which has played a crucial role in improving the quality of our work. We have carefully prepared detailed responses for your review.

Please don’t hesitate to let us know if further clarification is needed.

Best regards,

The Authors