DataMan: Data Manager for Pre-training Large Language Models

Questions3. What is the value of k in top-k?

Line 285: "We perform top-k sampling without replacement based on source and domain distribution for each quality criterion."

Here, top-k refers to the size of the pre-training data sample, which is typically 30B tokens in this paper.

Questions4. How do the 14 quality criteria correspond to the 3 principles?

Lines 201- 203: "How to define the quality criteria and domains of texts? We believe that excellent quality criteria should: 1)-be applicable to a wide variety of texts, 2)-demonstrate a deep understanding of content, capturing semantic levels, and 3)-complement each other. "

1. First, we want to clarify that these three principles are not to generate quality criteria; instead, the quality criteria should satisfy these three principles.
2. Following this, we illustrate how the 14 quality criteria satisfy three principles:
   1. All quality criteria are satisfied by principle 1;
   2. Coherence, Semantic Density, Knowledge Novelty, Topic Focus, Creativity, Professionalism, Originality, Sensitivity satisfy principle 2;
   3. Accuracy, Language Consistency, Style Consistency, Grammatical Diversity, Structural Standardization, these non-semantic level criteria complement with semantic level criteria that meet principle 2.
   4. As stated in the response to Reviewer8xFT's Weakness1, all quality criteria complement each other to form a comprehensive data quality system across four levels.

[1]Accuracy: the fewer grammar, referential, and spelling errors the text contains, and the more accurate its expression. _/5
[2]Coherence: the more fluent the content is expressed, and the stronger its logical coherence. _/5
[3]Language Consistency: the more consistent the use of language in the text, with less mixing of languages. _/5
[4]Semantic Density: the greater the proportion of valid information in the text, with less irrelevant or redundant information. _/5
[5]Knowledge Novelty: the more novel and cutting-edge the knowledge provided by the text, with more insightful views on the industry or topic. _/5
[6]Topic Focus: the more the text content focuses on the topic, with less deviation from the main theme. _/5
[7]Creativity: the more creative elements are shown in the text’s expression. _/5
[8]Professionalism: the more professional terminology appears in the text, with more accurate use of terms and more professional domain-specific expression. _/5
[9]Style Consistency: the more consistent the style of the text, with proper and appropriate style transitions. _/5
[10]Grammatical Diversity: the more varied and correct the grammatical structures used in the text, showing a richer language expression ability. _/5
[11]Structural Standardization: the clearer the structure followed by the text and the more standardized its format. _/5
[12]Originality: the fewer repetitions and similar content in the text. _/5
[13]Sensitivity: the more appropriately sensitive topics are handled in the text, with less inappropriate content. /5
[14]Overall Score: the better the comprehensive evaluation of the text, with superior performance in all aspects./5

4. Step 3: Identify Domain Categories.

   We selected 15 current domain categories based on factors such as the application industries, the number of existing industry LLMs, and the level of attention they have received (e.g., GitHub stars) [1,2].

[1] https://github.com/HqWu-HITCS/Awesome-Chinese-LLM

[2] Naveed, Humza, et al. "A comprehensive overview of large language models." arXiv preprint arXiv:2307.06435 (2023).

Questions1. Why do the handpicked results in Table 5 support "Pointwise ratings are better than Pairwise ratings"?

Table 5: We follow the Tabel 4 in Qurating by using 10 documents from different sources, ranked them by writing style, and use them to analyze pointwise and pairwise ratings

• • Initial Quality Criteria

    [1]Semantic Fluency/Coherence/Logic: Evaluate whether the text is smooth and easy to read, whether the content is coherent, and whether the logic is clear.
    [2]Content Consistency/Variability in Language Style: Evaluate if the information within the text is contradictory and if the language style is diverse.
    [3]Topic Diversity: Determine the richness and variety of topics addressed in the text.
    [4]Content Regularity/Formatting: Consider whether the text adheres to a certain structure or format.
    [5]Content Redundancy: Analyze the extent of information repetition within the text.
    [6]Proportion of Domain-Specific Vocabulary: Measure the frequency of professional terms or specific vocabulary used in the text (such as proper nouns, technical terms, or Classical Chinese).
    [7]Proportion of Sensitive Topics: Examine the percentage of content that addresses sensitive topics (e.g., involving politics, toxicity).
    [8]Proportion of Creative Expression: Assess the degree of creative or innovative expression in the text (e.g., use of rhetorical techniques).
    [9]Degree of Language Mixing: Analyze the extent to which different languages are used within the text (i.e., the ratio of text in various languages).
    [10]Complexity of Text Structure: Evaluate the overall complexity of the text's structure.
    [11]Proportion of Long Sentences: Assess the ratio of long sentences within the text.
    [12]Proportion of Grammatical, Reference, and Spelling Errors: Evaluate the ratio of grammatical errors (e.g., incorrect punctuation, unclear sentence breaks), reference errors, and spelling mistakes in the text.
    [13]Proportion of Content Lacking Semantics: Determine the ratio of parts within the text that lack meaningful content (e.g., garbled text, HTML tags, XML elements, navigation bars, incomplete chart numbers, or disjointed citations).
    

3. Step 2: Enhance Quality Criteria.

   a. We utilize GPT-4-turbo to rank the text quality criteria by importance, eliminating, merging, and supplementing them.

   [1] Text accuracy (grammar, references, spelling)
   [2] Semantic coherence and consistency
   [3] Language consistency
   [4] Effective semantic content ratio
   [5] Knowledge novelty
   [6] Topic focus
   [7] Creative expression ratio
   [8] Proportion of technical terms
   [9] Style variability
   [10] Complexity of grammatical structures
   [11] Content regularity
   [12] Content redundancy
   [13] Proportion of sensitive topics
   

   b. Thereafter, based on the principle that higher scores indicate better criteria, we revised the prompt. Finally, we also utilized GPT-4-turbo to help us simplify, resulting in 14 quality criteria derived from the original paper.

We sincerely appreciate your constructive comments, which are incredibly helpful in strengthening our paper! We hope that our response effectively addresses your concerns.

Weakness1. The presentation needs significant improvement.

All typos have been corrected and highlighted in the revised PDF.

Weakness2. The 14 quality criteria seem also intuitive and manually designed by the authors.

1. The 14 quality criteria are derived from the LLM's identification of the causes of abnormal perplexity (PPL) in documents, not through intuitive or manual design.

   Similar to previous self-improvement work [1,2], our motivation is to enable LLM to self-select quality criteria that can enhance its performance, as its pre-training capability is related to perplexity [3,4,5]. High perplexity indicates that the data is difficult to learn, and vice versa. Consequently, this "reverse thinking" approach encourages us to let the LLM self-identify the causes of anomalous perplexity to derive quality criteria, rather than relying solely on intuitive or manual design.

2. The "iterative refinement" described in the paper refers to experts organizing and arranging all the quality criteria identified by the LLM, assisted by GPT.

3. Reviewers 8xFT concur that our approach makes sense, and Reviewers Tq6k agree that it avoids pitfalls of criteria based on human intuition.

Questions2. How to select these 13 quality criteria and all the domain categories?

We will provide a detailed explanation of the process for obtaining all quality criteria and domain categories, which is conducted by experts with the assistance of GPT-4. The original text can be found in Appendix A.

1. Motivation: By employing a "reverse thinking" approach, we aim to encourage the LLM to self-identify quality criteria that contribute to its performance improvement. As noted in Weekness2, the pre-training capabilities of LLMs are closely related to perplexity[1,2,3]. Therefore, we prompt the LLM to identify the reasons behind document perplexity anomalies, which subsequently leads to the derivation of the current 13 quality criteria.

2. Step 1: Initialization of Quality Criteria. We devised an analytical prompt for GPT-4-turbo to investigate the reasons behind the textual perplexity anomalies (in the top and bottom 2%) from each source and extract initial text quality metrics. This yields the quality criteria for the first step. Table 1 provides several examples illustrating how these initial quality criteria are derived.

   • Analytical prompt:

   Read the following document, which has a {high/low} perplexity of {ppl_value} for LLM inference. Please analyze the reasons for the PPL anomaly.
   

   Table 1. Analyzing the reasons for the abnormal perplexity in documents using the analytical prompt in GPT-4-turbo, and extracting the initial quality criteria from the LLM's response.

I appreciate the authors' efforts during the rebuttal phase, especially the detailed experiments and thorough explanations. Most of my concerns have been addressed, and I have updated my score accordingly. However, due to the current presentation of the paper and the absence of an updated PDF version, I am inclined to maintain my score at 5 (rather than increasing it to 6). I have also adjusted my confidence score to allow the AC and other reviewers to take a more prominent role in the decision-making process.

Table 3. Detailed results for the perplexity of the SlimPajama validation sets, bold indicates the best performance.

Table 4. Detailed results for the perplexity of the SlimPajama test sets, bold indicates the best performance.

4. We further explain why we did not conduct Dolma experiments:

   i. FineWeb is a more recent release than Dolma.

   ii. Dolma conducts specific processing rules for each data source (Web, Code, Social, C4, Semantic Scholar, Books, Wikipedia) but lacks a comprehensive approach for overall data.

   iii. Some rules described in Dolma’s paper, such as C4 NoPunc, are not implemented in Dolma's code.

[1] Sang Michael Xie, et al. "Data selection for language models via importance resampling". Neurips 2023.

[2] Wenzek, Guillaume, et al. "CCNet: Extracting high quality monolingual datasets from web crawl data." LREC 2020.

[3] Marion, Max, et al. "When less is more: Investigating data pruning for pretraining llms at scale." Neurips 2023.

[4] Alexander Wettig, et al. "Qurating: Selecting high-quality data for training language models". ICML 2024.

[5] Penedo, Guilherme, et al. "The fineweb datasets: Decanting the web for the finest text data at scale." NeurIPS 2024 Track Datasets and Benchmarks.

We would like to thank for your thoughtful and insightful reply! We sincerely hope our response can resolve your concerns.

Weakness1. Relying on perplexity for quality assessment may limit generalizability and overlook essential data aspects.

1. DataMan assesses data quality using criteria derived from GPT-4-turbo's analysis of abnormal perplexity causes in documents, rather than using perplexity directly. This method helps mitigate the limitations of LLM's knowledge and capabilities in calculating perplexity.
2. DataMan's 14 quality criteria, detailed in Appendix A, include creativity and sensitivity.

   [7]Creativity: the more creative elements are shown in the text’s expression. _/5
   [13]Sensitivity: the more appropriately sensitive topics are handled in the text, with less inappropriate content. _/5
   

Weakness2 & Question1,2. Lack of comparison with related work.

1. Apart from Qurating, we also compared uniform sampling, data selection with importance resampling (DSIR) [1], and perplexity filtering with the lowest/highest perplexity scores [2,3].

2. It is also worth mentioning that DataMan is compatible and complementary to existing related works (e.g., deduplication, heuristic-based selection, domain mixture, and other LLM quality signals).

3. Following the reviewer's advice, we compared the latest FineWeb project to the strongest variant of existing methods (including DataMan's Overall Score l=5.0). We used FineWeb code to filter the SlimPajama dataset, uniformly sampled 30B tokens, and trained Sheared-Llama-1.3B from scratch. We then presented comparative results on three metrics: ten in-context learning tasks and the perplexity of the SlimPajama validation and test sets.

   Our conclusions are follows:

   • DataMan significantly outperforms FineWeb on all three evaluation metrics.

   • FineWeb performs similarly to Qurating in ten in-context learning tasks, likely due to its internal use of multiple quality filters.

   • FineWeb has poor perplexity performance on SlimPajama validation and test sets, performing only on par with DSIR.

Table 1. Results for ten in-context learning tasks, the perplexity of the SlimPajama validation and test sets, bold indicates the best performance.

Table 2. Detailed results for ten in-context learning tasks, bold indicates the best performance.

We would like to thank you for your valuable feedback and far-sighted insights. We hope the following statements will provide you gain an in-depth understanding of our DataMan method, thereby dispelling your concerns.

Weakness1. Lack of clarity in novelty.

1. We highlight the innovations and contributions of DataMan compared to Qurating as follows:
   1. Different motivation: Our motivation is to enable the LLM to self-select quality criteria that can enhance its performance, as its pre-training capability is related to perplexity [1, 2, 3]. High perplexity indicates that the data is difficult to learn, and vice versa. Thus, this "reverse thinking" encourages us to let the LLM self-identify the causes of anomalous perplexity to derive quality criteria, rather than relying solely on human intuition.
   2. Non-overlapping and more comprehensive quality criteria: We derived 14 quality criteria that do not overlap with Qurating, which together form a more comprehensive data quality system across four levels(see the response to Reviewer8xFT's Weakness1) .
   3. Additional annotated domain types: We have not only quality criteria but also domain types, which can support domain mixing of pre-training and post-training in vertical domains.
   4. Superior annotation, stronger base models, wider document length: We use a superior GPT annotator (GPT-4-turbo vs. GPT-3.5-turbo), a stronger base model (Qwen2-1.5B vs. Sheared-Llama-1.3B), and support a wider range of token lengths for documents ([1, 2k] vs. [256, 512]). This enhances DataMan's evaluation capability, enabling better result of longer and shorter documents.
   5. More feasible Rating: See the response to Question 1 for details.
   6. Better Quality Data Selection: Compared to Quating's sampling method, the data selected by DataMan leads to better LLM performance during pre-training.
2. Reviewers Pb2j and 8xFT indicated that our approach is overall sound and makes sense. Reviewer Tq6k mentioned that using "reverse thinking" to derive quality metrics is innovative, while Reviewer JcNL described it as effective.

Questions1. Concerns regarding i) the Pointwise Rating v.s. Pairwise Rating, ii) Prompt validation, and iii) the Mathematical equation.

1. Clarify misconceptions between Pointwise Rating and Pairwise Rating.

   1. First, the cases in Table 5 are sourced from Qurating's Table 4 and were not handpicked by us. Qurating selected 10 documents from various sources, ranked by the writing style, to test an LLM's nuanced judgment capabilities.

   2. Next, we use Table 5 (displayed in response to Reviewer Pb2j's Question 1) to analyze pointwise and pairwise ratings:

      1. Pairwise rating faces the challenge of minimal quality differences, while pointwise rating aligns with human judgment.

         Table 5 shows that the top three documents have only slight quality differences in "writing style", which evan humans find hard to distinguish (similarly seen in DataMan's "Overall Score.”). This challenges pairwise rating: how does one choose between documents pairs when quality differences are negligible? Generally, if documents meet an "acceptable" quality threshold, they should be accepted. In addressing this challenge, pointwise rating aligns well with human judgment.

      2. Revisiting Qurating's original paper on the use of pairwise evaluation and its issues — API calls, computational costs, and quality difference requirements.

         Appendix B.1 shows that constructing the Quarter's SFT dataset requires GPT-3.5-turbo to make 40 predictions for a a criterion and document pair, using the likelihood of the LLM preference as a label. Table 5 illustrates that document pairs needed at least a 50% selection probability difference in quality criterion, establishing a clear quality differential limit for SFT document pairs. Moreover, for N documents, pointwise rating requires only N assessments, whereas pairwise rating necessitates N(N-1) comparisons, with each comparison requiring 40 predictions*. Thus, when SFT evaluators, API calls, computational costs, and quality difference requirements for document pairs in pairwise ratings are significantly higher than those for pointwise ratings.Aslo, Qurating's Table 6 reveals that for inference, Quarter demands at least a 50% quality difference in criteria for document pairs, raising concerns about its practice applicability.

   3. Finally, we recognize that pointwise and pairwise ratings capture different essences. However, in both SFT evaluators or trillion-token scale pretraining, the applicability and cost-effectiveness outweigh the minor rating performance differences, leading us to choose pointwise rating over pairwise rating.

We express our gratitude for favorable feedback and valuable insights, and we sincerely hope that our response can address your concern.

Weakness1. DataMan needs to validate effectiveness on larger datasets (more than 30B).

Following the reviewers' suggestions, we used the best variants of DataMan (Overall Score l=5.0) and Qurating (Educational Value τ=2.0) to filter 60B data from Slimpajama and trained a Sheared-Llama-1.3B model from scratch. We then compared the results based on three performance metrics: ten in-context learning tasks, and the complexity of the Slimpajama validation and test sets.

Table 1. Results for ten in-context learning tasks, the perplexity of the SlimPajama validation and test sets, bold indicates the best performance.

Table 2. Detailed results for ten in-context learning tasks, bold indicates the best performance.

Table 3. Detailed results for the perplexity of the SlimPajama validation sets, bold indicates the best performance.

Table 4. Detailed results for the perplexity of the SlimPajama test sets, bold indicates the best performance.

Our conclusions are as follows:

1. With a 60B pre-training data scale, DataMan significantly outperforms FineWeb across ten in-context learning tasks.
2. At the 60B pre-training data scale, DataMan's perplexity performance on both the SlimPajama validation and test sets matches that of Qurating. However, apart from C4, we surpass Qurating in perplexity performance from the other six sources.

Weakness2 & Questions1. Analysis and results of applying DataMan to raw, unfiltered web data.

According to the reviewers' adivces, we conducted experiments to analyze DataMan's performance with raw, unfiltered web data. We extracted 1,401,749 documents from the Common Crawl (first 2 splits) of the RedPajama-1T dataset as test data and filtered it using both the Slimpajama [1] and FineWeb [2] data processing rules. We applied DataMan to the test dataset, filtering based on an overall score threshold of less than 4. We then evaluated the metrics between DataMan and the data processing rules based on the following definitions:

• True Positives (TP): Both labeled as dirty;
• False Positives (FP): Rules labeled as clean, DataMan labeled as dirty;
• False Negatives (FN): Rules labeled as dirty, DataMan labeled as clean;
• True Negatives (TN): Both are labeled as clean.

Table 5. Analysis and results of applying DataMan to raw, unfiltered web data.

Results and Analysis:

1. DataMan's high accuracy indicates its effectiveness is comparable to existing data processing rules for web data.
2. Low recall and precision suggest that DataMan is orthogonal to existing rules, especially in identifying low-quality data.
3. In addressing Reviewer JcNL's Weakness2 & Questions 1,2, DataMan outperforms FineWeb, affirming that they are not mutually exclusive as shown in conclusion 2.
4. Case studies on FP and FN show that DataMan better preserves or filters text at a semantic level, surpassing traditional rule-based filtering.
   • Slimpajama FN: A lesson on how to say "I love you" in Spanish

    How do I say \"I Love You\" in Spanish?\nYou can say \"Te quiero\" or \"Te amo\", depending on the intensity of the affection you feel for that person :)\nForum>How do I say \"I Love You\
   

   • Slimpajama FP: Contains numerous typos, lacks semantic and coherence

   Rognan_Norway\nJoAnn Frank Memorial Tree\nA | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | - | ? | \' | " | ( | [ | & | < | ¢ | … | 2 | Ø | … | None\nPierre de Conde\'\nPierreAAAAde Conde\' de Conde\'AAAAPierre 0 1066 2110591 953 Tileris, Normandy, France, France 0 0 1130 2133966 889 64 23375 M YES YES R\nFor technical support or genealogy questions, please contact ourpuzzlingpast_admin
   

   • Fineweb FN: A autobiography of Timothy Reckart as an Actor and Director

   Timothy Reckart\nActor | Director\nTimothy Reckart is an American animator and director based in New York City, specializing in puppet stop motion. He is best known for his 2012 film Head Over Heels, nominated for the Academy Award for Best Animated Short Film in 2013.[1] Reckart grew up in Tucson, Arizona, where he attended University High School.[2] He studied history and literature at Harvard University where he graduated in 2009.[2] ...
   

   • Fineweb FP: Contains nonsensical content and sensitive topics (such as racism, drugs, suicide, and pornography)

   The Murderers At Your Canadian School Boards\nSince 1985, for every Asian computer science major Canadian high schools produce, they must have also produced 15 or 20 white and black crack heads.\nIn 1994 or in 2007 they'd say that gay youth has the highest suicide rate. They'd also say that the media doesn't report on teen suicide because they don't want to encourage copycats.\nWhy doesn't the CBC do some investigative reporting and interview paramedics to see if back in an era when white people watched an episode of Seinfeld and the movie, 'Schindler's List,' all in one day whether good-looking and average looking boys with freckles and South Asian girls who were unattractive or couldn't speak English really had the highest suicide rate? ...
   

5. Compared to uniform sampling, DataMan further improves data quality selection on the Slimpajama dataset, proving its capability to complement existing processes to select superior data.

[1] SlimPajama: A 627B token cleaned and deduplicated version of RedPajama [paper] [Data processing code]

[2] The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale [paper] [Data processing code]

Our conclusions are as follows:

3. A minor drawback is that the MoE model consumes more time and memory during the SFT phase, but this does not affect its usability.
4. Thus, we report to the reviewer that these results demonstrate the effectiveness of the MoE method you suggested (👍). In the future, we will continue to update the DataMan’s base model to better assess quality criteria.

Question2. Methods to reduce the high computational cost of DataMan annotation.

Good question! We introduce specific methods based on two ideas provided by the reviewer:

1. Using lower-cost models for annotation:

   a. Firstly, as mentioned in Appendix B.2, the Score-only DataMan outputs only the overall score (1 token), thereby reducing computational costs.

   b. Additionally, we have trained a smaller Qwen2 0.5B model as DataMan and will release these models together.

   c. Also, changing the DataMan training objective from text generation to text classification can improve efficiency, but it may impact performance.

2. Using heuristic preprocessing techniques to reduce the amount of data annotation in advance:

   a. Before annotating the pre-training dataset, preprocessing techniques in related work can be employed to help lower annotation costs:

   • Deduplication: Fuzzydedup[4], Semdedup[5];
   • Rule-based heuristic selection: Mean word length and stop word fraction[6], C4 filter[7], Gopher rules[8];
   • Model-based heuristic selection: binary grammar discriminators[9];

   b. As suggested by reviewer JcNL, large open-source data processing tools such as Dolma[10] and Fineweb[11] also can reduce annotation costs;

   c. Additionally, the Slimpajama dataset used in this paper has already reduced the cost of DataMan annotation through heuristic rules like NFC normalization, Filter Short Documents, and MinHash Deduplication, on the Redpajama-1T dataset.

Questions3. Further analysis on the misalignment between perplexity and in-context learning performance.

We calculated the perplexity (PPL) and In-context learning (ICL) performance of all models across ten downstream tasks, then plotted the Pearson’s and Spearman’s rank correlation coefficients to explore the misalignment between PPL and ICL.

Table 5. Pearson’s and Spearman’s rank correlation coefficients between PPL and ICL of all models across ten downstream tasks

1. From Table 5, it is evident that the most pronounced misalignment between perplexity and ICL performance occurs in the tasks: LogiQA, and MMLU.

2. To gain deeper insight into how this misalignment affects downstream tasks, we analyzed these two specific tasks and found the following reasons:

   Domain Mismatch: Pre-training often uses broad general corpora, which reduces perplexity on common texts. However, for tasks like MMLU that cover 57 diverse specific domains (such as abstract algebra and anatomy), the issue of domain mismatch becomes more pronounced, leading to decling model performance on these tasks.

   Complexity of ICL Tasks: Many ICL tasks demand complex or multi-step reasoning, which perplexity fails to effectively capture. As seen in LogiQA, which uses expert-written questions from Civil Servants' Exams to test human logical reasoning.

Question4. Typo “vasrtdatasets”.

We have corrected it to "Vast dataset", and all typos have been corrected and highlighted in the revised PDF.

[4] Tao Jiang, et al. Fuzzydedup: Secure fuzzy deduplication for cloud storage. IEEE Transactions on Dependable and Secure Computing, 20(3):2466–2483, 2022.

[5] Amro Abbas, et al. Semdedup: Dataefficient learning at web-scale through semantic deduplication. arXiv 2023.

[6] Hugo Laurençon,et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Neurips 2022.

[7] Colin Raffel, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.

[8] Rae, Jack W., et al. "Scaling language models: Methods, analysis & insights from training gopher." arXiv 2021.

[9] Aakanksha Chowdhery, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.

[10] Soldaini, Luca, et al. "Dolma: An open corpus of three trillion tokens for language model pretraining research." arXiv 2024.

[11] Penedo, Guilherme, et al. "The fineweb datasets: Decanting the web for the finest text data at scale." arXiv 2024.

Questions1. How to resolve situations where DataMan makes unreasonable decisions on criteria?

1. DataMan rarely makes unreasonable decisions on criteria, making it unlikely to cause a 'snowball effect.

   First, we present Table 1. the detailed analysis of testing accuracy of the “Overall Score” quality criterion in DataMan.

   In addition to the 5-level classification accuracy (5-level Acc) reported in the paper, we classify samples with a score of 3 or above as positive samples, and vice versa as negative samples, thus obtaining 2-level classification accuracy: (<3, ≥3 Acc). We detail the accuracy for positive samples (≥3 Acc) and negative samples (<3 Acc), as well as error rates for specific misclassification cases: extreme false negative samples (<2 but ≥3 Error Rate), moderate false negative samples (≥2, <3 but >3 Error Rate), marginal false negative samples (≥2, <3 but =3 Error Rate), extreme false positive samples (≥4 but <3 Error Rate), and marginal false positive samples (≥3, <4 but <3 Error Rate).

   Table 1. The detailed analysis of testing accuracy of the “Overall Score” quality criterion in DataMan.

   	5-level Acc｜	<3, ≥3 Acc｜	≥3 Acc｜	<3 Acc｜	<2 but ≥3 Error｜	≥2, <3 but >3 Error｜	≥2, <3 but =3 Error｜	≥4 but <3 Error｜	≥3, <4 but <3 Error
   Overall Score	81.3	97.5	98.5	81.6	0.2	2.7	15.5	0.2	1.4

   As shown in Table 1, the error rate for DataMan in the two extreme misclassification cases of the "Overall Score" is very low, at just 0.2%. This indicates that DataMan rarely mistakes poor-quality documents for high-quality ones, and vice versa. Considering the strong fault tolerance of pre-training, the snowball effect will not become a bottleneck.

2. Great idea! Inspired by the reviewer, we fine-tuned DataMan using the Qwen 1.5-MoE-A2.7B model with the same SFT data and hyperparameters as the Qwen2-1.5B model. The tables below provide the comparative analysis of the MoE and Dense models in terms of time and memory during the SFT stage, speed and memory during the inference stage, and recognition accuracy across quality criteria and domain types.

   Table 2. Comparison of recognition accuracy on quality criteria between MoE and Dense models.

   Model Type	Score Avg.	Score Var.	Accuracy	Coherence	Language Consistency	Semantic Density	Knowledge Novelty	Topic Focus	Creativity	Professionalism	Style Consistency	Grammatical Diversity	Structural Standardization	Originality	Sensitivity	Overall Score
   Dense	79.2	0.2	78.8	84.1	76.7	82.2	78.4	78.6	79.5	76.8	76.4	73.9	74.8	92.3	75.6	81.3
   MoE	81.0	0.2	80.8	85.7	79.2	83.2	80.6	79.7	81.6	78.4	78.6	75.3	76.9	93.2	78.3	82.7

   Table 3. Comparison of recognition accuracy on domain types between MoE and Dense models.

   Model Type	Domain Avg.	Domain Var.	Medicine	Finance	Law	Education	Technology	Entertainment	Mathematics	Coding	Government	Culture	Transportation	Retail E-commerce	Telecommunication	Agriculture	Other
   Dense	86.0	0.23	95.8	89.8	93.4	86.8	89.0	86.2	90.4	90.6	81.0	80.2	77.8	84.9	87.1	85.8	83.4
   MoE	86.6	0.25	94.3	89.8	90.9	83.9	87.4	88.1	87.5	89.3	85.6	82.0	85.2	77.1	74.2	82.5	86.8

   Table 4. Comparison of time and memory during the SFT stage, and speed and memory during the inference stage, between MoE and Dense models.

   	SFT	SFT	Infer	Infer	Infer w/ vLLM	Infer w/ vLLM
   Model Type	Time (h)	MeM (G)	Output Speed (Toks/S)	MeM (G)	Output Speed (Toks/S)	MeM (G)
   Dense	9.6	313G	75.1	7.2	868	72.9
   MoE	15.6	1988G	6.2	34.1	1429	76.3

Our conclusions are as follows:

1. The MoE model slightly outperforms the Dense Model in quality and domain recognition. The gap may widen with the release of Qwen-2 MoE.
2. Initially, the MoE model is slower (12 times) and uses more memory (4.7 times) than the Dense Model. However, with vLLM optimization, their memory usage becomes comparable, and the MoE model's speed even improves to 1.6 times faster than the Dense Model.

We want to thank you for your thoughtful and insightful reply! We sincerely hope our response can resolve your concerns.

Weakness1. Why 14 scores were chosen and how they differentiate model abilities?

1. These 14 quality criteria were chosen by the LLM itself to enhance its performance, as its pre-training capability is related to perplexity [1, 2, 3]. "High perplexity indicates that the data is difficult to learn, and vice versa." We extracted documents with the top 2% and bottom 2% of perplexity from different sources, using GPT-4-turbo to identify the reasons behind the anomalies perplexity in the documents. Through iterative refinement, we developed 14 quality criteria associated with LLM performance. For a detailed process, please refer to the response to Reviewer Pb2j's Question 2.
2. These 14 quality criteria were selected to provide a step-by-step comprehensive assessment of text quality through the following four levels:
   • Basic Requirements: These include Accuracy, Language Consistency, Style Consistency, Grammatical Diversity, and Structural Standardization, ensuring readability.
   • Expression Requirements: Covering Coherence, Semantic Density, Originality, and Topic Focus, aimed at enhancing expression and informativeness.
   • Advanced Requirements: Involving Knowledge Novelty, Creativity, Professionalism, and Sensitivity, to deepen and secure text quality.
   • Overall Score: allows the LLM to weigh each quality criterion, avoiding human bias and providing a comprehensive quality evaluation.
3. According to the original paper's Table 8, the scores of quality criteria across different domains show how these criteria differentiate model abilities:
   • Knowledge Novelty excelled in mathematics, enhancing the model’s math abilities.
   • Creativity was highest in cultural domains, boosting the model's literary skills.
   • Professionalism performed exceptionally in mathematics, law, medicine, and finance, improving the model's performance in these professional fields.
   • Sensitivity scored lower in government and entertainment domains, aiding the model in filtering harmful speech and politically sensitive topics.
   • Originality and Semantic Density scored higher in vertical domain data, enhancing the model's performance in vertical fields.
   • Other criteria were performed evenly across domains, ensuring the basic requirements were met and enhancing the model’s general capabilities.
4. From the original paper's Table 15, one can observe the quality criteria that performed best in various ICL tasks, analyzing how these criteria enhance the LLM's abilities in specific downstream tasks:
   • High Semantic Density data improved the LLM’s ability to solve grade-school science questions (ARC-E, ARC-C), while more challenging science exam questions (SciQ) required data with high Semantic Density and Professionalism.
   • High Creativity data assisted the LLM in handling tasks like article summarization and movie subtitles (HellaSw., W.G.), while more complex physical interactive reasoning tasks (PIQA) required a combination of high Semantic Density.
   • High Originality supported the LLM in solving Wikipedia knowledge-related tasks, where redundant knowledge was not helpful for such problems.
5. The original paper's Table 13 shows the quality criteria with the lowest perplexity in different data sources, analyzing how these criteria enhance the LLM's fitting ability in specific data sources.
   • Sensitivity had the lowest perplexity in general web domains CommonCrawl and C4, indicating that when handling open web data, the key is to recognize and avoid sensitive topics, thereby improving adaptability to web content.
   • In the Wikipedia source, Semantic Density, Originality, and Topic Focus performed better, suggesting that the LLM absorbs world knowledge through high-information, original thematic content.
   • Creativity had the lowest perplexity in book sources, indicating that integrating more creative elements can help the LLM better learn and understand literature.

[1] Wenzek, Guillaume, et al. "CCNet: Extracting high quality monolingual datasets from web crawl data." LREC 2020.

[2] Muennighoff, Niklas, et al. "Scaling data-constrained language models." Neurips 2023.

[3] Marion, Max, et al. "When less is more: Investigating data pruning for pretraining llms at scale." Neurips 2023.

Thank you for acknowledging our work. We are pleased that our response has addressed your concerns. In the final version of our paper, we will polish it further to incorporate the valuable insights gained from the rebuttal discussion. Thank you once again!

Yours truly,

Authors