SCAR: Efficient Instruction-Tuning for Large Language Models via Style Consistency-Aware Response Ranking

Response to Reviewer Comments

We thank the reviewer for their insightful feedback and appreciate the opportunity to clarify our work. Below, we address your concerns point by point.

1. Clarification on the Definition and Measurement of Semantic Surprisal

We acknowledge that our initial definition of Semantic Surprisal—"the choices of solutions, ideas, or approaches in a response that affects how predictably or unexpectedly it addresses the instruction"—may seem broader than what perplexity measures. However, perplexity is a well-established metric for quantifying text surprisal levels, as supported by references [1]-[3]:

• [1] Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? Transactions of the Association for Computational Linguistics.
• [2] Strong prediction: Language model surprisal explains multiple N400 effects. Neurobiology of Language.
• [3] Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018).

In our work, we focus on the surprisal levels of the semantically relevant portions of the text. Specifically, we calculate the perplexity PPL(y_c | x), where y_c represents the content words (semantic components) of the response, and x is the instruction. By filtering out functional words y_p, we aim to isolate the semantic content.

Addressing the Concern About Disrupting Sentence Fluency

We understand that removing functional words might disrupt sentence fluency, potentially affecting perplexity measurements. To address this concern, we also calculated the perplexity of the complete responses PPL(y | x) in Table13 in Appendix and found similar patterns with those obtained using only the semantic portions PPL(y_c | x).

Further Analysis Using the LIMA Dataset

To validate that non-semantic linguistic form features contribute minimally to the variance of response surprisal, we conducted additional analysis using the LIMA dataset:

• Calculations:

  • PPL(y_c|y_p,x): Perplexity of the content words given the functional words and the instruction.
  • PPL(y_p|y_c,x): Perplexity of the functional words given the content words and the instruction.
  • PPL(y|x): Perplexity of the full response given the instruction.

• Variance Explanation:

  • We calculated the variance of PPL(y_c|y_p,x) and PPL(y_p|y_c,x) and examined how much each explains the variance in PPL(y|x) by dividing var(PPL(y_c|y_p,x)) by var(PPL(y|x)) and var(PPL(y_p|y_c,x)) by var(PPL(y|x)).
  • Results:
    • The variance of PPL(y_p|y_c,x) explains only 4% of the variance in PPL(y|x).
    • The variance of PPL(y_c|y_p,x) explains 67% of the variance in PPL(y|x).

• Regression Analysis:

  • We built a regression model where PPL(y|x) is the dependent variable, and PPL(y_c|y_p,x) and PPL(y_p|y_c,x) are the independent variables.
  • Findings:
    • The regression coefficient for PPL(y_c|y_p,x) is approximately 0.5.
    • The regression coefficient for PPL(y_p|y_c,x) is near zero (0.04).

The results indicate that the variation in the perplexity of the full response (PPL(y|x)) is largely influenced by the perplexity of the content words (PPL(y_c|y_p,x)), not by the functional words. This supports our hypothesis that semantic content (captured by PPL(y_c|y_p,x)) is the primary contributor to surprisal levels, justifying our use of perplexity over y_c to measure semantic surprisal.

We thank the reviewer for their valuable feedback and constructive suggestions. We address your questions below.

Q1: The "longest" method discussed in prior work [1, 2] appears to be a strong rule-based baseline. How does its performance compare with SCAR?

We conducted experiments comparing SCAR with the "longest" method as a baseline, using AlpacaEval in the open domain with LLAMA3-8B. The performance (L.C. WinRate) of models fine-tuned on data selected by the "longest" method is as follows:

Human Data Selected by Length:

• 5,000 examples: 1.46
• 2,500 examples: 1.75
• 1,000 examples: 1.27

Mixed Synthetic Data Selected by Length:

• 5,000 examples: 6.29
• 2,500 examples: 5.32
• 1,000 examples: 6.61

We observe that when selecting from human data, the performance of the "longest" method is significantly lower than that of SCAR. When selecting from mixed synthetic data, the performance is comparable to SCAR. This is because the longest responses tend to be those generated by a single methods like Evol-Instruct, unifying the writing style, which produce longer outputs due to their construction process. While length can be a strong indicator of certain stylistic features in linguistic form, our method considers a broader range of style elements, making SCAR more general and effective.

Q2: The performance of SCAR on some objective benchmarks is expected

Please refer to our response to Reviewer W39Q, where we provide detailed analysis and results on additional benchmarks, including MMLU. We have shown that SCAR improves performance on these benchmarks, demonstrating its effectiveness across various tasks.

Q3: Could you provide more details about how were the embeddings created for Figure 1 (Left)?

To validate that linguistic form is more consistent within synthetic data, we extracted non-semantic features from the responses using heuristics, including:

• Unigrams of functional words
• Type-Token Ratio (TTR) of non-semantic functional words
• Measure of Textual Lexical Diversity (MTLD) of non-semantic functional words
• Number of sentences
• Flesch Reading Ease score
• Punctuation frequency

We used these features to create embeddings representing each response. Using t-SNE, we visualized these embeddings in Figure 1 (Left). Although we could include more non-semantic features, we believe these are sufficient to demonstrate our point about the consistency of linguistic form in synthetic data.

Again, we appreciate your insightful comments and believe that addressing these points strengthens our work.

We sincerely thank the reviewer for their thoughtful feedback and valuable suggestions. We address each of your concerns and questions below.

1. Clarification on the Position of the Selection Method

Our method, SCAR, is designed to be versatile and applicable to both general-purpose and specialized models. We have demonstrated its effectiveness in fine-tuning models in both the code domain (e.g., CodeLLaMA) and the open-ended question-answering domain (e.g., OLMO-7B). The primary goal of SCAR is to improve the efficiency of supervised fine-tuning (SFT) by selecting style-consistent and high-quality examples, enhancing performance regardless of the model's specialization.

2. Benchmark Performance on General Tasks

To verify the effectiveness of SCAR in training general models, we conducted additional experiments on various downstream tasks. Following the evaluation protocol from "Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning," we evaluated the OLMO-7B model fine-tuned on SCAR-selected data using the following benchmarks:

• ARC (Easy + Challenge)
• HellaSwag
• MMLU
• TruthfulQA

Results:

We fine-tuned OLMO-7B on 10k examples selected by SCAR and compared its performance with the official OLMO-7B-SFT checkpoint trained on the full 320k dataset. Despite using 32 times less data, our SCAR-fine-tuned model achieves competitive performance, outperforming the official checkpoint on ARC, HellaSwag and TruthfulQA, while performing lower on MMLU.

Evaluation Results

Note: For ARC and HellaSwag, we used likelihood-based accuracy due to limitations in extracting generated answers using the lm-evaluation-harness toolkit. For TruthfulQA and MMLU, we evaluated the generated texts using BLEU scores and accuracy, respectively.

These results demonstrate that SCAR can effectively reduce the amount of required training data while maintaining or even improving performance on certain benchmarks. The decrease in performance on MMLU may be due to the reduced data size affecting the coverage of specific knowledge areas.

3. Selection of Format-Consistent Data vs. Matching GPT-3.5 Outputs

Given our analysis in Section 2, we observed that LLM-generated data, such as that from GPT-3.5, tend to be more consistent in style compared to human-generated data. Therefore, selecting data that is similar to GPT-3.5's style can enhance the overall style consistency of the dataset. Our results in Table 3 validate this, showing that the variance of linguistic metrics like TTR (Type-Token Ratio) and PPL (Perplexity) are reduced in most cases when we select data that aligns with GPT-3.5's style.

While our current implementation focuses on GPT-3.5 due to its availability and consistent outputs, the same approach can be adapted to select data that aligns with the style of other models, such as LLaMA, or any preferred style. The key aspect is that style consistency improves fine-tuning performance, and SCAR is flexible enough to be tailored to any target style to achieve this consistency.

5. Justification for the Referenced Response Method

The use of "referenced responses" is crucial for our analysis:

Referenced responses serve as intermediates that retain the semantic content of human responses while adopting a more consistent linguistic form.

Our rewriting approach aligns with techniques from the ACL paper "Self-Distillation Bridges the Distribution Gap in Language Model Fine-Tuning." Including the instruction in the prompt ensures that the model considers the context, reducing hallucinations and preserving semantic integrity.

This process allows us to disentangle the effects of linguistic form and semantic surprisal on fine-tuning performance, as we can control for one while varying the other.

Response to Concerns on Dataset and LLM Size Limitations:

We appreciate the reviewer’s feedback regarding the dataset and model size. However, we validated our method across a range of models and data sources, strengthening the robustness and generalizability of our results. Specifically:

• Model Variety and Real-World Relevance: Our method is tested across four different base LLMs, including CodeLlama, Llama3, and widely recognized open-source models like OLMO and Starcoder, all of which are commonly used in real-world applications. This diversity of models ensures that our approach is not restricted to a single model setup, enhancing the reliability of our findings.

• Dataset Diversity and Scale: We validated our selection method on the large-scale open-source dataset, the OLMO tulu dataset, with 320,000 examples, which offers a rich source of real-world data. Our own curated dataset (20,000 and 10,000 examples) for data selection is also comparable in scale to widely used benchmarks like Alpaca (52,000 examples), LIMA (1,000 examples), and Guanaco-Commits (13,000 examples). All these prior datasets have been effectively used to train numerous open-source models. This similarity in dataset sizes underscores that our approach is grounded in scales commonly used in prior studies.

• Validation Across Data Curation Methods: Our approach is further validated by applying it to both open-source datasets and our own curated datasets, collected by simulating two real-world data collection scenarios. This range of data sources demonstrates the flexibility and adaptability of our method, showing that it is effective across diverse data curation contexts and is not limited by dataset scale.

By leveraging multiple models and diverse datasets, including large-scale, real-world sources, our study provides strong validation for the scalability and applicability of our approach.

Motivation and Positioning on Diversity of Instructions:

• We appreciate the reviewer's feedback on our motivation. We would like to clarify that our work does not contradict the value of diversity in instruction-tuning data. Instead, our study aims to explore the role of style consistency within a diverse dataset, offering a complementary perspective to traditional diversity-based selection methods. As demonstrated in Table 1, we intentionally use the same set of instructions across different experimental conditions to isolate the impact of style consistency, thereby excluding diversity effects. This setup enables us to identify phenomena that are independent of instruction diversity and highlight how style consistency can enhance model performance even within diversified datasets. While we recognise the value of diversity, it is beyond the scope of our paper's discussion. Combining these two approaches could be a promising direction for future applications.

• Regarding the observed performance of diversity selection, we explain in the Experiments section that this method is less effective in our context because our full dataset is already inherently diversified due to the crowdsourcing data curation methods. Consequently, diversity-based methods like random sampling and diversity selection do not provide the same advantage here as they might in scenarios with less inherent diversity.

Clarification on Semantic Surprisal and Use of Perplexity:

Semantic surprisal is rooted in the well-established concept of text surprisal in NLP, commonly measured by perplexity, as shown in prior works mentioned in the paper [1,2,3]. Using this foundation, we adopt perplexity to quantify surprisal. However, our approach goes a step further by connecting text surprisal to LLM fine-tuning performance—an unexplored area in previous studies. To examine the distinct impacts of semantic and non-semantic components on LLM performance, we decompose responses into segments that capture semantic versus non-semantic content, allowing us to evaluate how each component individually influences alignment.

[1] Byung-Doh Oh and William Schuler. Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? Transactions of the Association for Computational Linguistics

[2] JA Michaelov, MD Bardolph, CK Van Petten, BK Bergen, and S Coulson. Strong prediction: Language model surprisal explains multiple n400 effects. neurobiology of language, 1–71.

[3] Adam Goodkind and Klinton Bicknell. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th workshop on cognitive modeling and computational linguistics (CMCL 2018),

Explanation of LLAMA2-13B Performance Degradation:

• The observed performance degradation with LLAMA2-13B occurs in two contexts:
  • Data Generation for Fine-Tuning: As shown in Table 1, LLAMA2-13B-chat was used to generate data for training the base models, CodeLlama-7b and Llama3-8b. However, the LLAMA2-13B-chat model often produced low-quality data, including hallucinations and unhelpful responses, which negatively impacted fine-tuning during both response rewriting and direct generation.
  • Ablation Study – Ranker Training: In the Ablation Study, degradation also arises when LLAMA2-13B-chat-generated data is used to train our ranker. Low-quality training data led the ranker to prioritize style-consistent but low-quality responses, which ultimately detracted from the LLM’s performance in SFT. Our findings emphasize the importance of both quality and style consistency in training data for effective ranker training and, consequently, LLM fine-tuning.