T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning

We thank the reviewer for the encouraging assessment on the technical novelty, efficiency, experimental thoroughness, and presentation quality of our work. We also appreciate the thoughtful questions and constructive suggestions. Below, we address the weakness (W) and the questions (Q) raised by the reviewer.

Notations and clarifications.

In our response, we follow the draft’s notations for several benchmark names: ARC-C (ARC-Challenge), HS (HellaSwag), TQA (TruthfulQA), GSM (GSM8k), AH (Arena-Hard), and AE-2 (AlpacaEval 2.0). We use $\mu_{\text{open}}$ to denote the average performance across six OpenLLM Leaderboard benchmarks, $\mu_{\text{LLM}}$ for the average performance across two LLM-as-a-Judge benchmarks, and $\mu_{\text{all}}$ for the average performance over all eight benchmarks. Additionally, we clarify that Gemini-2.5-Flash-Preview-04-17 was used in our draft as the judge for evaluating AH. Since this checkpoint is no longer available via API, we now use the stable version of Gemini-2.5-Flash, which affects the absolute scale of AH scores.

W1 & Q2: Applicability to larger models.

We thank the reviewer for this constructive suggestion. To evaluate the scalability of our method, we fine-tuned a Qwen-2.5-14B model using data selected from the Magpie dataset, following the setup described in Table 2. Due to time constraints, we used a token-selection ratio of $k$% $= 50$% without further hyperparameter tuning. The results are presented below.

T-SHIRT consistently outperforms the strongest baselines across $\mu_{\text{open}}$, $\mu_{\text{LLM}}$, and $\mu_{\text{all}}$. Notably, it achieves an average improvement of at least 1.75 points on the LLM-as-a-Judge benchmarks. These results suggest that our method remains effective and robust when applied to instruction tuning of larger models beyond the 7–8B scale.

We will include these results in the revised version of the draft.

Q1: Comparison between random noise injection and lexical perturbations.

Compared to random noise injection at the embedding level, lexical perturbations present several challenges:

• Utility. Rule-based lexical perturbations may fail to maintain semantic equivalence and fluency. Prior studies $1, 2$ typically rely on expensive human evaluations to verify the utility of perturbed examples.
• Efficiency. Generating lexical variants via prompting powerful LLMs through APIs is computationally and financially expensive. For instance, perturbing each of the 52k Alpaca-GPT4 samples 30 times would require over 1.5 million API calls.

Given these limitations, we adopt random noise injection as a lightweight and scalable alternative to lexical perturbations. Investigating how systematic lexical perturbations can affect selection efficacy remains an exciting direction for future work.

References

$1$ D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment, 2020.

$2$ B. Wang, C. Xu, S. Wang, Z. Gan, Y. Cheng, J. Gao, A. H. Awadallah, and B. Li. Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models, 2021.

We thank the reviewer for the positive feedback on the technical novelty, comprehensive experiments, and ablation studies in our work. We also appreciate the reviewer’s insightful comments and questions on token-level informativeness and the choice of scoring models. Below, we provide itemized responses and hope they can address weaknesses (W) and questions (Q) raised by the reviewer.

Notations.

We use the following notations throughout our response, and they are consistent with our draft.

• $\mathcal{D}$: instruction-tuning dataset;
• $\theta$: LLM parameters to be fine-tuned;
• $(x, y)$: instruction–response pair;
• $y_t$: the $t$-th token in the response;
• $|\Delta_t|$: token-level informativeness for the token $y_t$;
• Benchmark abbreviations: ARC-C (ARC-Challenge), HS (HellaSwag), TQA (TruthfulQA), GSM (GSM8k);
• $\mu_{\text{open}}$: average score over six OpenLLM Leaderboard benchmarks.

W1: Token-level advantages of T-SHIRT.

We appreciate the reviewer’s suggestion to investigate whether T-SHIRT offers token-level advantages during instruction tuning.

Tokenizer difference limits direct comparison. The models used in our pipeline, GPT-2 for scoring, and Llama-3.1-8B and Qwen-2.5-7B for fine-tuning, employ different tokenizers, and this makes direct token-level comparisons nontrivial. Therefore, we analyze the training dynamics as a proxy.

Higher training loss indicates more informative tokens. We observe that LLMs fine-tuned on data selected by T-SHIRT consistently exhibit higher training loss compared to those trained on baseline-selected data. This pattern holds across both Llama-3.1-8B and Qwen-2.5-7B, and across datasets (Alpaca-GPT4 and Magpie). Since the instruction-tuning loss is defined as: $\mathcal{L}(\theta) = -\frac{1}{\sum_{(x,y)\in\mathcal{D}} |y|} \sum_{(x,y)\in\mathcal{D}} \sum_{t=1}^{|y|} \log P_\theta\left(y_t | x, y_{<t}\right)$, this suggests that data selected by T-SHIRT provides more informative and challenging tokens for instruction tuning. We believe this observation provides meaningful evidence of token-level advantages.

W2: Distribution and impact of low $|\Delta_t|$ values.

We thank the reviewer for raising thoughtful questions about our decision to remove tokens with low $|\Delta_t|$ when computing S-IFD.

Cumulative effect of small $|\Delta_t|$ values. First, we clarify that tokens with small $|\Delta_t|$ values can still have a significant cumulative effect. For instance, when using a token selection ratio of $k$% $= 75$%, we remove the bottom 25% of tokens (those with $|\Delta_t| \le 0.013$) from the S-IFD calculation. For a relatively long input, a lot of tokens are less informative and have low $|\Delta_t|$, but cumulatively, they could still add up. As illustrated in Figure 2 in the draft, we can differentiate between two samples with nearly identical IFD scores (0.998 vs. 0.998) effectively by their S-IFD scores (0.751 vs. 0.968).

Distribution of $|\Delta_t|$. Second, since the rebuttal guideline does not allow presenting figures, we include a table to show the distribution of $|\Delta_t|$ values computed by GPT-2 across the Alpaca-GPT4 dataset.

For example, 20% of the tokens have $|\Delta_t|$ below 0.009, and 50% of the tokens have $|\Delta_t|$ below 0.050. suggesting a considerable portion of tokens are uninformative.

Alternative weighting schemes. Third, following the reviewer’s suggestion, based on Table 1, we conduct new experiments using Qwen-2.5-7B and Alpaca-GPT4 to explore alternative weighting schemes. In Equation (3), our default approach assigns a weight $w_t = 1$ to informative tokens and $w_t = 0$ to the rest. We compared this to two alternative schemes:

• Weights $1.5:1$. $w_t = 1.5$ for informative tokens, $w_t = 1$ for uninformative tokens.
• Weights $2:1$. $w_t = 2$ for informative tokens, $w_t = 1$ for uninformative tokens.

The results are shown below.

Experimental results show that both weighting schemes outperform the IFD baseline on $\mu_{\text{open}}$, confirming the benefit of emphasizing more informative tokens during data selection. However, our default setting that removes uninformative tokens (i.e., setting $w_t = 0$ like Equation (3)) achieves even stronger performance, both on average and across all six individual benchmarks.

These results validate our design choice. We will include the analysis and new experiment results in our revision.

Q1: Other scoring models.

GPT-2 vs. larger models. In our early explorations, we considered several alternative scoring models and ultimately chose GPT-2 for its strong efficiency and accessibility, as it can run quickly on a single consumer-grade GPU. While different scoring models can influence the quality of selected data, we refer the reviewer to the related work Superfiltering $1$ for further information. Its Table 3 shows that larger models may offer only marginal improvements in data selection quality.

Open-source models vs. proprietary models (APIs). Additionally, we chose open-source models over closed-source APIs because they provide access to token-level information and token embeddings, which are essential for our method. In contrast, querying more powerful proprietary LLMs via APIs typically returns only coarse-grained, categorical quality scores, as used in methods like DEITA $2$ and DS$^2$ $3$ . As shown in our Table 1, this lack of fine-grained information negatively affects the quality of selected data.

References

$1$ M. Li, Y. Zhang, S. He, Z. Li, H. Zhao, J. Wang, N. Cheng, and T. Zhou. Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning, 2024.

$2$ W. Liu, W. Zeng, K. He, Y. Jiang, and J. He. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning, 2024.

$3$ J. Pang, J. Wei, A. Shah, Z. Zhu, Y. Wang, C. Qian, Y. Liu, Y. Bao, and W. Wei. Improving Data Efficiency via Curating LLM-Driven Rating Systems, 2025.

Dear authors, thanks for your detailed response. All my concerns have been addressed and I have raised my score to 5.

We thank the reviewer for the encouraging reviews on the technical novelty, empirical strength, and efficiency of our approach, as well as the quality of our writing. We also greatly appreciate the insightful feedback. Below, we provide detailed responses to each question (Q) raised by the reviewer.

Notations and clarifications.

In our response, we follow the draft’s notations for several benchmark names: ARC-C (ARC-Challenge), HS (HellaSwag), TQA (TruthfulQA), GSM (GSM8k), AH (Arena-Hard), and AE-2 (AlpacaEval 2.0). We use $\mu_{\text{open}}$ to denote the average performance across six OpenLLM Leaderboard benchmarks, $\mu_{\text{LLM}}$ for the average performance across two LLM-as-a-Judge benchmarks, and $\mu_{\text{all}}$ for the average performance over all eight benchmarks. Additionally, we clarify that Gemini-2.5-Flash-Preview-04-17 was used in our draft as the judge for evaluating AH. Since this checkpoint is no longer available via API, we now use the stable version of Gemini-2.5-Flash, which affects the absolute scale of AH scores.

Q3: Applicability to larger models.

We thank the reviewer for this valuable suggestion. To address this concern, we use the settings presented in Table 2 and fine-tune a Qwen-2.5-14B model on data selected from the Magpie dataset. Here, we still use the token-selection ratio $k$% of 50% without further hyperparameter tuning because of time limitations. The evaluation results are shown below.

Our method consistently surpasses the strongest baselines in terms of $\mu_{\text{open}}$, $\mu_{\text{LLM}}$, and $\mu_{\text{all}}$. In particular, it yields an average gain of at least 1.75 points on LLM-as-a-Judge benchmarks. These findings indicate that our data selection approach remains robust and advantageous for instruction tuning of models beyond the 7–8B parameter scale.

We will include the new experiment in our revision.

Q1: Statistical significance of results for randomized methods.

We thank the reviewer for raising the importance of demonstrating the stability of our method, given the stochasticity in both T-SHIRT and certain baselines. We conduct repeated experiments on Qwen-2.5-7B, running both the Random baseline and our method (T-SHIRT with $k = 50$) three times with different random seeds. We report the mean and standard deviation of their performances across the OpenLLM Leaderboard benchmarks. Due to the budget constraints, we were unable to re-run DS$^2$ $1$ , since its randomness primarily stems from API queries, and re-scoring the dataset would require an expensive number of API queries. For context, we reiterate the performance for DS$^2$ and IFD from Table 1. Standard deviations for the Random baseline and T-SHIRT are shown in brackets.

These results show that our method consistently outperforms other baselines. Our method provides a considerable margin of 1.19 points over DS$^2$ on $\mu_{\text{open}}$, which makes it quite likely that this margin would remain robust, even under repeated trials for DS$^2$.

We will include these statistical significance results in the revised version.

Q2: Selection and tuning of hyperparameter $k$.

We agree that the token-selection ratio $k$ used in S-IFD is a critical hyperparameter. We offer the following recommendations based on our findings:

• Default recommendation. When resources are limited, we recommend starting with $k = 50$, which performs well across multiple models and datasets, as shown in both Table 1 and Table 2.

• Fine-grained tuning. On Llama-3.1-8B with the Alpaca-GPT4 dataset, we conduct a sweep over $k \in \lbrace 50, 60, 70, 80, 90 \rbrace$. We find that our method works well across different values of $k$. The performance peaks at $k = 70$, with T-SHIRT outperforming IFD on 7 out of 8 benchmarks and remaining competitive on AH. It also achieves a 1.12-point improvement in $\mu_{\text{all}}$.

These findings suggest that $k$ can be flexibly chosen within the range of 50–90. When tuning is feasible, it can provide additional gains.

• Transferability across model scales. For the Qwen-2.5 family, we find that $k = 50$ performs robustly across both 7B and 14B models, and across different instruction-tuning datasets (Alpaca-GPT4 and Magpie). We hypothesize that models within the same family share similar pretraining distributions, making it feasible to tune $k$ on a smaller model and transfer that setting to a larger one.

We will include this discussion and guidance in the revision.

Dear reviewer wj2T,

Thank you for your insightful feedback and for your active engagement during the discussion! We are very grateful for your helpful comments and suggestions, which have significantly contributed to improving our work. We will make sure to include the additional results and clarifications in the revised version of the paper.

We thank the reviewer for the positive assessment of our method’s scalability, well-motivated design, and efficiency, and we appreciate the constructive feedback. We hope the following clarifications help address the weaknesses (W) and questions (Q) raised by the reviewer.

Notations and clarifications.

In our responses, we follow the draft’s notations for several benchmark names: ARC-C (ARC-Challenge), HS (HellaSwag), TQA (TruthfulQA), GSM (GSM8k), AH (Arena-Hard), and AE-2 (AlpacaEval 2.0). We use $\mu_{\text{open}}$ to denote the average performance across six OpenLLM Leaderboard benchmarks, $\mu_{\text{LLM}}$ for the average performance across two LLM-as-a-Judge benchmarks, and $\mu_{\text{all}}$ for the average performance over all eight benchmarks. Additionally, we clarify that Gemini-2.5-Flash-Preview-04-17 was used in our draft as the judge for evaluating AH. Since this checkpoint is no longer available via API, we now use the stable version of Gemini-2.5-Flash, which affects the absolute scale of AH scores.

W1 & Q1: Performance comparison between IFD and T-SHIRT across benchmarks.

We thank the reviewer for the thoughtful suggestion regarding finer hyperparameter tuning. In our draft, we used $k = 50$ and $k = 75$ to show that T-SHIRT can perform well even without heavy hyperparameter tuning. To explore hyperparameter choice further, we conduct a finer sweep over $k \in \lbrace 50, 60, 70, 80, 90 \rbrace$ using the same dataset and evaluation benchmarks from Table 1. We found that for Llama-3.1-8B, performance peaks at $k = 70$. We show the results in the following table.

With $k = 70$, T-SHIRT outperforms IFD on 7 out of 8 benchmarks and performs competitively on the remaining one (AH). Specifically, T-SHIRT improves over IFD by:

• +1.01 points on $\mu_{\text{open}}$,
• +1.46 points on $\mu_{\text{LLM}}$,
• +1.12 points on $\mu_{\text{all}}$.

These results confirm that with appropriate tuning of $k$, T-SHIRT can consistently outperform IFD. We will include these results in the revised version.

W2 & Q2: Different choices of hyperparameter $k$ for Llama and Qwen.

Clarification on the token selection ratio $k$%. We clarify that, in Equation (3), we include a token $y_t$ in the computation of S-IFD only if its token-level informativeness $|\Delta_t|$ ranks in the top $k$% of the dataset. This means that we filter out the least informative $(100-k)$% of tokens, which are often easy to predict and potentially noisy.

Interpretation for Figure 6a. The trends in Figure 6a show that:

• On Llama-3.1-8B, using $k=75$, i.e., removing 25% of the least informative tokens from S-IFD computation, yields strong performance. In contrast, setting $k=25$, which removes 75% of the tokens, is too aggressive and degrades performance.
• On Qwen-2.5-7B, we observe the best performance at $k=50$, indicating that this stronger pre-trained model can benefit from filtering out a larger fraction (50%) of the least informative tokens.

These trends align with our pilot analysis (L151–L152 in our draft), which indicates that at least 22–28% of tokens are uninformative and can be excluded from S-IFD computation. We will include our clarification and additional results (e.g., for $k = 70$ that removes 30% of the least informative tokens) in the revision.

W4 & Q3: Token embedding perturbations and word substitutions.

The random noise is continuous. The noise injected into token embeddings is continuous and cannot be directly mapped back to discrete tokens, unlike word substitutions (e.g., replacing "mean" with "average"). By choosing an appropriate noise level, the perturbed embedding remains close to the original token embedding in the embedding space.

Intention of choosing token embedding perturbations. We clarify that we deliberately designed embedding perturbation as a computationally efficient alternative to simulating lexical perturbations such as word substitutions. Below, we provide further discussion of our rationale for choosing continuous embedding perturbation:

• Utility. Rule-based substitutions may struggle to preserve semantic equivalence and fluency. Prior works $1, 2$ require costly human evaluations to verify the utility.
• Efficiency. Prompting APIs to generate perturbed variants is prohibitively expensive. For example, perturbing each of the 52k Alpaca-GPT4 samples 30 times would require over 1.5 million API queries.

Therefore, embedding perturbation is a lightweight, efficient way to consider robustness in scoring.

We will clarify this distinction and rationale more clearly in the revision.

References

$1$ D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment, 2020.

$2$ B. Wang, C. Xu, S. Wang, Z. Gan, Y. Cheng, J. Gao, A. H. Awadallah, and B. Li. Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models, 2021.

W3: Feasibility of DEITA/DS$^2$ with smaller models (e.g., GPT-2).

We thank the reviewer for this thoughtful suggestion. It inspires us to reflect on how T-SHIRT compares with methods that rely on API-based scoring.

Discussion on DEITA. For DEITA, two critical components depend on powerful LLMs like ChatGPT:

• Dataset evolution. DEITA uses the API to generate increasingly complex and higher-quality instruction data following WizardLM $1$ . GPT-2 is not capable of performing this evolution.
• Scoring and supervision. DEITA prompts ChatGPT to score each evolved sample, then uses these scores as supervision to train scorer LLMs. GPT-2 is not suitable for this direct scoring via prompting. Using GPT-2 would likely require fine-tuning it to serve as a weak classifier.

Discussion on DS$^2$. DS$^2$ relies directly on prompting large LLMs for sample scoring, so it would likewise be infeasible to use GPT-2 as a substitution.

We believe this discussion illustrates a practical advantage of our method and will include this comparison in the revision.

References

$1$ C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, and D. Jiang. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions, 2024.

W5: Novelty of our method.

We acknowledge that token-level scoring and neighborhood smoothing for robustness have been explored in other contexts. We have reviewed related work in other contexts in Section 1.1 of our draft. However, to the best of our knowledge, we are the first to question the well-accepted paradigm in instruction tuning data selection, where each sample is scored as a whole unit and selected via a hard threshold. One of the key contributions of our work is a novel perspective: combining fine-grained token-level informativeness with neighborhood-level robustness for data selection. Our approach offers new insights into what makes a data point valuable for instruction tuning.

We thank the reviewer for appreciating our technical contributions and high-quality analysis. We also thank the reviewer for the constructive comments. We hope the following responses address the weaknesses (W) raised by the reviewer.

Notations and clarifications.

In our responses, we follow the draft’s notations for several benchmark names: ARC-C (ARC-Challenge), HS (HellaSwag), TQA (TruthfulQA), GSM (GSM8k), AH (Arena-Hard), and AE-2 (AlpacaEval 2.0). We use $\mu_{\text{open}}$ to denote the average performance across six OpenLLM Leaderboard benchmarks, $\mu_{\text{LLM}}$ for the average performance across two LLM-as-a-Judge benchmarks, and $\mu_{\text{all}}$ for the average performance over all eight benchmarks. Additionally, we clarify that Gemini-2.5-Flash-Preview-04-17 was used in our draft as the judge for evaluating AH. Since this checkpoint is no longer available via API, we now use the stable version of Gemini-2.5-Flash, which affects the absolute scale of AH scores.

W1: Performance gains of our method.

We believe this concern primarily stems from the hyperparameter settings used in our main experiments, particularly the token selection ratio $k$% defined in Equation (3). In the draft, we limited our evaluation to $k = 50$ and $k = 75$ to demonstrate that our method performs well even without extensive hyperparameter tuning.

To address this concern, we conducted a finer-grained sweep over $k \in \lbrace 50, 60, 70, 80, 90 \rbrace$, using the same instruction tuning dataset and benchmark setup as in Table 1. On Llama-3.1-8B, performance peaks at $k = 70$. The full results are provided below.

With $k = 70$, our method, T-SHIRT, yields a clear improvement over the strongest baseline, IFD:

• On $\mu_{\text{open}}$, T-SHIRT outperforms IFD by 1.01 points.
• On $\mu_{\text{LLM}}$, which better approximates human preferences, T-SHIRT shows a 1.46-point gain.
• Overall, T-SHIRT achieves a 1.12-point improvement in $\mu_{\text{all}}$.
• T-SHIRT outperforms IFD on 7 out of 8 benchmarks and performs comparably on the remaining benchmark (AH).

These results indicate that after tuning a single hyperparameter $k$, T-SHIRT consistently delivers stronger performance. We will include these results in the revised version.

W2: Interpretation of ablation study results.

Both components bring improvements. Our ablation study in Table 4 demonstrates that both of our proposed components, (1) S-IFD, which estimates token-level informativeness, and (2) hierarchical selection, which favors samples surrounded by consistently high-quality neighbors, contribute positively when applied independently. As shown in our response to W1, careful tuning of the hyperparameter $k$ further increases the performance gains of our method.

There is synergy between the two components. While combining both components results in improved performance compared to using either one alone, the gains are not strictly additive. We believe this is due to the synergy between the components, as they may target partially overlapping aspects of data quality. We will clarify this interpretation in the revised version and plan to investigate their interaction more thoroughly in future work.

W3: Applicability to larger models.

We thank the reviewer for this valuable suggestion. To demonstrate the scalability of our method, we follow Table 2 and apply our approach (with $k = 50$) to the Magpie dataset and fine-tune a larger model, Qwen-2.5-14B. Due to time limitations, we do not conduct further hyperparameter tuning. The evaluation results are shown below:

Our method consistently outperforms the strongest baselines across $\mu_{\text{open}}$, $\mu_{\text{LLM}}$, and $\mu_{\text{all}}$. Notably, it achieves an average improvement of at least 1.75 points on LLM-as-a-Judge benchmarks. These results suggest that our data selection method remains effective and beneficial for instruction tuning of models larger than 7-8B.

We will include these results in the revised version.

Dear reviewer WDFx,

We are glad that our rebuttal addressed your questions. Thank you for the time and effort you invested in reviewing our paper and engaging in the discussion! Your thoughtful feedback greatly helped us strengthen our work. We will incorporate the new results and clarifications into the revised version.