SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuning

We’d like to thank the reviewer for the insightful and positive feedback. We are encouraged that the reviewer found our work meaningful, novel, and effective. For the thoughtful questions and constructive suggestions. We'd like to share our responses below.

W.1:

We appreciate the reviewer's observation regarding the efficiency of SHED compared to classical Shapley value calculation methods. In our paper, we provide a detailed analysis of SHED's computational complexity and actual runtime. As shown in Figure 3 (c), we present the computational time for Shapley value calculations across different cluster numbers. This graph clearly illustrates how SHED's runtime scales with the number of clusters, providing concrete evidence of its computational efficiency.

In contrast, for the classical Shapley value calculation method, the time complexity is exponential in the size of the dataset. Given our dataset size is about 100,000 instances, this results in a computational cost that is prohibitively large. As a result, it is not feasible to provide actual runtime measurements for the classical method on our dataset. However, we can analyze the efficiency of SHED compared with the classic Shapley value method under the experimental settings of this paper.

For the classical Shapley value calculation method with a dataset of |D| = 100,000 instances: We need to consider all possible subsets (2^|D|). For each subset S, we fine-tune the model (taking time |S|t, where |S| is the size of the subset) and evaluate it on the test set (taking time T_m). Therefore, the time complexity would be: O(2^|D| * (|D|t/2 + T_m)) = O(2^100,000 * (50,000t + T_m)) This is computationally infeasible.

In contrast, SHED's complexity, as detailed in our paper’s discussion section, is: O((Ck/n)[(C+n)t/2 + T_m]) = O(500 * [1530t + T_m]). This demonstrates a significant reduction in computational complexity compared to the classical method.

W.2:

We appreciate the reviewer's insightful observation regarding the potential limitation of our clustering approach in SHED.

Our use of clustering and proxy data points is a trade-off we make to significantly reduce computational complexity while still capturing the overall data distribution effectively.

However, we have taken steps to mitigate the overlooking issue. The Quality-Weighted Cluster Sampling (QWCS) variant of our method allows for more diverse sampling. As shown in our experiments, despite this potential limitation, SHED still outperforms other methods across various tasks, suggesting that the benefits of our approach outweigh this drawback in practice.

Nevertheless, we agree that this is an important area for improvement. In our future work, we plan to explore more sophisticated clustering algorithms and sampling strategies that could better preserve rare but important samples. We thank the reviewer for highlighting this important aspect, which will help us improve our method.

Q.1:

We appreciate the reviewer's question about evaluating SHED on industry-level datasets. We'd like to address this point:

• Scalability: SHED's design, particularly its use of clustering and proxy data points, makes it inherently scalable to larger datasets. By setting the ratio C/n to a fixed value, as done in the experimental setup, the computational complexity of SHED grows linearly with the number of clusters, not exponentially with the dataset size like traditional Shapley value calculations.

• Current Limitations: Our current experiments were constrained by the computational resources available to us in an academic setting. Our current computational resources are limited and insufficient to conduct experiments on industry-scale datasets within a short timeframe.

In the future, we plan to rent additional server resources to apply SHED on much larger datasets. We are committed to open-sourcing the curated datasets from these experiments, contributing valuable resources to the research community. These efforts will further validate SHED's effectiveness at industrial scales and support broader research.

We thank the reviewer for this suggestion, allowing us to emphasize our commitment to contributing to the research community despite current resource constraints.

Q.2:

We appreciate the reviewer's insightful question about the impact of initial clustering on SHED's performance. The initial clustering step in SHED plays a significant role as it determines the proxy data points used for Shapley value calculation. Our method employs several strategies to ensure robustness to variations in initial clustering:

• Semantic-preserving embeddings: We use Sentence Transformers to generate embeddings, which effectively capture semantic similarities across various domains. This helps ensure that semantically similar data points are likely to be clustered together, even if the exact cluster boundaries may vary.

• Multiple initializations: In fact, we conducted multiple experiments with different random initializations to verify the robustness. We found that the number of clusters is a factor that affects the performance. Therefore, we conducted a sensitive analysis of the number of clusters. Our sensitivity analysis (Figure 3) shows that performance improvements plateau when the number of clusters exceeds 3√|D|. This suggests that as long as we choose a sufficiently large number of clusters, the exact choice is less critical.

• Shapley value calculation: This step considers the contribution of each proxy data point across multiple subsets, which helps to average out some of the variability introduced by clustering.

• Optimization-aware sampling: The final selection step considers the Shapley values of all clusters, providing another layer of robustness against individual cluster variations.

We thank the reviewer for this valuable question, which helps us clarify the robustness of SHED.

We thank the reviewer for the valuable feedback.

W.1:

We appreciate the reviewer's concern but note a possible misunderstanding about SHED.

SHED does consider the combinatorial effects between clusters:

• Combinatorial Effects: SHED calculates Shapley Value (SV) for cluster proxies, reducing computational cost while capturing inter-cluster relationships. Each cluster is represented by a proxy data point (closest to the centroid). SVs are calculated by considering possible combinations of these proxies, thus considering interactions between clusters. SV compresses the combinatorial space into a single score, encapsulating individual importance, synergies, and potential redundancies.

• Score-based Selection: The selection step uses these scores, which already encapsulate combinatorial effects. A high SV indicates a high likelihood of positive impact when combined with instances from other clusters.

Grid search for data selection incurs high costs. If there are C clusters and sampling ratio takes x values, with training time T. Grid search’s complexity is O(Tx^C), scaling exponentially with C, which is impractical.

In contrast, Shed’s complexity is O((Ck/n)[(C+n)t/2 + T_m]). As Figure 3 shows, SHED can scale linearly with the number of clusters by setting C/n to a fixed value, making it feasible for large datasets. Thus, SHED offers a significant computational advantage over grid search.

W.2:

We appreciate the reviewer’s concern. To address this, we compared SHED with LIMA and Cherry-LLM:

LIMA:

• MMLU: SHED (1k) - 41.71 vs. LIMA (1k) - 34.9
• ARC: SHED (1k) - 48.12 vs. LIMA (1k) - 46.16
• Note: SHED is fully automated, unlike LIMA’s manual curation.

Cherry-LLM (using WizardLM):

• MMLU: SHED (1k) - 33.62, SHED (7k) - 35.63 vs. Cherry (7k) - 33.08
• ARC: SHED (1k) - 51.36, SHED (7k) - 50.11 vs. Cherry (7k) - 52.90

SHED achieves comparable or better performance with less data. Since SHED used MMLU as test set during data selection, its performance on ARC could be improved if used as test set during selection, showing SHED's effectiveness for task-specific data selection.

Baseline Criteria:

We use DSIR because it, like SHED, selects data by importance. DSIR samples data based on estimated importance, similar to SHED's use of SVs. We included DQ as it was also used for fine-tuning in its paper.

We avoided methods requiring human or LLM evaluations due to high costs.

W.3:

We appreciate the reviewer’s concern about SHED's computational cost and would like to clarify:

Transferability & Cost Amortization:

• One-time Computation for Multiple Uses: SHED-refined datasets perform well across various model sizes and architectures. This transferability makes them reusable for different models or tasks. This one-time computational investment ensures long-term efficiency, reducing the need for repeated data selection. SHED's amortized cost over several models significantly lowers the per-use cost.

• Comparison with Non-transferable Methods: Unlike methods requiring computation for each new model or task, SHED's transferability eliminates recurrent computational expenses.

• Efficiency Gains: SHED-refined datasets lead to comparable or superior model performance with smaller datasets, directly translating to lower computational and time costs in the fine-tuning phase.

Efficiency Design: SHED reduces the computational cost by approximating SVs for cluster proxies instead of individual data points. Our analysis and experiments show its complexity can grow linearly with the number of clusters, not exponentially with dataset size.

Benchmark: Figure 3 (c) has shown the time for SV calculations across different numbers of clusters, which also indicates SHED’s scalability.

W.4:

We appreciate the reviewer’s comment but believe there may be a misunderstanding of our claims:

• Clarification of Objectives: As highlighted in our paper, SHED focuses on fine-tuning. However, Figure 3 (c) shows linear growth in time, suggesting potential for large-scale pre-training data selection. SHED is adaptable to various fine-tuning objectives, such as task-specific performance, model fairness, and domain focus, by modifying the value function in SV. This adaptability is intended within the fine-tuning context.

• Not Pure Cluster-Level Selection: While SHED uses clustering to reduce computational complexity, it does not solely rely on cluster-level selection. Instead, it considers individual samples within clusters. The final selection step, QWCS, allows for the selection of individual samples.

• Embeddings: We use Sentence-Transformer, which is well-suited for NLP tasks and produces semantically meaningful embeddings. Trained on a large and diverse corpus, these embeddings are robust across various NLP tasks. SHED's framework can adapt to different tasks with suitable embeddings.

W.5:

We emphasized SHED's advantages over TS-DSHAPLEY in Section 2.3, including lower computational overhead, transferability, flexible framework, and data diversity.

The unavailability of TS-DSHAPLEY code and datasets, along with significant computational demands, pose challenges for comparison. Nonetheless, the results presented in our paper demonstrate the superiority of SHED:

• Efficiency: Figure 3 (c) shows SHED's computational cost scales linearly with the number of clusters. While TS-DSHAPLEY's cost scales with data size.

• Transferability: Tables 6, and 7 show SHED’s robust performance across different models, underscoring its transferability. TS-DSHAPLEY lacks evidence of effectiveness across varied models.

• Human alignment: Table 5 shows SHED-refined datasets not only enhance accuracy but also align well with human preferences. TS-DSHAPLEY does not evaluate human preference alignment, leaving it unclear whether its selected datasets improve the model’s ability to follow human instructions.

Thank you for carefully reading our rebuttal and providing your feedback. We greatly appreciate your recognition that our paper does not contain any technical flaws. However, we are concerned about the score of 3, which is typically reserved for papers with significant technical issues or weak evaluations. Since you acknowledged there are no critical flaws, a score of 3 seems inconsistent with the paper's merit.

We would like to further clarify the distinctions and advantages of SHED compared to TS-DSHAPLEY: TS-DSHAPLEY randomly samples multiple subsets of the training data, estimates the contributions of data points in these subsets, and then repeats the process multiple times and aggregates the results to estimate the Shapley values for the entire training set. Notably, this method will calculate the Shapley value for all instances in the training set.

Advantages of SHED:

• Improved computational efficiency: SHED only computes Shapley values for representative proxy data samples selected from clusters, rather than for each individual data point as in TS-DSHAPLEY. This dramatically reduces the computational overhead, making SHED more scalable to large datasets.

• Model-agnostic data representation: SHED uses model-agnostic sentence embeddings (e.g., from Sentence Transformers) to represent data samples, while TS-DSHAPLEY relies on representations extracted from the target language model. The model-agnostic approach enhances the transferability of the curated datasets across different language models, amortizing the computational cost of data selection. Table 7 showcases the transfer performance of SHED-curated datasets across different models. SHED is trained on the MMLU and WizardLM datasets using the LLaMA-7B model, and the resulting optimal subsets are used to fine-tune various models, including LLaMA-13B, Vicuna-7B, and GPT-2. The results demonstrate that SHED-constructed datasets exhibit robust performance across a range of models, confirming their applicability across models and even different model families. This strongly supports the transferability of SHED datasets.

• Clustering-based data sampling: SHED employs clustering algorithms to group similar data samples and selects representative proxy data for each cluster. This helps capture the diversity and complexity of the original dataset while reducing redundancy. TS-DSHAPLEY does not explicitly consider data diversity in its sampling process.

• Flexible optimization objectives: SHED's value function in the Shapley value calculation can be customized for various optimization objectives, such as accuracy and fairness, allowing the curated datasets to align with task-specific requirements. TS-DSHAPLEY focuses primarily on model predictive accuracy.

• Unified and extensible framework: SHED presents a unified framework that integrates clustering, Shapley value computation, and data sampling, with the flexibility to accommodate different clustering algorithms, optimization objectives, and sampling strategies. This makes SHED more adaptable to various scenarios than TS-DSHAPLEY.

We hope this clarification helps to further understand the unique contributions and strengths of SHED, and we kindly request reconsideration of the score in light of this information. The existence of a related but less effective method should not be grounds for rejection.

Thank you once again for your feedback and consideration.

We thank the reviewer for the valuable feedback. We are encouraged by the positive comments on our well-motivated, clearly presented, and highly effective work. Below are our responses to the thoughtful questions and constructive suggestions:

W.1:

We sincerely thank the reviewer for highlighting this issue and the reference to DsDm. We acknowledge that our original manuscript overlooked DsDm, which shares similarities with our approach in using the impact of data points for selection. However, like other works, DsDm does not consider the combinatorial effects of data points in the same way that SHED does. The key differences are:

• Individual Impact: DsDm primarily focuses on estimating the impact of individual data points on model performance. It uses data models to approximate how each training example affects the model's predictions on target tasks.

• Linear Approximation: DsDm's data models are typically implemented as linear functions, which implicitly assume that each data point's contribution is independent of other points in the dataset.

In contrast, SHED's use of Shapley values allows it to consider the marginal contribution of data points, thereby capturing potential interaction effects between data points.

We will revise our manuscript to include a detailed discussion of DsDm and clearly articulate SHED's unique contributions. We appreciate this feedback, which will significantly improve our paper's comprehensiveness and scholarly value.

W.2:

We appreciate the reviewer's observation regarding SHED's use of proxy data points. Indeed, the effectiveness of this approach is influenced by the quality of embeddings and clustering parameters. We'd like to address these points:

• Sensitivity Analysis: As detailed in our paper, we have already conducted a sensitivity analysis on the number of clusters. Figure 3 demonstrates how performance and computational time vary with different cluster numbers. This analysis shows that performance improvements plateau when the number of clusters exceeds 3√|D|, providing empirical guidance for choosing this parameter.

• Embedding Quality: We use Sentence Transformers for generating embeddings, which are known for their ability to capture semantic similarities between sentences. By clustering on these embeddings, we group semantically similar text, which helps in discovering and organizing potential patterns and relationships in the text data, enhancing the representativeness of our proxy points.

While our current approach has shown robust performance, we acknowledge the importance of this aspect. In our future work, we plan to further investigate the impact of different embedding techniques and clustering methods on SHED's performance. We thank the reviewer for this valuable feedback, which will help us improve our work.

W.3:

We appreciate the reviewer's insightful comment on our experimental methodology. We'd like to clarify our approach and reasoning:

Fairness in Comparison and Focus on Data Impact: Our decision to use the same hyperparameters across all methods was deliberate, aiming to ensure a fair comparison that isolates the impact of data selection on model performance. This approach allows us to directly observe how different data selection methods affect performance without the confounding influence of varying hyperparameters.

We acknowledge that our current approach of using fixed hyperparameters across all methods may not fully capture the optimal performance of each method.

To address this concern, we propose the following improvements for our revised manuscript:

• Hyperparameter Sweep: We will conduct a hyperparameter sweep for each method, including learning rate, batch size, and number of epochs.

• Best Configuration Reporting: We will report results on the test set using the best hyperparameters found for each method during the sweep.

We believe these additions will strengthen our experimental methodology. We thank the reviewer for this suggestion, which will enhance the reliability of our results.

W.4:

We appreciate the reviewer's observation regarding the varying data sizes used in our experiments. We acknowledge that this may have caused some confusion in interpreting our results. To address this concern:

• Comprehensive Comparison: To ensure a fair comparison, we have included results for a fixed data size of 10k samples across all methods in Table 2. Additionally, as detailed in Appendix B of our paper, we have conducted experiments comparing different methods across various sample sizes. These provide a comparison of method performance when using identical amounts of data.

• Method-Specific Optimization: In Appendix B, we provide comprehensive comparisons of different methods across various sample sizes. Our analysis reveals that QOCS often performs better with smaller subsets. This is likely because it prioritizes the highest quality, most informative data, making it particularly effective when working with limited data. On the other hand, QWCS often benefits from slightly larger subsets. This method balances quality with diversity, allowing for the inclusion of a broader range of data patterns. As a result, QWCS can capture more comprehensive representations of the dataset. These characteristics explain the different optimal subset sizes for each method. By presenting the best-performing subsets in the main text, we aim to demonstrate the full potential of each approach.

We will revise our manuscript to more clearly explain our rationale for presenting the best-case scenarios and to direct readers to the appendix for these fixed-size comparisons.

We thank the reviewer for this feedback, which will help us improve the clarity of our results presentation.

Q1:

Please see the responses R.3 and R.4. We have also presented all the training hyperparameters in Appendix C.

You can find the code and dataset here: https://github.com/Lucidreamer9/SHED-Shapley-Based-Automated-Dataset-Refinement