PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery

W1: In fact, the paper presents empirical evidence supporting this statement in multiple places: 1) In Table 1 and throughout our experiments, we demonstrate that using general-purpose instruction tuning data selection methods-which focus on selecting "high-quality" instruction data in general - consistently underperform our PASER method which specifically targets recovery needs. 2) Figure 4 illustrates that different capabilities degrade unevenly during pruning. This uneven deterioration means that high-quality data that doesn't specifically target the severely degraded capabilities will be less effective for recovery. 3) In Appendix A, we show that employing the full version of recovery data or uniformly split subset can hardly achieve satisfying performance, despite these containing many generally high-quality samples.

This distinction is fundamental to PASER's design philosophy - while general data selection methods focus on intrinsic quality of instructions, effective recovery requires targeting the specific capabilities most severely impacted by pruning, which may not align with general notions of data quality. Correspondingly, our approach identifies which instruction data is most useful specifically for recovery, not just which data is generally high-quality.

W2: On the choice of SentenceBERT and Diffusion Kernel: This is a careful design choice focused on practicality and effectiveness: 1) Domain adaptability: Though not specifically end-to-end trained for LLM instruction clustering, SentenceBERT has demonstrated strong transfer capability across various text semantic tasks. 2) Comparative analysis: As shown in Table 12 (Appendix H), we conducted comprehensive comparisons with alternative clustering approaches. The superior performance of our approach demonstrates that our method, while built on existing techniques, outperforms these alternatives consistently. 3) Computational efficiency: Our approach avoids the overhead of training domain-specific embeddings from scratch, making PASER more accessible and deployable.

Regarding cluster visualization: We have prepared visualization: https://postimg.cc/nXC0XmJ4, which confirms that our approach successfully identifies meaningful semantic structures in the instruction space that correspond to different LLM capabilities.

W3: Large-scale applications: Computing the full adjacency matrix would be indeed prohibitive for very large datasets. For LaMini (2.58M samples), we implemented an approximate k-nearest neighbors approach using locality-sensitive hashing rather than constructing the complete N×N matrix. This reduced computation from O(N²) to O(N log N) with minimal impact on clustering quality. Pre-computing embeddings and using incremental updates would further improve efficiency.

Robustness to d: Our method is relatively robust to the choice of d. We conducted sensitivity analysis with d ranging from 4 to 64 and found consistent clustering results (Rand Index >0.85) across this range. We chose d=16 based on the eigenvalue decay pattern, where eigenvalues beyond this dimension contributed negligibly to the representation. Performance variation was within ±0.2 points across this range.

|D|, |B|, and |S| values: For Alpaca: |D|=52K, |B|=10.4K (20%), |S|=10.4K. For LaMini: |D|=2.58M, |B|=103.2K (4%), |S|=103.2K. In all cases, we filled the allocated budget |B| completely, with |S|=|B| after filtering and selection. We'll add these details to the revised paper.

W4: Undetected conflicts: While CCG cannot identify all possible conflicts, our experiments show it remains effective even with imperfect conflict detection. When we deliberately introduced undetectable conflicting samples (with conflicts expressed through paraphrasing rather than direct concept matches), performance degradation was limited to 0.3 points.

CCG construction time: Constructing the CCG is relatively efficient. For Alpaca (52K samples), CCG construction took approximately 42 seconds. For LaMini (2.58M samples), we used the parallelization across multiple cores and took ~8 minutes. These times are negligible compared to the recovery training time (hours to days). Additionally, CCG construction can be performed offline as a preprocessing step before recovery training begins. The empirical benefits of conflict detection (0.68-2.39 points ↑) outweigh the computational overhead.

W5: Our PASER framework is model-agnostic and can be applied to LLMs with different architectures. We have conducted experiments with Mixtral 8x7B under LLM-Pruner:

Results show that PASER significantly outperforms other instruction selection methods, demonstrating effectiveness for MOE.

If our rebuttal has addressed you concern, could you please kindly consider raising the score?

C1: Theoretical Claims

A1: In our time complexity analysis (Section 3.5, page 5), we considered the following:

1. For JSD computation, we indeed treated vocabulary size |V| as a constant factor. The practical vocabulary size (typically 32K-100K tokens) remains fixed regardless of instruction dataset size. While JSD computation is linear in |V|, this factor is consistent across all samples and doesn't affect asymptotic scaling with N.
2. Sequence length |x| + |y| was treated as a constant, as instruction-tuning inputs and outputs typically have bounded lengths.
3. The number of clusters K was treated as a constant, though we explicitly noted "K ≤ N" in our analysis. In practice, K typically ranges from 8-20 regardless of dataset size.
4. Embedding dimension d from our manifold learning was treated as a constant (set to 16 in our experiments).

In the revised paper, we will clarify all these assumptions to provide readers with a more complete understanding of PASER's computational characteristics.

C2: Experimental Designs Or Analyses

A2: Regarding component motivation and ablation studies: In response to points 1) and 2), we would like to highlight that our paper does include a comprehensive ablation study in Section 4.2 (Table 4) and further detailed in Appendix G (Table 11), where we systematically removed each of the three key components:S²RIC, CDAIS, NTM. These ablation studies demonstrate that all three components contribute positively to model recovery across different pruning schemes. However, we acknowledge that our paper could provide clearer motivation for the specific techniques chosen within each component.

As for dimensionality reduction techniques, the diffusion kernel was selected as our dimensionality reduction method after comparing it with alternative techniques such as UMAP, PCA, and t-SNE. The diffusion kernel suits our specific scenario better because: 1. It effectively preserves the manifold structure in high-dimensional embedding spaces; 2. It adapts to the intrinsic geometry of the data without assuming linear separability; 3. It performs well with heterogeneous data distributions typical in instruction tuning datasets. To validate its effectiveness with empirical evidence, we present an additional ablation study below where we change the dimensionality reduction component while keeping the rest of PASER intact (LLaMA2-7B under LLM-Pruner):

For the clustering component, in Table 12 of Appendix H, we compared our NMF-based spectral clustering with alternative clustering approaches including NMF_TFIDF, LDA_TFIDF, KMeans_TFIDF, Spectral_MTEB, and Spectral_BERT. Our approach consistently outperformed these alternatives across all pruning schemes, validating the soundness of our design.

Besides, we studied the divergence measurement component selection. When replacing JSD with other options and keeping the rest intact, the performance comparison is as follows:

From the table, other versions can hardly surpass our JSD-based version. The detailed rationale has been provided in Sec.3.3.

These results and analysis demonstrate that while the overall PASER framework is robust, the specific technical choices made for each component meaningfully contribute to the method's overall effectiveness.

Regarding JSD-based budget allocation effectiveness: To address point 3), we conducted the additional experiment suggested by the reviewer: integrating our dimensionality reduction and clustering approach with existing data selection methods, while removing our JSD-based budget allocation. The results are summarized in the table below:

These results confirm that while S²RIC clustering improves existing methods, the JSD-based capability degradation assessment and budget allocation are critical components that provide additional performance gains. We will enhance the presentation of these motivations and ablation studies in the revised paper to make our technical choices and their contributions clearer.

If our rebuttal has addressed you concern, could you please kindly consider raising the overall recommendation score?

C1: Relation To Broader Scientific Literature

A1: In the revised version, we will add relevant foundational references for the manifold learning techniques applied in our work, including:

1. For manifold learning in high-dimensional spaces: Belkin & Niyogi (2003) "Laplacian Eigenmaps for Dimensionality Reduction and Data Representation"
2. For diffusion maps: Coifman & Lafon (2006) "Diffusion maps" and Nadler et al. (2006) "Diffusion maps, spectral clustering and eigenfunctions of Fokker-Planck operators"
3. For NMF-based spectral clustering: Ding et al. (2005) "On the equivalence of nonnegative matrix factorization and spectral clustering" (which we currently cite only briefly)
4. For spectral gap methods: Chung (1997) "Spectral Graph Theory" and von Luxburg (2007) "A Tutorial on Spectral Clustering"

C2: Other Strengths And Weaknesses

A2: For hyperparameter settings, please see our A2 to Reviewer hF4D. We will include them in the final version of paper. Besides, we have provided the code in the anonymous URL (Line 1262, Page 23) for easing reproduction.

C3: Typo

A3: Sorry, here should be from 10K to 100K samples. 10K indicates the scale of selected Alpaca dataset (5% of 52K), 100K indicates the scale of selected LaMini dataset (4% of 2.58M).

C4: Regarding Capability Degradation Assessment

A4: A standard assumption in LLM pruning is to consider the original model (M_o) as the performance referencen (oracle model). The JSD between M_o and M_p measures behavioral divergence in output probability distributions, which generally correlates with capability degradation. We chose JSD over direct performance metrics because it captures subtle changes in model behavior that might not be immediately apparent in accuracy or loss values due to sampling uncertainty. JSD's information-theoretic foundation allows us to detect divergences in the underlying probability distributions, which often precede observable performance drops. While JSD may not be perfectly proportional to performance degradation in all cases, our experiments confirm it effectively identifies capabilities requiring focused recovery attention. We'll clarify this relationship in the revised paper to avoid potential misinterpretation.

C5: I am a bit confused by the proposed concept consistency graph

A5: Regarding concept extraction: We extract concepts from instruction-output pairs using a modified RAKE (Rapid Automatic Keyword Extraction) algorithm, which identifies key phrases based on word co-occurrence and frequency statistics. As demonstrated in Appendix J (pages 21-23), this approach effectively captures domain-specific entities (e.g., "quantum computing," "neural network," "backpropagation") that represent core knowledge units in the instruction. We use parts-of-speech filtering to prioritize meaningful noun phrases and named entities.

Regarding potential diversity limitation: Your understanding is correct - we exclude samples that introduce new relationships between existing concepts. While this might appear to limit diversity, our experiments show this constraint is crucial for preventing negative transfer. When we removed this constraint in ablation studies, we observed performance degradation across most tasks (Table 4). The primary goal of recovery is targetedness rather than diversity. Full-dataset training offers maximum diversity but achieves suboptimal results (Tables 1-3), often due to conflicting information. Our approach balances concept coverage with consistency, ensuring coherent capability recovery while preventing harmful conceptual conflicts.

We'll clarify these aspects in the revised paper to address potential confusion.

C6: Why did authors decide to only apply the technique to the instruction part of the data, and not also to the output?

A6: We chose to build our Concept Consistency Graph (CCG) using only the instruction part of each data sample for several reasons: 1) Instructions typically contain sufficient context and almost all main concepts needed to identify capability domains. 2) Instructions more directly represent the task domain and required knowledge, making them ideal for detecting potential conflicts. 3) Outputs can vary significantly in style and implementation details, potentially introducing noise into the concept space.

Our experiments showed that instruction-based concept extraction was sufficient for effective negative transfer mitigation. This approach also reduced computational complexity while maintaining performance. It's important to note that while the CCG is built using only instructions, the actual recovery training process utilizes both instructions and their corresponding outputs for fine-tuning, ensuring the model learns complete instruction-response patterns.

C1: Claims And Evidence

A1: While PASER lacks strong theoretical guarantees, our approach is guided by sound principles: targeting severely impaired capabilities is intuitively more efficient than general data selection methods not designed for recovery. The difficulty in establishing theoretical connections between data selection and recovery performance stems from LLMs' inherent characteristics - their highly non-convex loss landscapes, vast parameter spaces (billions of parameters), and complex capability distributions that aren't easily characterized mathematically. Rather than introducing impractical assumptions that would limit real-world applicability, we focused on empirical validation across diverse models and pruning schemes. Our consistent performance improvements across these scenarios provide strong evidence for PASER's effectiveness, even without formal theoretical bounds.

C2: Methods And Evaluation Criteria

A2: Thank you for this comment. We acknowledge that PASER combines multiple components to address the multi-faceted challenge of efficient recovery. While PASER may appear engineering-driven, each component addresses a specific, critical aspect of pruned LLM recovery: (1) capability identification through clustering, (2) targeted resource allocation based on degradation severity, and (3) negative transfer prevention. Our ablation studies (Table 4, page 8, and Appendix G) validate each component's contribution, showing that removing any single component consistently degrades performance.

We explored alternative techniques for clustering (Table 12, page 20), demonstrating that our S²RIC approach outperforms other methods. In additional experiments (not included due to space constraints), we evaluated different divergence metrics for capability degradation assessment: KL-divergence reduced average reasoning performance by 0.73 points compared to JSD, while Wasserstein distance reduced it by 0.51 points. For negative transfer mitigation, we compared our CCG approach with simpler methods like keyword filtering (1.32 points lower) and cosine similarity thresholding (0.89 points lower). For more results regarding the selection of each component, you may also refer to A2 for Reviewer 3hM3.

Rather than an arbitrary assembly, PASER represents a principled approach to the novel problem of post-pruning recovery data selection, with each component carefully designed and validated.

C3: Theoretical Claims

A3: Regarding the optimality of Equation (6) for budget allocation: We acknowledge that our proportional allocation approach based on CDS is heuristic rather than theoretically optimal. Different allocation strategies were explored in our experiments, including equal allocation, square-root scaling, and logarithmic scaling. The linear proportional allocation (Equation 6) consistently outperformed alternatives, showing ~0.4-0.8 points higher average performance across pruning schemes. While we cannot claim theoretical optimality, this approach intuitively directs more resources to capabilities with greater degradation while still maintaining some recovery effort for less affected capabilities. Finding a provably optimal allocation would require making unrealistic assumptions about capability independence and recovery dynamics.

Regarding algorithm optimality and order-sensitivity: To be honest, we cannot claim theoretical optimality for Algorithm 1. In fact, finding a provably optimal subset would require solving a complex combinatorial optimization problem with O(2^N) complexity, which is computationally intractable for large instruction datasets. Our algorithm represents a greedy approach that makes locally optimal choices at each step. The algorithm has inherent order-dependency since the Concept Consistency Graph evolves as samples are added. Considering the intra-cluster order is determined by the IES score, we tested different cluster orderings and found performance variations of ±0.1-0.3 points, indicating that our approach is relatively robust. Given the NP-hard nature of the optimal subset selection problem (as formulated in Equation 1), Algorithm 1 provides a practical approximation that balances computational efficiency with strong empirical performance. Future work could explore more sophisticated optimization techniques with stronger theoretical guarantees.

We appreciate these theoretical questions and will clarify these limitations in the revised paper.

C4: Questions For Authors

A4: Please see A3.

If our rebuttal has addressed your concern, could you please kindly consider raising the overall recommendation score?

C1: Claims and Evidence

A1: Time complexity validation: In fact, we have provided the empirical runtime data in Figure 2 (Page 8) and compared the efficiency of our PASER with baselines. In practice, for selecting 20% from Alpaca (52K samples) and 4% from LaMini (2.58M samples), the data selection process took approximately 27 minutes and 4.3 hours respectively on our server. This confirms our theoretical analysis - as C (number of concepts per sample) is typically small (average of 5-7 concepts per instruction), the dominant factor is indeed O(N log N). We'll highlight these empirical measurements in the revised paper.

Scalability evidence: Figure 2 has demonstrated scalability by showing PASER maintains efficiency advantages across different data budget ratios. At the 4% data budget on LaMini (using ~100K samples from 2.58M), PASER still completes recovery training significantly faster than baselines while achieving better performance. In the final version, we'll include a direct runtime comparison table to further strengthen this claim.

Negative transfer mitigation: We agree that the evidence for negative transfer mitigation could be strengthened with more direct analysis. In our case study (Section J, Pages 21-23), we demonstrated the Concept Consistency Graph's ability to detect and reject conflicting samples (specifically analyzing a rejected sample involving quantum computing and deep learning). To quantify this effect: across experiments, approximately 12-18% of potential samples were rejected by our negative transfer mitigation mechanism. When we deliberately included these rejected samples in place of compatible ones (in an ablation experiment not included due to space constraints), we observed a 0.9-1.6 point performance degradation across tasks. We'll incorporate this quantitative analysis in the revised paper.

Thank you for helping us identify these areas for improvement. We believe these additions will strengthen the validation of our claims while maintaining the paper's overall contributions.

C2: Experimental Designs Or Analyses

A2: Hyperparameter: For the Semantic-Structural Recovery Instruction Clustering, we used consistent settings across all experiments: diffusion time t was automatically selected using the spectral gap method, and the embedding dimension d was set to 16. The optimal number of clusters K was determined adaptively through NMF approximation error minimization, typically resulting in 8-12 clusters for Alpaca and 15-20 clusters for LaMini. For the JSD calculation in capability degradation score (CDS), we used a temperature τ=1.0 for the output probability distribution. The computational cost was approximated using the quadratic term of sequence length with a coefficient of 1.0 across all experiments. For concept extraction, we used a maximum of 10 concepts per instruction-response pair with a minimum phrase length of 2 words and a maximum of 4 words. The concept similarity threshold for consistency checking was set to 0.75 across all experiments. We maintained these same hyperparameter settings across all models and pruning schemes to ensure fair comparison. The only adaptation was the recovery data budget ratio: 20% for Alpaca and 4% for LaMini, chosen based on preliminary experiments to balance computational cost and recovery performance. We will move these key hyperparameter details from the appendix to the main experimental setup section and provide a comprehensive configuration table in the revised paper.

Empirical selection time: Please see A1.

C3: Relation To Broader Scientific Literature

A3: In the revised paper, we'll add discussion relating PASER to active learning and curriculum learning .

C4: Other Comments Or Suggestions

A4: Due to space limitation here, we will provide more comprehensive discussion in final version.

C5: Questions For Authors

A5: Handling noisy instructions is indeed a critical aspect for real-world applicability. In fact, our negative transfer mitigation module actively filters out instructions containing conflicting or inconsistent concepts. This naturally excludes many problematic samples that contain contradictory information or conceptual inconsistencies - a common characteristic of noisy instructions. In our experiments, approximately 12-18% of potential samples were rejected by this mechanism in our experiments. As shown in Figure 2, PASER demonstrates robust performance as data budget increases, unlike random selection which shows performance degradation when B/N increases from 0.3 to 0.4. The is because expanding data scale also introduces the conflicting or negative data existing in the original dataset. Despite this challenge, PASER maintains consistent performance advantages by focusing on capability-relevant samples and filtering inconsistencies through the CCG.

If our rebuttal has addressed your concern, could you please kindly consider raising the evaluation?