Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization

W1: A potential limitation is the calibration data curation is not fundamentally different from Pretraining or SFT data curation.

A1: We thank the reviewer for raising this insightful point. Here, we would like to clarify why calibration data curation in post-training compression differs from pretraining or SFT data curation, which can also be found and is empirically supported in our paper:

1. Objective and Stage Difference: Pretraining and SFT data aim to teach the model capabilities from scratch or to align them with specific tasks, and occur during the model training stage. In contrast, calibration data is used in a post-training setting, where the model's capabilities are already learned, and the goal is to preserve these capabilities during compression (e.g., pruning or quantization). This is particularly crucial because post-training methods do not update weights via gradient descent, making the data-dependent preservation effect more sensitive, requiring more careful calibration data selection.

2. Scale and Constraints: Pretraining/SFT typically leverage large-scale corpora (e.g., hundreds of GBs~TBs) to improve generalization. In comparison, calibration data is extremely limited in size (often <1K samples) and must be carefully curated to maximize activation representativeness and diversity, which we show are critical for effective capability preservation.

3. Mechanistic Distinction: As detailed in Sections 3–4 and Appendix E, we demonstrate that the utility of calibration data lies in its ability to activate critical patterns in the model's learned parameter space. This is distinct from the information distribution coverage goals of pretraining/SFT data, which is less related to the model itself. We further formalize and operationalize this insight in our proposed COLA framework, which systematically selects calibration samples based on activation-space clustering, a process not applicable or meaningful in the pretraining phase considering its initialization from scratch.

4. Empirical Evidence: Our extensive experiments (see Table 2 and Figure 2–6) show that even small changes in calibration data (e.g., format, domain, language) can lead to larger drop than changing compression method on specific capabilities, which is rarely observed in pretraining or SFT scenarios. This further reinforces that calibration data curation principles should be capability-specific and post-training aware.

To further illustrate the fundamental distinction, we provide the following analogy: Calibration data is like a "diagnostic test", whose purpose is to evaluate whether the model’s internal functions and capabilities in concerned domains are preserved after compression, by systematically triggering a wide range of activation patterns. This, in turn, guides compression methods to focus on preserving the most vulnerable or degraded capabilities (large activation discrepancy between original and compressed models). In contrast, pretraining and SFT data resemble a "textbook", designed to teach the model new knowledge or skills, or to adjust its behavior. Accordingly, good calibration data focuses on diverse, clean examples that elicit different model capabilities, emphasizing quality over quantity. Meanwhile, good pretraining/SFT data requires both scale and coverage, emphasizing quantity plus quality to ensure robust generalization and task performance. For example, for enhancing model math capabilities with SFT, we may need 10K+ samples. However, to preserve model learned math capabilities during post-training compression, we only need to ensure selected calibration data samples covering each subcategory of math, thus about hundreds of samples are enough. This difference in purpose and design principle further supports the view that calibration data curation is a unique and distinct challenge in the LLM lifecycle.

We thus believe that our work offers the first principled and mechanistic exploration of calibration data optimality in LLM compression, clearly distinguishing it from general data curation practices in model training.

Q1: What are the fundamental differences between the calibration data curation method proposed in this paper and pre-training or SFT data curation?

A2: See the response in A1.

Hope our above explanation can effectively address your concern. If satisfied with our response, could you please kindly consider raising the score?

Dear Reviewer qoua,

We sincerely appreciate your time and effort in reviewing our submission. As the discussion phase will end in 2 days, we would be very grateful if you could kindly take a moment to share your thoughts on our rebuttal and let us know if any concerns remain unresolved.

Your feedback would be extremely helpful for ensuring a fair and constructive evaluation.

Thank you again for your consideration!

Best regards,

The Authors

Dear Reviewer qoua,

Considering the author-reviewer discussion deadline is approaching very soon (only one day left), could you please kindly consider our rebuttal and potentially raise the score?

Your feedback would be extremely helpful for ensuring a fair and constructive evaluation.

Thank you again for your consideration!

Best regards,

The Authors

W1: Generality: given the hundreds of pruning and quantization methods in existence, ... would make the paper stronger.

A1: Thanks for your insighful comment. When preparing the empirical analysis in this paper, we have carefully selected the most common/representative LLM compression approaches. Their citation numbers are as follows: SparseGPT (800+), Wanda (700+), GPTQ (1500+), and AWQ (1100+). Besides, we have explored the generality of our method in Appendix H (Scalability to Larger Language Models) and I (Generalizability to Other LLM Compression Methods). Specifically, we explored whether our proposed COLA calibration data framework can help improve the LLM performance compressed by more recent semi-structured pruning method RIA, more recent quantization methods SmoothQuant (1200+ citations) and FlatQuant, and also the representative structured pruning method LLM-Pruner (800+ citations and 1K+ Github stars). From the results shown in Table 10, we can observe our COLA framework consistently outperforms baseline calibration approaches across all evaluated compression methods for both LLaMA3-8B and Qwen2.5-7B models. The performance improvements are particularly notable for pruning methods like LLM-Pruner, where our approach achieves 44.65% on commonsense reasoning tasks for LLaMA3-8B compared to 42.35% with WikiText random sampling, representing a 2.30 percentage point improvement (See Line 760-765).

To further alleviate reviewer's concern, we supplement the experiments on a recent layer pruning method: Streamlining Redundant Layers to Compress Large Language Models (ICLR2025 Spotlight). We follow the same evaluation setting in Table 10 and the results are as follows:

LLM: LLaMA3-8B

LLM: Qwen2.5-7B

From such two tables, we can find that even under strong layer pruning method, our COLA framework can also help achieve relatively better pruning performance than other calibration data curation baselines. Thus, base on above these evidence, we can at least draw a preliminary conclusion: our proposed COLA framework which grounds on our empirical observations and insights can generalize well to other structured/layer pruning and more recent quantization methods. Indeed, our COLA framework design is motivated by experiments in Section 3, but definitely not limited to compression settings there. Hope our these analysis can help reduce your concerns regarding the generality. We will add these results to the final version paper.

W2: Also somewhat related to generality, the calibration data curation framework described in Section 4.2 seems to largely rely on ... will translate to these new datasets (stage 2).

A2: We thank the reviewer for the insightful feedback regarding the generality and automation of our calibration data curation framework. We respectfully clarify the following points:

1. On Stage 1 – Dataset Selection and Domain Correlation: While Stage 1 does incorporate domain knowledge (e.g., selecting MathQA for math capability), our framework is not restricted to hard-coded, manually defined mappings. In practice, the capability-domain correlation can be inferred using lightweight proxy evaluations. For instance, a user can evaluate a small subset of target benchmark tasks using compressed models calibrated on each candidate dataset, measuring capability preservation scores as a proxy for correlation. This approach is efficient (since only small calibration subsets are needed) and has been implicitly validated by our domain correspondence experiments in Section 3.3, where relative improvements across domains are consistently observed even with 128-sample subsets. Moreover, we formulate this stage as an optimization problem (Equation 1 in Line 267), where coverage(S, c) can be instantiated with either empirical proxy scores or metadata-derived similarity (e.g., topic tags, lexical overlap, or data embeddings). While our current paper focuses on well-known datasets for clarity and reproducibility, our formulation naturally supports generalization to unseen sources.

2. On Stage 2 – Sequence Length and Sample Size Transferability: Our observations on sequence length and sample amount in Section 3.2, though empirically grounded in selected datasets, are not arbitrary heuristics. They stem from trends consistent across models (LLaMA3, Qwen2.5), methods (AWQ, GPTQ, SparseGPT, Wanda), and diverse capabilities (math, commonsense, code, etc.). Notably, we observe clear diminishing returns for sample sizes beyond 128 (Figure 3), and a saturation point for sequence lengths beyond 1024–2048 tokens (Figure 2), across all methods. Therefore, these are not dataset-specific phenomenon but indicative of underlying activation space saturation, which is further substantiated in our spectral analysis in Appendix E. This mechanism-driven observation offers our heuristics strong transferability to new datasets, especially when models or compression methods remain similar. In fact, we have validated this transferability in Appendix F, where COLA is applied to domain-specific datasets (CommonsenseQA, MathQA, CodeQA) that are different from the pretraining datasets (WikiText, C4, SlimPajama) used for our sequence length and sample size studies in Section 3.2. Despite this shift in dataset types and contents, COLA still outperforms the SOTA calibration curation baseline (Self-Gen) across all compression methods and capabilities. This empirical result provides strong evidence that our observations regarding optimal sequence length and sample count generalize to new datasets, supporting the practical robustness of our Stage 2 design.

3. On Automation Potential: While we acknowledge that parts of Stage 1 and Stage 2 currently involve empirical decision-making, our framework already includes automation in Stage 3 (Sample Selection). We believe extending automation to earlier stages is a promising future direction, e.g., via dataset embedding similarity or automatic reasoning-difficulty labeling (as explored in Appendix D).

In summary, although Stage 1 involves initial domain reasoning, our formulation allows empirical proxy estimation for domain alignment, and the compositional guidelines in Stage 2 are grounded in generalizable trends tied to activation diversity and representativeness, not dataset-specific heuristics. We will make these clarifications more explicit in the camera-ready version.

W3: Writing: while the paper's main ideas have been expressed with sufficient clarity, the writing overall can be improved; in particular, there are several grammatical errors throughout the paper that need correction.

A3: Thanks for your appreciation to expression clarity. We will correct such grammatical errors and improve the overall writing in the final version paper.

Q1: Please provide a brief response to the weaknesses listed above (relating to generality).

A4: Thanks for your comment. Please see our response in A1 and A2.

Q2: What is the proportion of each dataset selected by the framework described in Section 4.2 for each compression technique in Table 2?

A5: Thank you for this question. We provide the detailed breakdown of dataset proportions selected by our COLA framework for each compression method in Table 2:

From this table, we can find that each compression method shows different sensitivities to various capabilities: 1. Higher C4 proportion for quantization methods (GPTQ: 44%, AWQ: 45%): Quantization methods are particularly vulnerable to code generation degradation due to precision loss in complex reasoning patterns, requiring more C4 data to guide optimal quantization parameter selection for preserving code synthesis capabilities. 2. Balanced distribution for pruning methods with slight WikiText emphasis: Pruning methods affect all capabilities more uniformly by directly removing weights, necessitating balanced calibration coverage with slight WikiText emphasis to maintain fundamental language modeling stability during weight elimination.

These proportions were validated through ablation studies showing that deviation from these ratios by ±10% results in non-negligible averaged 0.7-1.2 percentage point performance drops across key capabilities. We will include this detailed breakdown table and corresponding ablation analysis in the camera-ready version.

Q3: Figure 1 could benefit from a more detailed caption.

A6: Thanks for your suggestion. We provide our revised caption for Figure 1 as follows: "Calibration data in overall LLM compression pipeline. Most recent research focuses on developing advanced compression strategies, while neglecting to explore the impact of calibration data on compression performance and further what constitutes optimal calibration data.".

Q4: Figure 5: line graph not recommended here since it's not showing a trend.

A7: Thanks for suggestion. The bar chart here is a better scheme for line graph. Considering the rebuttal policy does not allow any external link, we will add the revised figure in the final version paper.

Hope our above response may successfully address your concerns. We would be sincerely grateful if you could consider raising the score if possible.

Dear Reviewer jQ8K,

We sincerely thank you for your appreciation of our work and for raising your score. We are also very grateful for your insightful suggestion regarding implementing the lightweight proxy evaluation idea mentioned in our rebuttal, which can further automate and strengthen Stage 1 (capability–domain correlation) in our COLA framework.

---

1. Reliability of Proxy-based Coverage Estimation
Motivated by your feedback, we first validated whether proxy evaluation can reliably estimate capability–domain correlation. Specifically, for each candidate calibration dataset, we selected only 32 samples to calibrate a compressed model and evaluated it on a small set of representative benchmarks (one per capability: BoolQ for commonsense, GSM8K for math, HumanEval for code). The relative performance scores obtained from these proxy evaluations were then used as the coverage(S, c) term in Equation (1) of Section 4.2. To verify the reliability of these proxy-based coverage values, we measured their correlation with the coverage values obtained from full evaluations (average correlation coefficients between proxy scores and full evaluation scores over 5 runs), as shown in the table below.

These high correlations confirm that proxy evaluation provides a reliable and efficient estimate of capability–domain correspondence.

---

2. End-to-End COLA Performance with Proxy-based Coverage
To further validate its practicality, we replaced the full evaluation coverage in Stage 1 of COLA with the proxy-based coverage values, then ran the complete COLA pipeline under the same experimental setup as Table 2 of the paper. Due to time limitation, we only conduct experiments on LLaMA3-8B. Below are representative results:

The proxy-based COLA achieves nearly identical performance to the original COLA (average gap ≤ 0.2 percentage points), while significantly reducing Stage 1 computation cost and avoiding manual heuristics.

---

Conclusion
These results demonstrate that:

1. Proxy evaluation can accurately estimate capability–domain correlation.
2. When integrated into COLA, it preserves almost all performance benefits of the full evaluation version.
3. This enhancement greatly improves the automation and scalability of our framework for unseen datasets.

We truly appreciate your constructive input, which has helped us strengthen both the completeness and practicality of our method. We will incorporate these new experiments and results into the revised version to further improve the paper’s quality.

Thank you again for your time, effort, and valuable guidance. Although the author–reviewer discussion is about to conclude, we hope that the additional experimental results provided here can further increase your satisfaction with our work and receive your corresponding support during the reviewer–AC discussion.

Best regards,
The Authors

A1: Thanks for your comment. We would like to clarify that we did not report the compressed model’s performance after further SFT because most of existing LLM compression works follow the post-training manner, which assumes the compressed model is directly usable without any further SFT. Here, to alleviate reviewer's concern, we conduct instruction finetuning to various compressed models with Alpaca dataset, for evaluating if our COLA can still effectively help compressed models keep abilities even under domain drift. Due to time limitation, we only perform experiments on LLaMA3-8B.

AWQ (4 bit)

SparseGPT (50%)

Wanda (4:8)

GPTQ (4 bit)

From above results, we can notice that though further instruction finetuning with Alpaca can help recover compressed model capabilities to some extent, our COLA can still ensure the better capability preservation effect than other calibration data curation baselines even after futher SFT. We will add these results and analysis to the final version paper.

W2: The sample selection step (random projection + K-means) adds compute and storage cost. The paper does not give any analysis on this overhead.

A2: Thanks for your comment. We would like to clarify that our use of random projection and K-means is motivated by their computational efficiency, which makes them more suitable for our activation-space sample selection than other alternatives, as clarified in Line 285-286. To further alleviate the reviewer's concerns on compute and storage cost, we provide the statistical results as follows. Note these results are averaged over four studied compression methods in the main text, corresponding to the general deployment scenario in Table 2.

Time Consumption

Peak Memory Usage

These overheads remain modest, especially considering they are one-time offline costs and only required for curating a compact calibration set. Moreover, they are significantly lower than the compute/memory overhead of any training-involved compression methods. We will supplement more detailed efficiency analysis regarding the overall calibration data selection process in the appendix of final version paper.

We honestly hope above explanation and clarification can help reduce your concerns. We would be sincerely grateful if you could consider raising your score accordingly.

Dear Reviewer 1Hx7,

Considering the author-reviewer discussion deadline is approaching very soon (only one day left), could you please kindly consider our rebuttal and potentially raise the score?

Your feedback would be extremely helpful for ensuring a fair and constructive evaluation.

Thank you again for your consideration!

Best regards,

The Authors

W1: One of the major weakness on paper presenting is that, please add baseline performance of each model before applied to any compression methods. This will set a better comparison for demonstration.

A1: Thanks for your comment. We would like to clarify that the reasons why we did not show the model performance before compression are two folds: 1) Our focus in this paper is the impacts of different calibration data variations on the compressed model performance, so presenting the pre-compression performance is relatively less important; 2) The performance difference between original and compressed models is generally more obvious than that among various compressed models. So presenting such pre-compression performance in figures may hinder highlighting the impacts brought by the calibration data variations.

Though, to present a more comprehensive comparison, we agree it's helpful to add such pre-compression performance into figures and tables. As rebuttal phase does not allow providing external links or revising paper, we provide the results as follows:

W2: The paper seems doing a lot of research on the data features in the first part of the paper and in appendix. But they seem separated from the method proposed, for example language domain. And on length and sample size, the maximum length studied is 2048, and the max sample size is 256. In my opinion, under the current trend of long CoT LLM, 2048 is a context length far from enough, and all the base model used in the paper can support upto 32k context. And sample size, I think the author should study much larger sample size, which I believe would make a difference as we alway observed during development, for example, the author can try a stride of 2^3 on sample size instead of 2^1. This is basically the main drawback of the paper.

A2: We appreciate the reviewer’s thoughtful feedback and would like to clarify that our empirical studies on calibration data features are tightly coupled with the design of our proposed COLA framework (Section 4). As shown in Section 4.1(Discussion on Calibration Data Optimality for Capability Preservation), we discussed the empirical observations from above extensive experiments (Section 3) and conducted following deeper analysis on calibration data optimality for capability preservation. Based on these, we proposed the COLA calibration data curation framework in Section 4.2. For example:

1. In Line 238 of Section 4.1, we highlight a key insight: "matching calibration data language to the deployment language is crucial for multilingual applications". This finding directly informs our Stage 1 (Domain Correspondence) in COLA (Section 4.2), where we explicitly state: “Language alignment is crucial... calibration data in each target language should be included.”

2. Similarly, in Stage 2 (Compositional Properties), we incorporate insights from our experiments on sequence length and sample size, especially the diminishing returns after 128 samples and 2048 tokens (as summarized in Line 234-236 of Section 4.1), which justify our practical choices and design constraints.

Regarding the reviewer's suggestion on extending context length and sample size:

On Context Length: While we acknowledge that recent LLMs support context lengths of up to 32k, most existing LLM compression works, including AWQ and Wanda, operate within a calibration sequence length of 2048 or shorter. Our choice of 2048 tokens aligns with this established setup and is sufficient to observe meaningful performance trends. Importantly, as shown in Figure 2, performance gains often plateau or even degrade beyond this length for certain tasks and methods (e.g., non-monotonic patterns in code generation with AWQ), suggesting that longer calibration data may not yield additional benefit and can even introduce noise. In fact, 2048 length has been enough for accommodating reasoning chains for many normal questions.

On Sample Size: Similar to above, max sample size of 256 is the common setting is previous works like AWQ and Wanda. Besides, as detailed in Figure 3, we observe diminishing returns beyond 128 samples for most capabilities. Larger sample sizes not only increase computational cost but can introduce variance that hinders capability preservation, especially evident in AWQ and GPTQ where performance occasionally drops as more samples are added. Thus, we focus on the range of 16 to 256 samples to balance informativeness and practicality.

To further validate such points, we rent some cloud GPU resources to conduct the experiments with longer context and larger sample size. Due to time and resource limitation, we only run AWQ and the results are as follows (K=1024).

From tables above, we may find further increasing context length/sample size does not bring abvious additional benefit at least for our evaluation benchmarks. In fact, it may even result in slight performance drop and significant computation overhead, as discussed above.

In summary, the empirical insights and method design in our paper are highly interdependent, and the chosen ranges for length and sample size are not only consistent with prior works but also empirically validated for trend saturation and practical feasibility.

W3: On the proposed method, given the previous problematic analyses, it diminishes the significance and novelty of the method.

A3: Thanks for your comment. Hope our above explanations and clarifications can help alleviate your concerns to our proposed COLA framework.

If satisfied with our response, could you please kindly consider raising the score?

Dear Reviewer WzEF,

Considering the author-reviewer discussion deadline is approaching very soon (only one day left), could you please kindly consider our rebuttal and potentially raise the score?

Your feedback would be extremely helpful for ensuring a fair and constructive evaluation.

Thank you again for your consideration!

Best regards,

The Authors