Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression

Dear Reviewer Sovg,

Thank you for your thorough and constructive review of our work. We sincerely appreciate your recognition of the experimental rigor and practical relevance of our evaluation benchmarks, as well as your encouraging feedback. Your insights strongly support our goal of providing actionable guidance on the trade-offs in LLM compression for agentic workflows.

We are pleased that the detailed results and visualizations in the appendix proved useful—ensuring methodological transparency was a key priority for us. Should you have any further suggestions or require additional clarifications for the final version, we would be happy to incorporate them.

Thank you once again for your time and thoughtful evaluation. We look forward to any additional comments you may have.

Best regards,

Authors of #6400

Q1: About Theory.

Theoretical claims are poor for this article. What is the relationship between the degradation of the compression has on LLMs and the degradation of the compression has on the Agentic Behaviors?

Ans for Q1: This paper is primarily an empirical study, as indicated by the title. To address this, we have developed a concise framework to explain how quantization errors propagate in sequential decision-making (agent scenarios). For details, please refer to this anonymous link.

Q2: About the setup for Action Execution

Action Execution aspect, lacks evaluation metrics setup.

Ans for Q2: The Action Execution metrics (Sec. 4) adopt T-Eval's setting, evaluating Plan (similarity/LIS), Reason/Understand (Sentence-BERT), Retrieve (exact match), Instruct (format/parameter accuracy), and Review (classification accuracy).

Q3: About the Embodied AI tasks.

The evaluation in Workflow Generation and Real-World Application seem to have overshadowing parts about the Embodied AI task.

Ans for Q3: In this paper, we employed ScienceWorld and AlfWorld as they inherently require structured, multi-step reasoning, making them ideal for evaluating workflow generation. BabyAI, while useful for basic instruction-following, lacks the complexity needed for higher-level planning assessment. Refer to Tab.8,9 and anonymous link for more details.

Q4: About the compression methods

The compression methods are limited to only five compression methods. LoRA and Distillation are not systematically analyzed.

The passage use results from distilled model and original models to evaluate the distillation methods but not thoroughly analyzed.

Larger Models (only contains size from 1.5B to 7B) are not evaluated.

Ans for Q4: As presented in Sec.2.2 and Sec.8, we prioritize these compression methods for the following reasons:

1. Practical Impact & Compatibility: The selected methods (GPTQ, AWQ, SmoothQuant, SparseGPT, Wanda) are foundational and widely adopted in serving systems, which are supported by vLLM[3] and SGLang[4]. They are critical for real-world high-throughput serving (e.g., 10–24× speedups over HuggingFace).
2. Distillation/LoRA: While we include distilled models (e.g., R1-Distill series) in baseline comparisons, a systematic evaluation of distillation or LoRA lies beyond our focus on post-training compression. These techniques inherently require retraining or architectural modifications.
3. Model Scale: We evaluate up to Qwen2.5-32B (Tab.9), but larger models (e.g., 70B) are prohibitively expensive (>1 month/run on 8 GPUs). Memory constraints in long-context agent tasks further limit scaling.

Q5: About the evaluation

Four capabilities should be evaluated multi-step planning, long-context coherence, adaptive reasoning. How is each Benchmark related to these core aspects?

Ans for Q5: Thank you for raising this important point. We explicitly connect each benchmark to the core capabilities in the following ways:

• Multi-step planning is evaluated in Workflow (Sec. 5) and Real-World Applications (Sec. 7).
• Long-context coherence is assessed in Long-Context (Sec. 6).
• Adaptive reasoning is demonstrated in both Workflow (Sec. 5) and Real-World Applications (Sec. 7).

In line 35-52, we have already refined these tasks. We will further enhance clarity in the revised manuscript

Q6: About the writing.

Some expressions are vague. For example, “Knowledge Distillation Shows Unexpected Performance Characteristics” is not a good summary sentence to conclude without showing what characteristic is.

Ans for Q6: To address this, we have revised the statement to:

"Knowledge distillation from a reasoning model lead to performance degradation in agent scenarios."

Q7: About the Optimal Strategy:

Optimal Compression Strategy Recommendations proposed in Chapter 4 should be analyzed after all the aspects are evaluated

Ans for Q7: To align with the feedback, I propose refining the section title to "Compression Strategy Recommendations for Action Execution". And for overall guidings, please refer to Q3 to Reviewer DuJV.

Q8: About the "Multi-Turn".

The framework stresses the importance of “Multi-Turn” conversation testing, but it is not highlighted in the following experiments

Ans for Q8: Thank you for your feedback. The "Multi-Turn" is implicitly integrated into the experimental design:

• In Sec. 5): Tasks in WorfBench inherently involve multi-turn interactions, as agents must dynamically plan and execute
• In Sec.7, Benchmarks like Jericho and PDDL explicitly test multi-turn reasoning. For instance, agents must navigate 10+ conversational turns to solve complex puzzles

Thank you for the thorough comments and recognition of our work. We appreciate that you acknowledge our experiments are “comprehensive and sound”. Please see our responses to your questions and concerns below.

Q1: About the metrics.

Can you clarify if (and how) the novel metrics (ERank, Top-K Ranking Correlation, Energy-based Analysis) were validated against simpler, traditional benchmarks and metrics (e.g., perplexity, accuracy, simpler correlation metrics)?

Ans for Q1:

As explained in the Abstract and Introduction (lines 97-108), our proposed metrics are not against traditional evaluation metrics. Instead, they focus on compression explainability by revealing how compression influences LLM behavior and causes degradation. While traditional metrics only reflect the overall performance decay after compression, they cannot show how compression specifically affects the model behavior. For example, our Topk Ranking Consistency metric focuses on the inference feature of language models: when quantization/pruning causes he token ranking dislocation. For instance, in an instruction fine-tuning scenario, quantization can reverse the probability order of "of course" and "sorry" tokens, directly changing the output affective tendency.

Besides, inspired by this question, we explored that whether our metrics can be used not only to exploiting of compression, but also how to choose compression methods. We first conducted experiments on InternLM2.5-20B:

Then, we compute the correlation of traditional metric (acc, ppl) with topk ranking, and find that the ranking consistency between ppl and topk ranking are relative high, meaning that topk ranking can reflect the evaluation performance in some extend.

Compared with ppl, our topk ranking achieved generally better performance, indicating stronger predictive capability for downstream tasks.

Q2: About deeper insight of the unexpected degradation

Can you elaborate on why distilled reasoning models unexpectedly underperform specifically on agentic tasks? Is this issue related directly to compression methods, or is it indicative of more general difficulties these models have with multi-step reasoning?

Ans for Q2: The underperformance stems from two key factors:

• Knowledge Gap in R1: Prior Deepseek V3/R1 lacked agentic capabilities (like tool use) until the recent V3 0324 version. Therefore, the distillation process couldn’t transfer the agentic knowledge from Teacher R1 to student Qwen. This distillation process mainly focus on reasoning and dialogue skills (evidenced by Qwen2.5 32B's improvement from 50.0%to 72.6% in AIME2024)[1].

• Capacity Tradeoff in Distillation: The distilled models tested have limited capacity. When distillation priorizes core reasoning skills (math), agentic skills (function call) may be deprioritized. From the perspective of information bottleneck perspective, we formulate it as:

  $$\mathcal{L} _ {\text{distill}} =I(\theta_S; y_{\text{reason}})-\beta I(\theta_S; y_{\text{agent}})+\lambda\|\theta_S\|$$

  where $I(\theta_S; y_{reason})$ maximizes reasoning performance, $\beta I(\theta_S; y_{\text{agent}})$ represents the penalty on agentic skill retention, $\lambda\|\theta_S\|$ enforces model compactness. So, for capacity-constrained LLMs, maximizing reasoning performance often suppresses agentic capabilities due to competition for parameter space. This suggests that larger models can mitigate this trade-off. In the Wanda (2:4) benchmark, Qwen2.5-14B experienced an 11.4% performance degradation, while Qwen2.5-7B saw a more significant drop of 17%(from Table 7).

[1] DeepSeek AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Q3: About Figures

Figure 5 is hard to read, and Figure 6 seems incomplete?

Ans for Q3: We apologize for the clarity issues. For Fig.5, we improved readability by using darker color. For Fig. 6, we removed the redundant metric and add missing one(SmoothQ) to ensure completeness. You can check the anonymous link over the updated figures.

Q1: About the practical utility of the proposed metrics:

the proposed metrics significantly enhance understanding of compression impacts is less convincing, as their practical utility remains unclear.

Ans for Q1: . We would like to address each metric:

• ERank: Diff-eRank[1] is a theoretically grounded method based on information theory and geometric principles. It analyzed differences between base and trained models using ERank. We extend it to quantized LLMs shows ERank's effectiveness with experimental results (extending Tab.1 of Diff-eRank), where higher ERank values correlate with better model performance (see below table).
• Top-K Ranking Correlation: As detailed in Q1 to Reviewer K5g3, this metric provides meaningful insights by focusing on the model's behavior regarding top-k token ranking. The correlation analysis captures important aspects of the model's predictive distribution.
• Energy-based Analysis: We find that compressed models' energy distributions gradually align with uncompressed ones over timesteps (Fig.14), showing that while quantization disrupts LLM representations, parameter redundancy compensates for the loss. Aggregated energy reflects this recovery and correlates with performance (see below table).

[1] Lai Wei etc, Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models, NeurIPS 2024

Q2: About the connection between the metrics with traditional metric.

the connection between the experimental outcomes and the practical implications of the proposed metrics requires additional clarification. How the proposed metrics correlate with the specific benchmarks' performance metrics.

Ans for Q2: Our proposed three metrics serve as complementary tools to traditional benchmarks, offering causal insights into performance variations. Specifically, they help explain why certain models achieve better or worse results on standard metrics like perplexity (PPL) or accuracy. Also, please refer to our response to Reviewer K5g3 (Q1), where we provide additional experiments demonstrating how Top-K Ranking Correlation reflects changes in traditional benchmark performance. You can check the anonymous link about the topk-ranking results.

Q3: Guiding Principles for Practitioners

The paper lacks deeper insights and guiding principles on how practitioners might effectively leverage these extensive observations to make informed decisions.

Ans for Q3: We have refined our guidelines to provide clearer, actionable insights for practitioners:

1. For Specific Agent Capabilities:

   • If targeting single capability like tool use, directly consult the task-specific results in Sections 4-7 to select the optimal compression method.
   • eg: For tool use, AWQ preserves JSON-structured outputs better than GPTQ (Fig. 3); for long-context, AWQ surpass GPTQ in most of cases.

2. For General-Purpose Agent Deployment:

   • Model Choice > Compression Method: Base model capability is critical. For instance, Qwen2.5 outperforms R1-Distill-Qwen2.5 in four capabilies (Tab. 8). Prioritize high-quality base models first.
   • Default to AWQ: AWQ shows stable performance across all benchmarks (workflow, tool use, long-context).
   • If Quantization is Infeasible: Use Wanda (outperforms SparseGPT in 80% of cases).
   • Avoid R1-Distill for Agents: Despite its reasoning strengths, it fails in real-world agent tasks (Fig. 7). Use quantized base models instead.

3. For hybrid scenarios (e.g., workflow + tool use): Start with Qwen2.5-7B (strong base) → Apply AWQ.

You can refer to anonymous link for guideline flow chart.

Q4: About the importance：

Could you elaborate on the significance of your proposed metrics to the main contributions of the paper?

Ans for Q4:

These metrics are foundational to our three key contributions:

• ERank can reflect how the information are compressed from the perspective of information theory. See Q1 to Reviewer DuJV.
• Topk Ranking Correlation is intuitive over how the compression influence sampling process. we also added additional experiments to show that topk ranking correlation have the ability to predict the downstream performance. See Q1 to Reviewer K5g3.
• Energy is helpful for us to understand what is going on during the decoding stages, where as the time goes, the distribution shifted to uncompressed one and become stable. See Q1 to Reviewer DuJV