The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement

Weakness 1: The proposed framework is very close to SELU [1], apart from the fact that SELU addresses multimodal LLMs rather than purely text-based ones. In SELU they also have a critic-guided improvement mechanism. Thank you for pointing out this related work. We would like to clarify the key differences between our approach and SELU[1]:

Thank you for bringing this concurrent work to our attention. While there are high-level similarities, CGI and SELU differ in fundamental ways:

• Feedback Type: SELU's critic provides trajectory-level binary feedback ({yes, no}). In contrast, CGI's critic provides step-level, rich, structured linguistic feedback (grading, feasibility, revision suggestions). This is a core design difference.
• Critic Objective: SELU's critic is a simple classifier to filter trajectories for training data. Our critic is a generative model trained to produce actionable guidance.
• Problem Setting: SELU is designed for settings without expert data and works by simplifying tasks when the agent fails. Our work focuses on demonstrating that high-quality language feedback itself is a powerful mechanism for improving performance on complex, fixed tasks.

We will add a detailed discussion of SELU to our related work section.

Weakness 2: A key ablation is missing: removing the initialization with expert trajectories.

This is a critical point about the reliance on expert data. We have conducted the requested ablation study by replacing AgentGym's gold paths with GPT-4o generated trajectories on ScienceWorld, following the exact pipeline described in Section 3.2 to construct $D_{critique}$ for training the critic model.

The results are clear: CGI remains highly effective even without gold-standard expert data. Our method still achieves a performance of 56.93, significantly outperforming both the baseline and the GPT-4o model that generated the initial data. This shows the robustness of our framework. We will add this to the appendix.

Weakness 3: To insist on this potential weakness, if the authors have a way to generate expert trajectories, why learn another agent that will perform the same trajectories? Is it just a matter of distilling into a smaller size agent?

This is not just distillation. The expert trajectories from AgentGym [2] were collected via many expensive sampling runs with GPT-4o. Our framework achieves two things:

1. Superior Performance: Our final CGI agent (72.91 on ScienceWorld) significantly outperforms the expert model (GPT-4o at 46.91) used to generate the initial data.
2. Efficiency: Our framework uses much smaller models (Llama-3-8B), making it far more efficient at inference time than running a large, proprietary model like GPT-4o.

Weakness 4: The results presented in Tables 1 and 2 seem to result from a single run for each condition. Is there some variability in the results if one performs the same experiment several times?

Our main results use greedy decoding (temperature=0), which is deterministic. However, to address your valid concern about stochasticity, we ran a new experiment on ScienceWorld and WebShop with temperature=1 for 3 separate runs.

The standard deviation is very low, confirming that our method is robust and its performance is stable across runs. We will add this analysis to the appendix.

Weakness 5: Verbal vs. numerical feedback comparison

The comparison between verbal feedback and numerical feedback is potentially very interesting but the proposed study does not adequately focus on this specific question, as it might also be the critic accuracy that is different between DGAP, Explicit RM and CGI, and not only the feedback modality.

Thank you for this insightful comment. As mentioned in Appendix G.1, we ensured fair comparison by using the same number of gold paths for training CGI, DGAP, and ExplicitRM. Our goal is precisely to compare different critic capabilities, which is why we selected two major categories: numerical methods (DGAP, Explicit RM) and verbal methods (CGI, self-refine), while keeping the actor model fixed as Llama3-8B.

Expanding on the above point, isn't the point that CGI proposes (potentially not proposed by the actor) action refinements and the actor is fine-tuned on the difference to these actions, whereas in DGAP and Explicit RM, the actor will be fine-tuned on the difference to the best-ranked action among the ones proposed by the actor?

We would like to clarify the process: For CGI, the actor samples 5 candidate actions, the critic provides critique for each action, and the actor takes the next step based on the critic's feedback. For numerical methods, we also sample 5 candidate actions, and the critic selects the best-ranked action for the next step. Both approaches are completely aligned in terms of sampling count and critic evaluation frequency.

Other Points:

• Training on Failed Trajectories: This is an interesting direction. Our current design focuses on learning from success to ensure the quality of the critic's training data. Learning from failure is a complementary research problem, as seen in methods like SELU that modify the task itself. We believe our focus on high-quality critique generation is a valid and important contribution.
• Fine-tuning Original Model (πθ): As shown in Appendix F.1, Table 4, we use three types of data to train the actor at each iteration: (1) correct paths (mixture of $D_{gold}$ and $D_{correct}$ from model-environment interaction), (2) $D_{refine}$ (critic-action pairs from model-environment interaction), and (3) $D_{general}$ (general datasets to maintain general capabilities). As model capability improves, $D_{correct}$ and $D_{refine}$ increase, while $D_{gold}$ and $D_{general}$ remain constant.
• Train/Test Split: Yes, we strictly followed the official splits from AgentGym [2]. There is no data leakage.
• Confusing Sentence: We will rephrase to: "Our CGI approach, using an 8B critic, significantly boosts the performance of various actor models (including Llama-3-8B, Llama-3-70B, and fine-tuned variants) far beyond what GPT-4o can achieve alone."
• Longer Trajectories: We will rephrase the analysis in Section 6.1 to be about "trajectory length" rather than "difficulty" to be more precise.
• Related Work & Typos: We will move the related work to the main body and fix all typos you kindly pointed out. We initially did this due to space constraints. In the final version, we will move the Related Work section to the main text.

Thank you for taking the time to review our paper! We are pleased that our response addressed your concerns. We also appreciate your engagement during the discussion.

Weakness 1: Ambiguity in the Definition of Environmental Reward (R).

Thank you for asking for clarification. The environmental reward R is defined solely by task completion, following the standard setup of the AgentGym benchmark [1] we use.

• ScienceWorld: A score from 0 to 100 based on sub-task completion.
• TextCraft & WebShop: A binary reward (1 for success, 0 for failure).

Our reward function does not include factors like efficiency or path length. This is a deliberate choice to ensure a fair and direct comparison with prior work on these standard benchmarks. We will add this explicit definition to Section 4.1.

[1] AgentGym: Evolving Large Language Model-based Agents across Diverse Environments

Weaknesses 2&3: Lack of a structured reward function and inadequate handling of incorrect candidate actions.

We respectfully clarify that our framework explicitly handles both of these points.

1. Structured Critiques: As described in Section 3.2, our critiques are highly structured. The critic is trained to generate feedback across five dimensions: Contribution, Feasibility, Efficiency, Overall Grading, and a Suggested Revision. We enforce this structure by filtering the training data, ensuring the critic learns to produce standardized and effective feedback.
2. Handling Incorrect Actions: Our framework is designed for the common scenario where all candidate actions might be incorrect. The critic's role is not just to pick the best action, but to provide a Suggested Revision that guides the actor toward a better, potentially entirely new action.

To make this concrete, we will add the following example to the appendix, where all initial actions are flawed:

• Task: Use a thermometer in the bathroom.
• State: The agent is in the kitchen.
• Bad Candidate Action: focus on thermometer (it's in another room).
• Our Critic's Feedback: Overall Grading: Very Poor. Suggested Revision: The agent should first 'open door to hallway' to begin navigating toward the bathroom...

This shows how CGI provides actionable guidance even when the initial proposals are unworkable.

Weakness 4 In the experimental section, although the paper mentions that task difficulty is related to trajectory length, it does not provide a clear numerical threshold or boundary to define long-term and short-term tasks.

Thank you for requesting this clarification. In our experiments, task length classification is based directly on the ScienceWorld environment [1], which provides gold-standard trajectories from 30 hand-coded oracle agents. Following prior work [2, 3], we adopt the established categorization:

These lengths correspond to the average number of steps taken by the oracle agent for each task difficulty level. This classification enables us to analyze how CGI performs across varying task complexities, as shown in Figure 4. We will include this detailed information in Section 4.1 and ensure consistent terminology throughout the paper.

[1] ScienceWorld: Is Your Agent Smarter than a 5th Grader? EMNLP 2022

[2] SWIFTSAGE: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks. NeurIPS 2023

[3] Discriminator-Guided Embodied Planning for LLM Agent. ICLR 2025

Weakness 5: While action refinement is shown to improve the model's performance, there is a lack of clear ablation comparison experiments. It is recommended to include relevant explanations or experiments.

Thank you for raising this concern. We want to clarify that we have conducted comprehensive ablation studies, which are presented in the appendix:

Ablation Study 1 - Data Component Analysis (Appendix H): We conducted an ablation study on the contribution of three types of data (D_correct, D_refine, and D_general) used to enhance the actor model during action refinement. Our results confirm that critique-action pairs (D_refine) provide the most significant contribution to performance improvement.

Ablation Study 2 - Iteration Count Analysis (Appendix F.3): We analyzed the effect of different numbers of action refinement iterations, demonstrating that three iterations achieve optimal performance while a fourth iteration provides no additional benefit.

We believe these ablation studies provide sufficient evidence for the effectiveness of our approach. If you have specific aspects you would like us to investigate further, we would be happy to include additional ablation experiments.

Limitations

Although the paper demonstrates the effectiveness of the CGI method, it lacks a discussion of its limitations or potential issues in real-world applications.

Thank you for this point. We analyze computational cost in Appendix E.2, showing that CGI provides a +43.31% performance gain for a ~4x inference time cost, a much more favorable trade-off than using a larger model like GPT-4o (+4.29% gain for a 7x time cost). While we selected complex, long-horizon environments to approximate real-world challenges, we agree that scaling to fully open-ended tasks is a key direction for future work. We will add a dedicated Limitations section to discuss this.

Sincerely thank you for taking the time to review our paper. We are glad to know that our response addressed your concerns. Thanks for your engagement during the discussion!

Thank you for highlighting the importance of our work's perspective and its "great" performance.

Weakness 1: The idea of using feedback to improve LLMs is not entirely new.

We agree that feedback-based improvement is an established direction. However, as we state in Lines 26-32, our novelty lies in how we generate and utilize this feedback. Prior work has struggled with two key challenges that our CGI framework is designed to solve:

1. Low-Quality Feedback: Relying on self-correction [2,3] or simple numerical rewards [1] provides limited guidance. Our key contribution is a specialized, trained critic that generates high-quality, structured, and actionable natural language critiques.
2. Poor Utilization: Agents often struggle to effectively integrate verbal feedback. Our second key contribution is the iterative action refinement stage, which explicitly fine-tunes the actor to understand and apply these critiques.

Our work is the first to systematically address both the generation and utilization of high-quality language feedback in a unified framework.

[1] Discriminator-guided embodied planning for LLM agent. ICLR 2025

[2] Self-refine: Iterative refinement with self-feedback. NeurIPS 2024

[3] Reflexion: Language agents with verbal reinforcement learning. NeurIPS 2023

Weakness 2: There lacks analysis on why the proposed critic model can significantly outperform GPT-4o.

This is an excellent question. Our fine-tuned critic excels because it becomes a specialist for the task domain. Compared to the generalist GPT-4o, our critic develops superior spatial reasoning and goal-directedness. We will add the following examples to the appendix to illustrate this.

• Example 1 (Spatial Reasoning):
  • State: Agent is in the workshop. Task requires an object in the bathroom.
  • Action: focus on thermometer
  • GPT-4o (Incorrect): Overall Grading: Excellent. It approves the action based on the task description, ignoring the agent's location.
  • Our Critic (Correct): Overall Grading: Very Poor. It correctly identifies the thermometer is not in the current room and suggests navigating first.
• Example 2 (Goal-Directedness):
  • Task: Find a living thing.
  • Action: move to the closet (an exploratory but inefficient action).
  • GPT-4o (Inefficient): Overall Grading: Neutral. It allows this exploratory step.
  • Our Critic (Efficient): Overall Grading: Poor. It rejects the inefficient action and suggests a more direct one, like opening the door to explore other rooms.

Q1: Would the generated natural language feedback be still useful to powerful models that already have a good baseline performance?

Thank you for pointing this out. To address this concern, we conducted experiments using powerful actor models (GPT-4o and Llama3-70B) guided by our critic model. The results are shown below:

Key findings:

• Llama3-70B shows substantial improvements: +23.24 points on ScienceWorld and +43.85 points on WebShop
• GPT-4o also benefits significantly: +29.00 points on ScienceWorld and +33.11 points on WebShop

These results demonstrate that even powerful models with strong baseline performance can benefit from our critic's guidance. We will add these results to Table 1 in the final version.

Dear reviewer, since the discussion deadline is approaching, could you kindly let us know if our responses have addressed your concerns? We appreciate your time and feedback.

Thank you for the positive feedback on the timeliness of our work and the robustness of our results. We address the concerns about clarity and the critic model below.

Weakness 1: Method clarity, undefined symbols ($D_{general}$, $D_{critique}$, $y$, $\beta$), and unclear actor loss $L_{actor}$.

Thank you for pointing out the lack of clarity. We will add a comprehensive symbol table to the appendix and clarify all definitions in the main text.

Dataset Definitions

Loss Function Parameters

We will revise Lines 132-143 to ensure the mathematical formulation aligns with the textual description and add these definitions explicitly in the main text.

Weakness 2: The expert critic requirement limits the performance ceiling, and the critic is static.

This is an excellent point. Our framework has two key strengths here:

1. The expert does NOT create a performance ceiling. As shown in Table 1, our fine-tuned Llama-3-8B critic already surpasses the GPT-4o expert by a large margin (+29.16%). This demonstrates that our method distills and refines the expert's knowledge, rather than being capped by it.
2. The critic can be dynamically updated. Following your insightful suggestion, we ran a new experiment with an alternating critic-actor update scheme. We used the first iteration's critic to generate new critique data and retrained the critic.

Results on ScienceWorld and TextCraft:

As the results show, an updated critic yields further performance gains. This confirms the critic is not limited to being static and can evolve with the actor. We will add these results and the methodology to the final paper.

Questions:

Does Table 1 result come from the 3rd iteration as Table 2 and 4?

No. Table 1 is designed to isolate and evaluate the quality of the critic model itself. It shows the performance of a base Llama-3-8B actor guided by different critics, without any iterative actor refinement. This is in contrast to Tables 2 and 4, which show the full CGI framework with iterative actor improvement. We will state this explicitly in the caption of Table 1 to make this distinction clear.

Minor suggestion: change "critique" to CGI to avoid confusion in Figure 5.

Thank you for the suggestion. We agree this improves clarity and will update Figure 5 in the revised version.

Thank you for your thoughtful and thorough response. You have addressed my confusion of the paper, and I believe with proposed changes made to the draft, it would be a very good paper and Im looking forward to the final version. I will improve my score.