We sincerely appreciate your valuable comments and your positive remarks about our paper’s clear writing, strong motivation with accurate problem identification, and the effective design based on the anchor state grouping. We're also glad you found the step-level advantage computation efficient and the experimental results impressive. Regarding your questions, we provide our responses below:

(Weakness 1) Generalizability to complex and open multi-step environments.

Thanks for your valuable insight. As discussed in Sec. 6 (Conclusions and Limitations), we can use state similarity to better capture structurally equivalent states in complex and open scenarios. We have explored this perspective and conducted the following two experiments to evaluate how GiGPO performs under realistic noise where exact state matches are rare. We find using similarity for GiGPO works well.

1. Experiment 1

   • (Noisy WebShop): We used GPT-4o to generate 551 diverse advertisements, which are randomly inserted into the states returned by WebShop to simulate noise from ad pop-ups. This substantially increases the state spaces.
   • (Similarity‐based grouping): We use Ratcliff–Obershelp similarity (ranges from 0 to 1) to construct step-level groups: grouping the states whose similarity $\geq$ sim_thresh. Exact match is the special case sim_thresh=1.0.

   The results are shown as follows:

   	Score	Success Rate	Avg. Step Group Size
   GiGPO (sim_thresh=1.0)	74.5	56.7	1.01
   GiGPO (sim_thresh=0.9)	82.6	66.0	4.50
   GiGPO (sim_thresh=0.8)	83.2	66.2	4.96
   GiGPO (sim_thresh=0.5)	80.7	64.8	6.69
   GiGPO (sim_thresh=0.2)	78.7	58.6	12.15

   We can see that:

   • Exact-match failure. With sim_thresh=1.0, the avg. step group size collapses to $\approx 1$, indicating that almost no identical states are found. As a result, GiGPO degrades to GRPO and performance drops.
   • GiGPO remains robust and effective when using similarity-based grouping even under noisy conditions. As shown, sim_thresh=0.9/0.8 works well in such a setting. As sim_thresh decreases further (e.g., 0.5 or 0.2), group size increases, but similarity quality diminishes, revealing a trade-off between group generalization and precision.

2. Experiment 2

   • To further demonstrate GiGPO's applicability in open environments, we applied it to the Search-R1 environment[1] (web query task), where the agent interacts with a search engine (tool-calling) to answer questions.
   • We tested both GRPO and GiGPO using Qwen2.5-1.5B-Instruct.
   • Since search results are inherently variable, with extremely large state spaces, we used similarity-based grouping for GiGPO.

   	NQ	TriviaQA	PopQA	HotpotQA	2wiki	Musique	Bamboogle	Avg.
   Search-R1 (Qwen2.5-3B-Instruct)	0.341	0.545	0.378	0.324	0.319	0.103	0.264	0.325
   GRPO (ours)	0.380	0.541	0.423	0.303	0.269	0.088	0.317	0.331
   GiGPO (ours)	0.406	0.574	0.441	0.294	0.284	0.103	0.312	0.345

   We can see that, even in this open setting, GiGPO's step-level signal provides benefits over GRPO, demonstrating its effectiveness in open agentic tasks.

(Weakness 2) Hyperparameter sensitivity analysis.

Thanks for your valuable suggestion. We conduct the sensitivity study on the hyperparameter $\omega$ in WebShop.

We find that

• GiGPO needs an appropriate $\omega$ to work best.
• Increasing $\omega$ initially improves performance due to the added fine-grained step-level reward. However, performance declines beyond the optimum ($\omega=0.8$), suggesting that excessive emphasis on step-level signals can suppress useful trajectory-level guidance.
• GiGPO is relatively insensitive to $\omega$ within the range $[0.4, 1.2]$, demonstrating a reasonable degree of robustness to $\omega$.

(Weakness 3) LRM baselines.

We conduct experiments for DeepSeek-R1-0528 and the results are provided below:

Compared with Table 1 in the paper, DeepSeek-R1 achieves performance between GPT-4o and Gemini-2.5-Pro across most tasks.

(Weakness 4) Classic multi-step RL baselines.

Thanks for your valuable suggestion. We conduct experiments for RAGEN[2] on WebShop and the results are provided below:

GiGPO shows improvements over RAGEN, further supporting its effectiveness in multi-step settings.

(Weakness 5) High computational resource requirements.

We appreciate the concern, but we would like to clarify that the cited resource demand is ONLY the code-level optimization choices, which are fully independent of the GiGPO's algorithmic core and can be adjusted based on available hardware.

• In our experiments, we used 2 H100 GPUs and 64 CPUs only to shorten wall-clock time ($\approx$ 9h) via high parallelism. This setup is an implementation convenience, not an algorithmic requirement.
• The same training can be run on 1 H100 GPU + 32 CPUs (or even smaller setups) by, for example, leveraging the accumulation steps or allowing workers to execute sequentially. This change lengthens training time ($\approx$ 18h) but leaves results unchanged.

(Question 1) How are $r_t^{(i)}$ and $r_k^{(i)}$ given?

• This depends on the specific task and environment. Some environments provide dense rewards, while others are sparse-reward scenarios. In the case of sparse-reward environments, your understanding is correct: $r_t^{(i)} = 0$ for $t < T$; $r_T^{(i)} = 1$ if the task is successful, and $0$ otherwise.
• However, it is worth noting that when constructing step-level groups, we do not directly use the raw per-step reward $r_t^{(i)}$ from the environment. Instead, we use the discounted return $R_t^{(i)}$, as defined in Equations (5) and (6). This allows us to assign non-zero values to earlier steps ($t<T$) and its effectiveness is illustrated in Figure 3 of the main paper.

Finally, thank you again for your valuable and encouraging feedback. We will incorporate these clarifications and improvements into our revision.

---

References:

[1] Jin, Bowen, et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv, 2025.

[2] Wang, Zihan, et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv, 2025.

Thank you very much for your constructive review and positive feedback. We truly appreciate you highlighting the strengths of our method: the simplicity and intuitive design that make it easy to experiment with, the clarity and structure of the paper, and the strong empirical performance. Below, we provide our detailed responses to your remaining questions.

(Weakness 1) Universal GRPO improvement.

We appreciate you pointing out this confusion. To clarify: GiGPO is not intended for single-turn tasks (e.g., math or question answering), but is designed as a universal GRPO improvement for LLM agent scenarios, which involve multi-turn or multi-step interactions. We will revise the text to reflect the intended scope and applicability of GiGPO more accurately.

(Question 1) Section E.3 or E.4 might be more important.

Thank you for this helpful suggestion. We agree that Sections E.3 and E.4, which demonstrate the generality of GiGPO, offer important insights and may be more impactful than Section 5.5. We will consider moving them into the main paper to better highlight the broader applicability of GiGPO.

(Question 2) Demonstrating non-precise anchor state matching.

Thanks for your valuable suggestion. We have explored this perspective and conducted experiments to evaluate how GiGPO performs under realistic noise where exact state matches are rare. We find using similarity for GiGPO works well.

Specifically:

• (Noisy WebShop): To simulate realistic noise, we used GPT-4o to generate 551 diverse advertisements, which are randomly inserted into the states returned by WebShop to simulate noise from ad pop-ups. This substantially increases the state spaces.
• (Similarity‐based grouping): We use Ratcliff–Obershelp similarity (ranges from 0 to 1) to construct step-level groups: grouping the states whose similarity $\geq$ sim_thresh. Exact match is the special case sim_thresh=1.0.

The results are shown as follows:

We can see that:

• Exact-match failure. With sim_thresh=1.0, the avg. step group size collapses to $\approx 1$, indicating that almost no identical states are found. As a result, GiGPO degrades to GRPO and performance drops.
• GiGPO remains robust and effective when using similarity-based grouping even under noisy conditions. As shown, sim_thresh=0.9/0.8 works well in such a setting. As sim_thresh decreases further (e.g., 0.5 or 0.2), group size increases, but similarity quality diminishes, revealing a trade-off between group generalization and precision.

(Question 3 & Question 4) Can similar advantage calculations be applied to other RL frameworks?

1. Yes. As discussed in Section E.4, the core ideas of GiGPO (anchor state grouping & step-level advantage) are general and can be applied to all group-based RL frameworks. We have already extended GiGPO to DAPO in Section E.4 and additionally applied it to RLOO for this response. Results are summarized below: ||Score|Success Rate| |-|-|:-:| |DAPO (Sec. E.4)|84.6|66.1| |DAPO + Step-Level Adv. (Sec. E.4)|87.5|75.0| |RLOO|73.9|52.1| |RLOO + Step-Level Adv. |83.3|63.2|

2. Regarding PPO: As PPO is not a group-based RL method and does not require multiple rollouts for the same task, applying anchor state grouping to PPO is not straightforward. However, one potential way is to introduce task grouping into PPO by rolling out a set of trajectories under the same task. In such a setting, the core idea of GiGPO can be incorporated into PPO.

Thank you again for your valuable feedback. We will incorporate the above points in our revised version to enhance the technical contributions.

Thank you very much for your reply. Regarding your questions, we provide our responses below.

What specific tasks can GiGPO be applied to?

GiGPO can be applied to all LLM agentic tasks, which typically involve:

1. Interaction with external tools/environments: Not just generating text — it uses tools, checks websites, calls APIs, operates a computer/phone, or works inside a simulated environment.
2. A multi-step process: The model doesn't finish in one go (“instruction → answer”) — it takes a step, looks at the result from external environments, decides the next move, and keeps going until the goal is met (“instruction → intermediate steps → answer/goal”).

Examples of LLM agentic tasks to which GiGPO can be readily applied:

What other tasks have the potential to use the framework proposed by GiGPO?

Beyond agentic tasks, multi-turn dialogue is a natural fit. Multi-turn conversations between a user and an LLM are inherently multi-step, and the “user” can be treated as an “external environment”. This makes GiGPO a potential candidate for multi-turn dialogue optimization.

---

References:

[1] C Zhang et al. AppAgent: Multimodal Agents as Smartphone Users, CHI 2025.

[2] J Wang et al. Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration, NeurIPS 2024.

[3] K Zhang et al. CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges, ACL 2024.

[4] J Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024.

[5] Z Gou et al. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving, ICLR 2024.

[6] Z Xue et al. SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning, 2025

[7] OpenAI. Introducing ChatGPT Search, 2024.

[8] B Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning, arXiv 2025.

[9] M Chen et al. ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning, arXiv 2025.

[10] MJ Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model, CoRL 2024.

[11] G Lu et al. VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning, arXiv 2025.

[12] T Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, NeurIPS 2024.

Thank you very much for your detailed and thoughtful review. We truly appreciate you highlighting our credit assignment design as effective and pointing out that GiGPO enables fine-grained guidance while avoiding additional rollouts or costly value model estimations. We also thank you for your kind comments on the clarity of the paper and the substantial performance gains across benchmarks and model scales. Regarding your questions, we provide our responses as follows.

(Weakness 1) Constructing step-level groups demands a substantial amount of data.

We agree that increasing data volume can lead to more robust step-level groups. However, we would like to clarify that our experiments on GiGPO were actually conducted using relatively small data settings (batch_size=16, group_n=8). For comparison, prior agentic training work Search-R1[1] uses (batch_size=512, group_n=5).

Despite this, GiGPO was able to form effective step-level groups and maintain stable training, demonstrating its efficiency and practicality even under small data conditions.

(Weakness 2) Short-horizon or simple tasks.

Thanks for your valuable insight. To explore this, we evaluated GiGPO on a short-horizon agentic task using the Search-R1 environment[1]:

Specifically:

• This task involves answering questions via search engine tool-calling, with a maximum of 4 steps (short-horizon).
• We tested both GRPO and GiGPO using Qwen2.5-1.5B-Instruct.
• Since search results are inherently variable, with extremely large state spaces, we used similarity-based grouping for GiGPO as introduced in our response to (Weakness 3).

The results are shown as follows (Note: The “Search-R1” results are sourced from the original paper [1]):

We can see that, despite the short-horizon nature of the task, GiGPO's step-level signals still provided meaningful improvements over GRPO. Naturally, this advantage becomes more pronounced as the number of steps increases.

(Weakness 3) State noise and variability.

Thanks for your valuable insight. As discussed in Sec. 6 (Conclusions and Limitations), we can use state similarity to better capture structurally equivalent states in noisy scenarios. We have explored this perspective and conducted experiments to evaluate how GiGPO performs under realistic noise where exact state matches are rare. We find using similarity for GiGPO works well.

Specifically:

• (Noisy WebShop): To simulate realistic noise, we used GPT-4o to generate 551 diverse advertisements, which are randomly inserted into the states returned by WebShop to simulate noise from ad pop-ups. This substantially increases the state spaces.
• (Similarity‐based grouping): We use Ratcliff–Obershelp similarity (ranges from 0 to 1) to construct step-level groups: grouping the states whose similarity $\geq$ sim_thresh. Exact match is the special case sim_thresh=1.0.

The results are shown as follows:

We can see that:

• Exact-match failure. With sim_thresh=1.0, the avg. step group size collapses to $\approx 1$, indicating that almost no identical states are found. As a result, GiGPO degrades to GRPO and performance drops.
• GiGPO remains robust and effective when using similarity-based grouping even under noisy conditions. As shown, sim_thresh=0.9/0.8 works well in such a setting. As sim_thresh decreases further (e.g., 0.5 or 0.2), group size increases, but similarity quality diminishes, revealing a trade-off between group generalization and precision.

Thank you again for your valuable and encouraging feedback. We will incorporate all these clarifications and improvements into our revision.

---

References:

[1] Jin, Bowen, et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv, 2025.

We sincerely appreciate your in-depth and encouraging feedback. Thank you for highlighting that our method provides “an elegant solution” for obtaining fine-grained credit signals and finding the experimental results convincing. We also appreciate your kind comments on the clarity of the paper and the “deeper analysis” offered through ablation studies and training dynamics visualizations. Regarding your concerns, we provide our responses below:

(Weakness 1 & Question 2) The robustness of the anchor state grouping.

Thank you for this insightful suggestion. As discussed in Sec. 6 (Conclusions and Limitations), we can use state similarity to better capture structurally equivalent states in noisy scenarios. We have explored this perspective and conducted experiments to evaluate how GiGPO performs under realistic noise, where exact state matches are rare. We find using similarity for GiGPO works well.

Specifically:

• (Noisy WebShop): To simulate realistic noise, we used GPT-4o to generate 551 diverse advertisements, which are randomly inserted into the states returned by WebShop to simulate noise from ad pop-ups. This substantially increases the state spaces.
• (Similarity‐based grouping): We use Ratcliff–Obershelp similarity (ranges from 0 to 1) to construct step-level groups: grouping the states whose similarity $\geq$ sim_thresh. Exact match is the special case sim_thresh=1.0.

The results are shown as follows:

We can see that:

• Exact-match failure. With sim_thresh=1.0, the avg. step group size collapses to $\approx 1$, indicating that almost no identical states are found. As a result, GiGPO degrades to GRPO and performance drops.
• GiGPO remains robust and effective when using similarity-based grouping even under noisy conditions. As shown, sim_thresh=0.9/0.8 works well in such a setting. As sim_thresh decreases further (e.g., 0.5 or 0.2), group size increases, but similarity quality diminishes, revealing a trade-off between group generalization and precision.

(Weakness 2 & Question 1) Ablation study for the hyperparameter $\omega$

Thanks for your valuable feedback. We conduct the sensitivity study on the hyperparameter $\omega$ in WebShop.

We find that

• GiGPO needs an appropriate $\omega$ to work best.
• Increasing $\omega$ initially improves performance due to the added fine-grained step-level reward. However, performance declines beyond the optimum ($\omega=0.8$), suggesting that excessive emphasis on step-level signals can suppress useful trajectory-level guidance.
• GiGPO is relatively insensitive to $\omega$ within the range $[0.4, 1.2]$, demonstrating a reasonable degree of robustness to $\omega$.

(Weakness 3) The assumption of identical initial states

• The assumption of identical beginning is a standard setting in all group-based RL methods such as GRPO, RLOO, and DAPO. GiGPO is designed for group-based RL and therefore follows this established setting.
• We also explored the non-identical initial state case in our response to (Weakness 1 & Question 2), where we introduced randomized ad pop-ups to perturb the initial states. GiGPO remained effective using similarity-based grouping, suggesting that exact initial state matching might not be strictly necessary.
• This indicates that knowledge sharing is feasible among similar tasks, providing promise for relaxing this assumption in future work.

Thank you again for all your constructive feedback. We will include these clarifications, especially the experiments on robustness to noise and the hyperparameter sensitivity analysis.