VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents

We sincerely thank Reviewer ortS for the detailed and constructive feedback. We are encouraged that the reviewer finds our paper to be well-written and easy to follow, with an interesting and relevant topic and experiments that span a wide range of environments.

The reviewer's main concerns are related to the strength of our baselines, the performance analysis on the PrimitiveSkill task, the need for additional RL comparisons, and the reporting of variance. We believe our clarifications and the new experimental results provided below will address these points.

---

1. (Regarding Weakness 1) Strength of Baseline Models

TL;DR: We added experiments with stronger proprietary models (OpenAI GPT-o3 and GPT-o4-mini). Our 3B VLM outperforms these models on most tasks, demonstrating the effectiveness of our method.

We appreciate the reviewer’s suggestion to benchmark against stronger recent models. To address this, we ran additional experiments.

We compared our method against strong proprietary models like GPT-o3 and GPT-o4-mini. It is important to note that these models were used with a 4096-token budget (much larger than the 500-token budget for our method and other baselines). We also trained the model for longer steps in Sokoban and FrozenLake using our method (300 vs. 150 in the original submission).

Our 3B VLM outperforms both GPT-o3 and GPT-o4-mini on most tasks, highlighting the effectiveness of our method. (Note: Qwen3 only supports text inputs. We will test it when its vision-language version releases.)

2. (Regarding Weakness 2) Clarification on the PrimitiveSkill Task Performance

TL;DR: We acknowledge the reviewer's observation is accurate. Our method's main benefit in the PrimitiveSkill task is improved sample efficiency in early training, while performance converges to Base RL in later stages due to task-specific properties. We will revise our claims to reflect this nuance.

We thank the reviewer for this insightful observation. We agree that our original statement "Visual Reasoning RL consistently outperforms Base RL" needs to be more carefully qualified for the PrimitiveSkill task.

• We re-analyzed PrimitiveSkill by averaging results across its subtasks.

1. The averaged results show Visual Reasoning RL learns faster and achieves higher performance in the initial training phases, confirming its ability to improve training efficiency.
2. In the later stage, Visual Reasoning RL converges to Base RL. We attribute this to two reasons: (i) The PrimitiveSkill task provides object coordinates directly, reducing the need for visual grounding, which is a key strength of our reasoning component. (ii) The action space is relatively predictable, allowing the base agent to eventually learn the world model without extra supervision.

• Edits to the paper: We will edit our manuscript to state that Visual Reasoning RL provides significant sample efficiency improvements, especially in early training, while acknowledging that performance may converge in later stages for tasks with reduced grounding complexity. Thank you for helping us present our contributions more precisely.

3. (Regarding Weakness 3) Isolating Contributions with More RL Baselines

TL;DR: Our paper already includes an action-level RL baseline (“No-Think RL”, Table 1), which corresponds to the model directly predicting actions without reasoning. We further added new baselines (Vanilla-PPO, GRPO w/mask, Turn-PPO w/mask). Our method outperforms the next-best baseline by 49%, highlighting the effectiveness of Visual Reasoning RL.

We thank the reviewer for this suggestion. Our paper (Table 1) already includes an action-level RL baseline (“No-Think RL”), which removes the reasoning step entirely and lets the model output actions directly. This setting serves as a strong reference point for RL without explicit reasoning.

To further isolate the effect of our proposed components (visual reasoning reward and Bi-Level GAE), we conducted additional experiments with the following baselines:

• Vanilla-PPO: Standard PPO without state masking.
• GRPO (w/mask): Trajectory-level GRPO with state masking.
• Turn-PPO (w/mask): PPO variant where all tokens in one turn share the same advantage.

Our method outperforms the next-best baseline by 49%, confirming that our method is effective.

4. (Regarding Weakness 4) Reporting Variance Across Multiple Runs

TL;DR: We have conducted new experiments on FrozenLake and Sokoban with random seeds. The results show that our Visual Reasoning RL achieves a higher mean performance.

We agree that reporting variance is crucial for RL results.

• We have run 3 experiments with random seeds for both Base RL and Visual Reasoning RL on the FrozenLake and Sokoban tasks. We report the mean and variance below.

(Note: Results shown as mean ± standard deviation)

The results confirm that Visual Reasoning RL performs better on average.

• Due to computational constraints, we cannot feasibly run multiple seeds across all 40 experiments (including tasks, strategies, and ablations). Each full run requires 1040 H100 GPU hours, making multiple seeds for every setting prohibitive.

5. (Regarding Question 1) Longer Training Curves and Table 3 Reporting

TL;DR: We provide longer training curves for FrozenLake and Sokoban, which show the previous decline in Base RL was a temporary fluctuation. Table 3 reports the best checkpoint for fair comparison. Our new multi-run results confirm that Visual Reasoning RL still achieves superior average performance despite this.

Thank you for this question. RL training can indeed be noisy.

• Longer Training Curves: We have extended the training for FrozenLake and Sokoban to 300 steps.

The longer curves show that the decline in Base RL was a temporary fluctuation. However, our Visual Reasoning RL still achieves higher final performance.

• Clarification on Table 3: For Table 3, we reported results from the best-performing checkpoint for all methods to ensure a fair comparison. We will clarify this reporting methodology in the paper.

Dear Reviewer ortS,

We sincerely thank you for maintaining your positive score and for your constructive feedback throughout the review process. Your comments and suggestions were highly insightful and have greatly helped us strengthen our paper. If you have any remaining unaddressed concerns, please feel free to let us know, and we would be happy to provide further clarification. Your guidance has been invaluable, and we deeply appreciate your engagement in improving our work.

Best,

Authors of Submission 19716

We sincerely thank Reviewer wZ7m for the detailed and constructive feedback. We are encouraged that the reviewer finds our work to be a well-written paper with extensive evaluation, and considers our POMDP formalization of the multi-turn problem as sound and interesting.

The reviewer’s main concerns seem to arise from the clarity of our experimental design and the significance of the performance gains achieved by our method. We believe our clarifications and the new experimental results provided below can resolve these concerns.

---

1. (Regarding Weakness 1 & Question 1) Clarification on the experiment design and new Bi-Level GAE experiments.

TL;DR: Bi-Level GAE was designed for fine-grained credit assignment when turn-level visual reasoning rewards are introduced. New experiments confirm that Bi-Level GAE also improves performance with Free-Think, but its full benefit emerges under fingrained intermediate rewards.

• Why we didn't use Bi-Level GAE in the first part: In the first part, each task’s reward design was coarse-grained, and our focus was to study the impact of reasoning strategies rather than the RL algorithm itself. Most tasks only provide (i) a simple turn-level format reward, which is easy to learn per our experimental observations, and (ii) a trajectory-level task-success reward. Therefore, we adopted a standard RL algorithm (with modifications for multi-turn tasks) instead of introducing Bi-Level GAE at this stage.

• Why Bi-Level GAE was introduced later: When we incorporated turn-level visual reasoning rewards, we observed that fine-grained credit assignment became crucial. The standard advantage estimation failed to effectively propagate supervision across turns. This motivated the development of Bi-Level GAE, which explicitly combines turn-level and trajectory-level credit assignment to better leverage intermediate reasoning signals.

• New Experimental Results: We appreciate the reviewer’s suggestion to test Bi-Level GAE more broadly. We have now run new experiments applying Bi-Level GAE directly to the FreeThink strategy.

While the effect on Sokoban is small, FrozenLake shows a substantial gain, validating the effectiveness of Bi-Level GAE. We hypothesize that the degree of improvement varies because Bi-Level GAE is most effective when it can propagate correct and strong intermediate reward signals. In our full method, the visual reasoning reward provides exactly this kind of dense, turn-by-turn signal. However, in the base task setup, the intermediate rewards are often sparse, with the most significant reward only being delivered upon final task success. This sparsity can limit Bi-Level GAE from demonstrating its full advantage.

2. (Regarding Weakness 2) Edits for missing equation and contribution summary.

TL;DR We thank the reviewer for pointing out these clarity issues. We will add the missing equations and a more explicit, structured summary of contributions to the camera-ready version.

• We will add the formal definitions for the turn-level delta ($\delta_n^{\text{turn}}$) and advantage ($A_n^{\text{turn}}$) after line 256:

  $$\delta_n^{\text{turn}} = r_n + \gamma_{\text{turn}} V_\phi(p_{n+1}) - V_\phi(p_{n})$$ $$A_n^{\text{turn}} = \delta_n^{\text{turn}} + \gamma_{\text{turn}} \lambda^{\text{turn}} A_{n+1}^{\text{turn}}$$

• We will revise the end of the introduction as follows:

  Our contributions are fourfold. (1) We provide empirical evidence for the importance of explicit visual state reasoning in multi-turn RL for VLM agents. (2) We analyze how different visual state representations affect reasoning quality and task performance. (3) We introduce visual reasoning rewards to guide visual state reasoning and Bi-Level GAE for fine-grained credit assignment. (4) We present VAGEN, a scalable training framework that decouples environment setup from model training, enabling efficient experimentation and algorithmic extensibility. Together, these contributions establish a principled pathway for developing VLM agents capable of robust visual reasoning in dynamic environments.

• We will also add a brief summary of contributions at the beginning of each relevant section to improve readability.

3. (Regarding Weakness 3) Clarification on the inclusion of the SVG task.

We thank the reviewer for raising this point. We included SVG: (1) to evaluate whether our method extends beyond tasks with well-defined success and limited action space. SVG introduces open-ended text generation and non-binary evaluation metrics (DreamSim, DINO); (2) we reformulated this task from single-turn to multi-turn, inspired by findings that converting single-turn problems into multi-turn dialogues with feedback can improve LLM reasoning [Liu et al., 2025; Zheng et al., 2025; Shinn et al., 2023]. Our work represents an early but important step in exploring this in the VLM domain.

Reference:

Liu et al., 2025. "A Simple 'Try Again' Can Elicit Multi-Turn LLM Reasoning."

Zheng et al., 2025. "What Makes Large Language Models Reason in (Multi-Turn) Code Generation?"

Shinn et al., 2023. "Reflexion: Language Agents with Verbal Reinforcement Learning."

4. (Regarding Summary/Impact) New results demonstrating significant performance gains.

TL;DR We have conducted longer training and new baseline comparisons, which show that our method achieves overwhelmingly better results. Our lightweight 3B VLM now outperforms best evaluated proprietary model by 14% overall and surpasses other RL baselines by 49%, demonstrating clear significance and impact.

• New Results vs. Advanced Models: We trained our model for longer in FrozenLake and Sokoban (300 steps vs. 150) and added comparisons against Open AI o3 and Open AI o4‑mini. The results show our 3B parameter model now achieves a new state-of-the-art.

Our relatively small 3B model outperforms Open AI o3 by +14% overall.

• New Results vs. RL Baselines: To better highlight the novelty and contribution of our method, we also ran new experiments comparing against other RL baselines.

Our method leads the next‑best baseline by +49%.

• We believe the two new experiments above confirm the effectiveness and significance of our method.

5. (Regarding Question 2) Clarification on "non-comparable" models.

We apologize for the confusing phrasing. "Non-comparable" was intended to mean "not a controlled comparison". Our intent was to emphasize that for a scientifically rigorous evaluation of reasoning strategies and optimization algorithms, it is crucial to hold the base model constant (i.e., same size and architecture). We will correct the phrasing in the camera-ready version to "not a controlled comparison" to avoid future misunderstanding.

Dear Reviewer wZ7m,

We are pleased to know that our rebuttal satisfactorily addressed your concerns. We sincerely thank you for your invaluable points regarding Bi-Level GAE, paper edits and clarifications, the role of the SVG task, and other helpful suggestions. We will incorporate the expanded experiments, clearer equations, a concise contribution recap, and the detailed SVG discussion, as well as all updated results, into the paper to ensure that your insightful feedback is fully reflected.

Best,

Authors of Submission 19716

We sincerely thank Reviewer HHiJ for the detailed and constructive feedback. We are encouraged that the reviewer finds our contributions novel and the empirical evidence compelling.

The reviewer's main concerns relate to the generalizability of our model architecture, the complexity of the experimental scenarios, and the need for deeper analysis on certain results and mechanisms. We believe our clarifications and the new experimental results provided below will address these points.

---

1. (Regarding Weakness 1) Generalizability of the Model Architecture

TL;DR: We ran new experiments on a different model family (InternVL3). The results show that our Visual Reasoning RL method generalizes and provides better performance compared Base RL.

We appreciate the reviewer's point regarding the potential limitation of using a single model family. We chose Qwen2.5-VL because it is a powerful and widely used open-source VLM in RL research, allowing for a fair and controlled comparison with existing work. As the reviewer noted, we acknowledged this in the limitations section of the paper.

To explicitly address this concern, we conducted new experiments on the InternVL3-2B model for the Sokoban task.

The results demonstratea that the Visual Reasoning RL generalizes to a different VLM and achieves higher performance compared to Base RL.

2. (Regarding Weakness 2) Simplicity of Scenarios, Limited Evaluation, and Choice of Algorithms

TL;DR: We argue that our tasks are challenging, as evidenced by the poor performance of existing powerful VLMs. While the reviewer’s comment that “the set of evaluation metrics is limited” and “the evaluation methodology and choice of algorithms exhibit inherent limitations” essentially reiterates points that we explicitly acknowledged in our limitations section, we agree this deserves clarification. We explain below why our current evaluation is sufficient and how our framework supports richer analysis. We also conducted new experiments with additional RL baselines to directly address the concern regarding algorithm selection.

We thank the reviewer for encouraging us to clarify the complexity and scope of our evaluation.

• Task Complexity: We respectfully disagree that our scenarios are simple. As shown in Table 1 of our paper, a highly capable model like GPT-4o achieves an overall score of only 0.6, which is far from the upper bound of 1.0. This indicates that the tasks are sufficiently challenging to differentiate even state-of-the-art models.

• Evaluation Methodology: Regarding the concern on evaluation methodology, we want to emphasize that this is one of the points we explicitly listed in our limitations section. We focused on a small set of fundamental and widely recognized metrics in the main paper due to space constraints and to keep our analysis clear. Nevertheless, our evaluation is already extensive, covering 5 diverse tasks, spanning 2D and 3D states, and involving varied action spaces (discrete, continuous, open-ended), and evaluation metrics (success rate, DINO score, DreamSim score). Importantly, our framework supports a wide range of metrics (e.g., format reward, grounding rewards, world-modeling rewards, entropy loss, response length, valid actions, effective actions, etc), which we will include in the appendix of the camera-ready version as auxiliary analyses.

• Choice of Algorithms: Similarly, the point about the limited choice of algorithms is also one of the limitations we explicitly acknowledged. To further address this, we have now added experiments with additional RL baselines:

FrozenLake and Sokoban trained on 300 steps

These results demonstrate that our proposed algorithm significantly outperforms the additional RL baselines, further validating the effectiveness of our design choices.

3. (Regarding Question 1) Failure Analysis of Grounding-WorldModeling on PrimitiveSkill

This is an excellent question. In the PrimitiveSkill task, we give a list of object coordinates as the additional information. We hypothesize that the "Grounding" step, which involves translating these precise coordinates into a natural language description, can introduce noise. This linguistic abstraction is less useful when precise textual information is given. The agent can directly perform "WorldModeling" that leverages the structured input to predict the next state. This makes it inherently more robust for such precision-critical tasks. This insight is validated in Section 4 of our paper, where we show that structured representations are indeed better for PrimitiveSkill.

4. (Regarding Question 2) Off-the-Shelf VLMs Outperforming RL

We thank the reviewer for raising this important question about the performance comparison between off-the-shelf VLMs and our RL approach.

1. Task Complexity is Adequate: As discussed previously (Regarding Weakness 2), we have explained that the tasks are challenging. Off-the-shelf VLMs do not solve them perfectly.
2. "No Training" is an Unverifiable Assumption: The statement that off-the-shelf VLMs have received "no training" for the tasks we present here might not be accurate. Proprietary models like Claude and GPT are extensively fine-tuned, often with methods like RLHF, and their training data and processes are not public. They might also have been pre-trained or finetuned on the task we choose.
3. Our RL Methods on a 3B-Model Show Better Overall Performance: Both Base RL and Visual Reasoning RL on a 3B-model outperform proprietary VLMs in overall scores. Proprietary VLMs used in our paper are often assumed to be significantly larger than 3B.
4. New Experiments with Extended Training Steps: To further showcase the potential of our method, we extended the training on FrozenLake from 150 to 300 steps. The results show that our trained agent now surpasses the best-performing proprietary model evaluated for this task in the paper.

5. (Regarding Question 3) Improving Generation of Structured Visual States

This is a very insightful question about a key challenge. We propose several training paradigms and architectural modifications to mitigate performance degradation from noisy grounding:

1. SFT: We can first collect a dataset of scene images paired with their ground-truth text-based structured states. Fine-tuning the VLM (either the full model or just the vision tower) on this data would strengthen its ability to generate accurate structured representations for specific scenarios.
2. Integrating Better Vision Encoder: We can incorporate existing fine-grained image understanding approaches like GLIP, RegionCLIP, and FG-CLIP into the VLM architecture. These models are explicitly trained for enhanced grounding and could provide more robust object-centric features.
3. Scaffolding and Prompting: Techniques that guide the VLM's internal processing, such as prompting it to generate a "cognitive map" or spatial representation of the scene before outputting the final structured state, could improve accuracy.

References: Li et al., "Grounded Language-Image Pre-training", CVPR 2022. Zhong et al., "RegionCLIP: Region-based Language-Image Pretraining", arXiv 2021. Xie et al., "FG-CLIP: Fine-Grained Visual and Textual Alignment", arXiv 2025. Yin et al., "Spatial Mental Modeling from Limited Views", arXiv 2025.

6. (Regarding Question 4) Stability of Bi-Level GAE vs. Base RL's Decline

This question gets to the core of our method's advantage. The performance decline in Base RL during later training stages is a problem in sparse-reward, multi-step tasks.

• Why Base RL Fails: In later training stages, the agent has already mastered the "easy" trajectories. For more challenging cases that require precise, long-horizon reasoning, Base RL provides only a single reward at the very end. If the agent executes several correct turns but makes one mistake at the final step, the entire sequence of correct actions is penalized along with the final error because the trajectory-level advantage is negative. This noisy credit assignment discourages the necessary exploration to solve hard cases and leads to training instability.
• How Bi-Level GAE Provides Stability: Our Bi-Level GAE directly counteracts this. By computing advantages at both the trajectory level and the turn/token level, it provides turn-level rewards. Even if a trajectory fails at the end, our method can assign positive rewards to the intermediate turns that were executed correctly. This fine-grained feedback allows the agent to learn progressively, reinforcing correct sub-policies even when tackling difficult problems.

We sincerely thank Reviewer 77zp for the excellent review and insightful feedback. We are greatly encouraged that the reviewer recognizes our work as complementary to existing research, praises our comprehensive experiments, and highlights the novelty and importance of the Bi-Level GAE method for enabling multi-turn visual reasoning.

The reviewer’s main questions are related to the applicabitliy of our method to more complex environments, training efficiency, and its scalability to larger size and generalizability to other VLMs. We believe our clarifications and the new experimental results provided below will address these concerns.

---

1. (Regarding Weakness 1) Generalizability to Complex Environments

TL;DR: We chose controlled environments to enable fundamental and deep analysis. Our framework is directly applicable to most current agentic task simulators (e.g., AI2THOR, ALFWorld, WebShop, OSWorld) which provide text-based state information for supervision. The core principle of world modeling is essential for any agent's success, making our method broadly relevant.

We thank the reviewer for this important point and acknowledge this limitation.

1. Rationale for Controlled Environments: Our use of controlled environments was a deliberate choice. These settings are essential for conducting a rigorous and systematic analysis of the core components of visual reasoning. They allow us to isolate variables and understand the fundamental training dynamics.
2. Applicability to Current Agent Benchmarks: We want to clarify that our framework is directly applicable to the majority of simulators used in current agentic research. Environments for embodied agents and GUI agents like Maniskill, AI2THOR, ALFWorld, Minecraft, OSWorld, and WebShop all provide access to text-based ground truth states that can be used as supervisory signals for grounding and worldmodeling. Since nearly all current agentic RL research is conducted in simulated environments, our method is highly relevant to the field's present state.
3. Fundamental Importance of Grounding and WorldModeling: While understanding current state and predicting the next state might be challenging in cluttered and dynamic environments, an agent's ability to grounding and worldmodeling is fundamentally tied to its ability to understand the task and succeed. Therefore, our method is inherently correlated with task success and is broadly applicable in principle.

2. (Regarding Weakness 2) Discussion on Training Efficiency

TL;DR: We provide approximate training costs below and will add a detailed section on training efficiency to the appendix in camera ready.

We appreciate the reviewer raising this practical concern. Training efficiency is indeed an important consideration. We have compiled the training costs for our primary experiments below.

We will add a dedicated section to the appendix of our camera-ready paper with a full analysis of these costs. We also note that efficiency could be further improved by integrating advanced techniques like async RL, which we are exploring as a future optimization.

3. (Regarding Weakness 3) Scalability and Transferability to Other VLMs

TL;DR: We ran new experiments on a larger model (Qwen2.5-VL 7B) and a different model family (InternVL3-2B). The results are highly positive, showing that our method can be scaled ups and generalizes across different VLM family.

This is a crucial question, which we acknowledged as a limitation of our work in our submission draft. Now we are excited to share new results that address it directly. We conducted additional experiments on the Sokoban task using both a larger model from the same family (Qwen2.5-VL 7B) and a model from a different family (InternVL3-2B).

The results confirmed the scalibility and transferability of our method.

Dear Reviewer 77zp,

We sincerely thank you for recognizing the value of our work and for confirming that our rebuttal effectively addressed most of your concerns. The issues you raised regarding method applicability, training cost, scalability, and generalizability are all very important. We are pleased that our additional experiments and analyses helped clarify these points. We will incorporate these new results into the paper to ensure that your constructive feedback is fully reflected.

Best, Authors of Submission 19716