PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments

Response to Reviewer dTPo

Dear Reviewer dTPo,

Thank you for your constructive feedback and for raising important questions about our experimental design and model comparisons. Your comments on potential biases in the multi-choice setup, the fairness of our model comparisons, and the scalability of our approach are highly valuable and have prompted us to conduct additional experiments and analyses to strengthen our work.

We have addressed each of your points below, including an analysis of the multi-choice setup, new experiments with a same-scale baseline, and an investigation into the effects of model scaling.

Thank you once again for your thoughtful review.

---

Q1: On Potential Bias in the Multi-Choice Setup

Reviewer's Comment: The multi-choice setup mixes candidate answers and actions; I wonder if this would introduce a bias towards choosing an answer or an action. Can the authors provide an analysis on this?

Our Response:

Thank you for raising this important concern about potential bias in our experimental setup. We analyzed the fairness of the multi-choice format from two perspectives:

1. Balanced Option Design: In the CLEVR-AVR benchmark, the candidate sets for most questions are designed to be balanced. A typical question includes 3-5 answer options and 3-6 action options. The number of options in each category is comparable, which avoids introducing a bias stemming from a significant quantitative difference between the two types of choices.

2. Analysis of Model Behavior: The prompt provided to the models includes a detailed description of the task, explicitly distinguishing between answer options and action options and explaining their respective meanings. This ensures that the model understands how to correctly select either an answer or an action. Furthermore, we have qualitatively inspected the models' generated reasoning traces and their final selections. We found no instances where the semantic intent of the reasoning process was inconsistent with the type of choice made (i.e., the model did not reason about taking an action and then select a final answer, or vice-versa). This indicates that the mixed-option setup does not negatively impact the model's decision-making process.

---

Q2: On the Fairness of Model Comparison

Reviewer's Comment: The comparison between AVR-QWen2.5-VL-7B and PhysVLM-AVR-3B is unfair: 1) AVR-QWen2.5-VL-7B does not rely on the more complex multi-stage training that is applied to PhysVLM-AVR-3B; 2) the model sizes are not on the same scale... Can the authors at least keep the same VLM backbone?

Our Response:

Thank you for this critical comment on the fairness of our model comparisons. We acknowledge your concerns and have conducted a new experiment with a same-scale baseline to provide a more direct comparison.

First, we wish to clarify our original intention. The Qwen2.5-VL model is a pre-trained foundation model, so we directly fine-tuned it rather than applying our full pre-training pipeline. The comparison between the fine-tuned AVR-Qwen2.5-VL-7B and our PhysVLM-AVR-3B was intended to highlight the significant impact of our multi-stage, progressive training strategy.

However, we fully agree that a comparison with a model of the same scale is essential. We have now trained and evaluated AVR-Qwen2.5-VL-3B, which has the same scale as our model and was fine-tuned using the same protocol as the 7B version. The results are as follows:

These results demonstrate that PhysVLM-AVR-3B significantly outperforms the same-scale baseline. This suggests that its superior performance stems from our architectural modifications (e.g., max pooling for efficient multi-image processing) and our dedicated multi-stage training strategy, rather than being an artifact of model size.

---

Q3: On Scaling Laws and Comparison with GPT-4o

Reviewer's Comment: GPT-4o, in general, outperforms all other models, including those trained on CLEVR-AVR, presumably because of its larger model size. Is it possible for the authors to train larger models and check if AVR-pretraining remains helpful as the model size increases?

Our Response:

Thank you for this excellent suggestion regarding model scaling. To investigate this, we fine-tuned a larger model, AVR-Qwen2.5-VL-32B, to analyze the impact of model scale on active visual reasoning performance.

Due to time and resource constraints, we were unable to train a larger version of our PhysVLM-AVR from scratch, as this requires a full pre-training pipeline unlike the direct fine-tuning of the Qwen2.5-VL models. However, we hope the analysis of the 32B model provides valuable insights.

• Performance of Fine-tuned AVR-Qwen2.5-VL-32B:
  • The ACC_FA improved to 41.2% (compared to 29.6% for the 3B version and 45.7% for GPT-4o).
  • The IGR improved to 39.5% (compared to 34.7% for the 3B version and 50.8% for GPT-4o).

These results confirm that increasing the model's scale leads to improved performance on active visual reasoning tasks. While our fine-tuned 32B model closes the gap, it does not yet surpass GPT-4o, suggesting that both scale and potentially architectural differences play a role. A more in-depth analysis would require training our PhysVLM-AVR at a larger scale, which we plan to explore in future work to investigate its potential to match or exceed proprietary models.

---

Q4: On the Applicability to Vision-Language-Action (VLA) Models

Reviewer's Comment: The current experiments focus only on MLLMs, but in the context of embodied reasoning, models that can predict and execute actions (e.g., vision-language-action models) seem to be better candidate models to be tested, can the authors explain if it is possible to apply CLEVR-AVR to VLA evaluation and in which ways if so?

Our Response:

Thank you for suggesting this exciting application of our benchmark. We agree that Vision-Language-Action (VLA) models are a natural fit for this task, and we designed CLEVR-AVR with this possibility in mind. CLEVR-AVR can be readily adapted for VLA model evaluation in the following ways:

1. Direct Simulation Platform Support: CLEVR-AVR is built on the Genesis simulation platform, which natively supports direct control of the robot arm via action sequences. This means the environment is already equipped to handle the low-level action outputs from VLA models.

2. Action Space Mapping: The discrete, high-level action space of CLEVR-AVR (e.g., "move object X," "rotate view by Y degrees") can be straightforwardly mapped to the action vocabulary of prominent VLA models (e.g., the action tokens used by RT-1 or RT-2). The VLA model's task would be to generate the correct low-level action sequence corresponding to the high-level reasoning step.

3. Expanded Evaluation Metrics: In addition to our existing metrics (ACC_ISJ, IGR, ACC_FA), we can introduce VLA-specific metrics such as:

   • Action Execution Success Rate: Measures whether the generated action sequence successfully accomplishes the intended high-level goal (e.g., was the correct object moved to the correct location?).
   • Reasoning-Action Consistency: Evaluates whether the executed action aligns with the model's stated reasoning for information gathering.

We plan to extend our work to evaluate state-of-the-art VLA models like RT-2 and RoboCat on CLEVR-AVR. This will allow for a comprehensive assessment of their ability to perform targeted, reasoning-driven actions to acquire information in service of a broader goal.

Response to Reviewer wcc4

Dear Reviewer wcc4,

Thank you for your valuable feedback and your suggestions for improving our experimental evaluation. Your points regarding fair model comparisons at the same scale, evaluating larger open-source models, and clarifying training time are all well-taken.

We have conducted additional experiments to address your first two points and provide a detailed breakdown of the training time as requested. The results and analysis are included in our detailed responses below.

Thank you for helping us strengthen our paper.

---

Q1: On Fair Comparison at the Same Model Scale

Reviewer's Comment: In Table 1, the comparison is between Qwen2.5-VL-7B (trained on AVR data) and the proposed PhysVLM-AVR-3B. It would be more informative to also include results for Qwen2.5-VL-3B trained on the same data, allowing for a fair comparison at the same model scale.

Our Response:

Thank you for this excellent suggestion. To ensure a fair comparison, we have conducted a new experiment by training Qwen2.5-VL-3B on our AVR data, using the same fine-tuning protocol as the 7B version. This allows for a direct, same-scale comparison with our PhysVLM-AVR-3B.

The results are as follows:

The performance gap between the two 3B models highlights two key advantages of our approach:

1. Architectural Optimization: The max pooling layer in PhysVLM-AVR is specifically designed to enhance the efficiency of integrating features from multiple images, a crucial capability for multi-step reasoning that is not a native focus of the standard Qwen2.5-VL architecture.
2. Targeted Training Strategy: Our multi-stage training curriculum places a strong emphasis on the AVR-Embodied Reasoning subset, which is rich in multi-image reasoning tasks. This targeted training better equips our model for the temporal reasoning demands of the AVR paradigm.

---

Q2: On Evaluating Larger Open-Source Models

Reviewer's Comment: According to Table 1, API-based models significantly outperform open-source 7B models, which score below 10% or even 0% accuracy on many tasks. It would be helpful to evaluate larger open-source models to better understand their potential on AVR.

Our Response:

Thank you for suggesting this. To better understand the potential of larger open-source models on our task, we have conducted a new experiment with the Qwen2.5-VL-72B model.

The results are presented below:

These results clearly show that scaling up the open-source model from 7B to 72B parameters leads to a substantial improvement in performance on the AVR task. This indicates that model scale is indeed a significant factor for acquiring active visual reasoning capabilities.

While there is still a gap compared to API-based models like GPT-4o, this experiment suggests that combining larger open-source backbones with our specialized multi-stage AVR training strategy could be a promising direction for future work to further close this gap.

---

Q3: On the Training Time Comparison

Reviewer's Comment: How does the training time of PhysVLM-AVR compare to that of standard fine-tuning?

Our Response:

Thank you for your question about training efficiency. Here is a comparison of the training times:

• PhysVLM-AVR-3B (Full Training): The complete multi-stage training process for our model takes approximately 76 hours on a server with 8x A800 GPUs. This includes all stages detailed in Section 4 and Appendix C of our paper, starting from feature alignment.

• Standard Fine-tuning (AVR-Qwen2.5-VL-7B): Fine-tuning the pre-trained Qwen2.5-VL-7B model only on our AVR-Core dataset takes approximately 2 hours on the same hardware.

It is important to note that the Qwen2.5-VL model has already undergone an extensive, multi-stage pre-training process by its developers. Our PhysVLM-AVR training starts from a much earlier stage (aligning visual and language features), which accounts for the significant difference in training time.

While our multi-stage approach is more time-intensive, it improves data efficiency by allowing the model to "focus" on learning specific capabilities at each stage, leading to superior performance across multiple benchmarks. We will explore methods to improve training efficiency in our future work.

Dear Reviewer F7gX,

Thank you for dedicating your time to review submissions for NeurIPS 2025. We are the authors of Paper #23215, titled "AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments".

After carefully reading your insightful questions, we believe there may have been a significant mix-up with another paper. Our work introduces AVR, a framework for Multimodal Large Language Models (MLLMs) to perform active information gathering in physical environments.

Your questions, however, seem to be directed at a paper focused on Visual Reasoning Reinforcement Learning (VR-RL), referencing specific components and results that are not part of our submission. For example, your questions mention:

• A "Table 1" comparing "Grounding-WorldModeling" with "WorldModeling".
• An "RL algorithm" whose performance is compared against "off-the-shelf VLMs".
• A "visual reasoning reward" and a "Bi-Level GAE" architecture designed to ensure stable training.

These specific technical terms, experiments, and architectural components are entirely absent from our paper. To be clear:

• Our paper does not contain a "Table 1" with "Grounding-WorldModeling".
• Our proposed AVR framework is not an RL algorithm in the sense described, and we do not use components like a "visual reasoning reward".
• The term "Bi-Level GAE" is not mentioned or implemented in our work.

We understand that reviewing numerous papers is a challenging task, and we suspect your detailed review was mistakenly assigned to our submission.

We would be deeply grateful if you could take a moment to re-examine our paper. We believe you will quickly see that your current review does not apply to our work.

Thank you for your understanding and your invaluable contribution to the community.

Best regards,

The Authors of Paper #23215

Response to Reviewer CyMs

Dear Reviewer CyMs,

Thank you for your meticulous review and insightful comments on our work. Your questions regarding the necessity of active reasoning, the sim-to-real gap, and dataset diversity directly address the core challenges of our research and have helped us to further refine our understanding and presentation.

In response to your feedback, we have provided additional analysis and clarification on several key points, including the expanded scope of active reasoning, the connection of our work to real-world scenarios, a precise definition of our evaluation metrics (IGR), and a detailed failure case analysis. These are detailed in the point-by-point responses below.

Thank you again for your valuable feedback.

---

Q1: On the Necessity and Scope of Active Reasoning

Reviewer's Comment: [...] Are there more applicable scenarios for active reasoning that passive reasoning fails? Is active reasoning also necessary for tasks where static images can already show enough information?

Thank you for this insightful question, which is crucial for clarifying the value of our work. We argue that the necessity of Active Visual Reasoning (AVR) extends far beyond resolving simple occlusion and stacking, and we frame its importance on three levels:

1. Expanded Application Scenarios: Active reasoning is indispensable in numerous scenarios where passive observation is inherently insufficient:

   • Limited Field of View: Tasks that require information from all sides of an object, such as checking for labels on the back of a container or inspecting its contents.
   • Dynamic Environments: Verifying the state of an object after it has been moved or interacted with by another agent.
   • Functional Reasoning: Determining an object's affordances, such as checking if a drawer can be opened or if a container is empty, requires physical interaction.

2. Enhancing Robustness in Static Scenes: Even when a static image appears complete, active reasoning serves as a critical verification mechanism:

   • Disambiguation: A slight nudge can differentiate whether two objects are merely adjacent or are partially overlapping.
   • Hypothesis Verification: Actively changing the viewpoint can rule out visual illusions caused by lighting, shadows, or specific perspectives, thereby increasing the robustness of the final conclusion.

3. Simulating Higher-Order Cognition: Human cognition often involves a hypothesis-verification loop, even in seemingly complete scenes (e.g., leaning in to read fine print). AVR aims to model this fundamental cognitive process, moving beyond a simple tool for "filling in missing information" to become a general strategy for robust and grounded reasoning.

---

Q2: On the Sim-to-Real Gap and Real-World Applicability

Reviewer's Comment: CLEVR-AVR is a synthetic benchmark, and is hard to transfer to real-world scenarios. [...] Also, can the introduced PhysVLM-AVR-3B operate in real-world scenarios?

Thank you for raising this critical point about real-world transferability. We address this concern from two perspectives and provide new experimental evidence to demonstrate our model's real-world capabilities.

1. The Scientific Value of a Synthetic Benchmark: The primary purpose of CLEVR-AVR is to serve as a controlled environment to precisely isolate and evaluate core agent capabilities, such as information-gathering efficiency, multi-step reasoning, and uncertainty estimation. This allows for rigorous scientific analysis, free from confounding variables like complex textures and lighting found in the real world.

2. Bridging the Gap to the Real World:

   • Real-to-Sim Generalization: A key aspect of our work is that the AVR-Core dataset is collected entirely from real-world physical interactions. Our model, trained on this real-world data, demonstrates effective active reasoning capabilities in the unseen simulation environment (CLEVR-AVR). This itself is evidence of promising real-to-sim generalization.
   • New Real-World Experiments: To directly address your concern, we conducted 20 new sets of experiments in real-world tabletop scenarios with task setups not seen during training. In these experiments, PhysVLM-AVR-3B achieved a Final Answer Accuracy (ACC_FA) of 40.2% and an Information Gain Rate (IGR) of 30.9%. This performance is consistent with its behavior on the CLEVR-AVR benchmark and provides strong evidence that our proposed method can operate and solve tasks in real-world settings.

We fully agree that scaling up the diversity of real-world training data is crucial for broader generalization, and this is a key direction for our future work.

---

Q3: On the Diversity of the AVR-Core Dataset

Reviewer's Comment: The AVR-Core dataset seems to be very useful in the ablation. However, this subset only contains the tabletop scenarios. The diversity is a concern.

We appreciate your concern regarding dataset diversity. Our design choice was deliberate, and we explain our rationale from two perspectives:

1. Tabletop Scenarios as a "Minimal Viable Unit": We position the tabletop environment as a "minimal viable unit" for studying active information gathering. This approach is analogous to how the original CLEVR dataset used simple geometric shapes to isolate and study core visual reasoning skills. While simple, tabletop scenes contain the fundamental physical interactions central to our research—occlusion, stacking, and spatial relationships—allowing us to focus on and rigorously test an agent's ability to close the perception-reasoning-action loop.

2. Future Work and Scalability: Our work on tabletop scenes establishes a strong methodological foundation. Having demonstrated the effectiveness of our approach in this controlled setting, our roadmap includes extending this paradigm to more complex and diverse environments, such as kitchen manipulation, warehouse sorting, and lab assistance. We believe this work provides a solid starting point for the community to build upon and apply active visual reasoning to a wider array of scenarios.

---

Q4: On the Calculation Details of IGR

Reviewer's Comment: How is the IGR calculated? What is the definition of the information-gaining decision?

Thank you for pointing out the need for clarification. A precise definition of our metrics is essential.

The Information Gain Rate (IGR) measures the efficiency of an agent's exploratory actions.

• Formula:

  IGR = (Total Number of Information-Gaining Decisions) / (Total Number of Decision Steps)
  

• Core Concept: What constitutes an "Information-Gaining Decision"?

  A decision (i.e., selecting an action) is defined as information-gaining if and only if its execution reveals new, task-relevant visual information that was previously unavailable to the agent.

  This new information must directly help in reducing the agent's uncertainty about the correct answer. In our benchmark, this is determined objectively using the simulator's ground truth.

• Examples:

  • Effective Decisions (Gain Information):
    • Moving a cup to reveal a key hidden underneath (to answer "Where is the key?").
    • Changing the viewpoint to see the color on the back of an object.
    • Lifting a block from a stack to count the total number of blocks beneath it.
  • Ineffective Decisions (No Information Gain):
    • Moving the camera to a viewpoint that reveals no new relevant objects.
    • Interacting with an object that is already fully visible and irrelevant to the question.
    • Repeating a previous action that yields no new information.

---

Q5: On the Failure Case Analysis (High ISJ but Low ACC_FA)

Reviewer's Comment: [...] The authors may analyze the issue in detail, that the ISJ of the model is really high, but the Final Answer Accuracy is still low. Could this be because the model does not understand the visual scene well?

Thank you for this excellent suggestion, which prompted us to conduct a deeper analysis of our model's limitations. Your intuition is correct.

Our analysis of failure cases where the model correctly identifies the need for more information (high ACC_ISJ) but still provides an incorrect final answer (low ACC_FA) reveals two primary causes:

1. Perception Errors:

   • Phenomenon: The model correctly decides to interact with the environment to get a clearer view (e.g., moving an occluder), but upon observing the new, unobstructed scene, it misidentifies a key attribute of a newly visible object (e.g., its shape, color).
   • Example: To count all cylinders, the model successfully moves a cube occluding a target object. However, it then misclassifies the newly revealed short, wide cylinder as a sphere, leading to an incorrect final count.

2. Temporal Integration Failures:

   • Phenomenon: After a sequence of interactions, the model accumulates a history of multiple images and reasoning steps. However, during the final reasoning stage, it fails to effectively integrate the entire sequence of historical observations, sometimes forgetting or confusing details from earlier steps.
   • Example: In a task, the model correctly identifies a red cube on the left in step 1 and a blue sphere on the right in step 2. When asked, "What are the shapes of the red and blue objects?", it might only focus on the most recent observation (the blue sphere) and fail to recall the information from the first step, leading to an incomplete or incorrect answer.

Your assessment is spot-on: these failures highlight remaining challenges in both fundamental visual perception and long-horizon temporal reasoning. Addressing these weaknesses is a key priority for our future research.