Real-World Reinforcement Learning of Active Perception Behaviors

Thank you for your comments!

Q1: What is the fundamental difference between the proposed method and existing approaches that learn active perception by introducing an additional motor policy? Q3: …it would be sensible to also compare with Shang et al., 2023 to empirically demonstrate the differences

Shang et al. requires a factorized action space, while AAWR does not. The fundamental difference between AAWR and such prior work like Shang et al. [1] is that Shang et al. only works on POMDPs with a “factorized” action space (explained below), whereas AAWR is derived with general POMDPs in mind, including those without factorized action spaces.

Specifically, Shang et al. requires that the action space can be factorized into “sensing” actions that move the sensor without affecting the scene, and “motor” actions that move the robot without affecting the sensor’s viewpoint. In their simulated robotic tasks, they assume access to a 3rd-person camera that can freely move around the scene to observe the robot. This assumption is mentioned in their limitation section, where they mention future work is needed to address tasks where there is a dependence between sensing and motor actions.

AAWR tasks cannot be factorized. The tasks evaluated in AAWR do not meet Shang et al.’s requirements. In our active perception tasks, the only moving camera is a wrist camera that is attached to the same robot arm that must manipulate the scene. This is very common in modern manipulation setups including diffusion policies, VLA models such as pi-zero, and more. In these settings, any action will simultaneously change the viewpoint and affect the scene. To take another example from our paper, in our Blind Pick task, the robot needs to perform interactive perception by grasping a block to detect it before picking it up. There is no clear distinction between sensor and motor actions, since the robot needs to physically affect the scene by grasping the block to detect it. Since AAWR does not assume a factorized action space, it can solve these tasks, while Shang et al. cannot.

Shang et al. is not ready for real world RL. Aside from assumptions in task formulation, extensive algorithmic modifications would be needed to make Shang et al. work in the real world. Shang et al. was trained only in simulation with online RL. In contrast, AAWR learns strictly in the real world from offline and online data, with a sample budget several orders of magnitude below the sample budgets of Shang et al.

In summary, we do not think it is feasible to apply Shang et al. to our tasks since our tasks’ action spaces cannot be factorized, which is a key assumption of Shang et al. Additional and substantial algorithmic modifications would be needed to make it more sample efficient for real world learning. Extending Shang et al. to real world tasks, potentially using ideas introduced in AAWR to make it efficient, would be an interesting future direction, but is not feasible within our rebuttal timeline.

In those cases, privileged information is also often used to train the critic. One example is Shang et al., 2023.

After reading Shang et al. 2023, we could not find any equation or text mentioning a privileged critic. It is possible we missed something, could the reviewer point to where it mentions the usage of a privileged critic?

Q2: Is it possible to allow the agent to actively select the privileged information to use also during training time, rather than relying on a pre-specified set during policy improvement?

We agree that this would be a cool direction to pursue! In fact, this is close to our own motivations for pursuing this work, and so we’ve already mentioned it as a potential future direction in the conclusion in line 349. One major challenge we anticipate is learning efficiency. Learning to choose sensors through RL might be too sample-inefficient for real-world settings. We believe alternative strategies from the active feature acquisition [2] and co-design [3] literature could be promising ways to perform sensor selection efficiently.

Q4: I feel it would be sensible to present the AAWR component with more details.

Thank you for the feedback on the presentation, and here’s our plan for improving clarity.

…the beginning of Section 3 should explicitly define the problem setting, clearly distinguishing what constitutes privileged information, what is considered sensory input,

• Line 104: The paragraph before motivated modelling active perception tasks as POMDPs. At the start of this paragraph, we will explicitly define the problem setting as “finding a policy that maximizes return in such POMDPs”.
• Next, we will clearly define what is the sensory input and privileged information. The sensory input is the observation space of the POMDP. The privileged information is either the hidden state of the POMDP or additional observations that are informative of the state.

what the final objective is, and conceptually, how the side information is used during training.

• The final objective is equation 3, and we will explicitly write that it is the final objective of our method. To make it clear where the privileged information is used, we will highlight the privileged input $s$ in blue in equation 3, and in lines 147-148.
• In the next version of the manuscript, we will add a figure visualizing the computation graph of equation 3. It will show two components, a policy and a privileged critic. The policy will just take in the observation, while the privileged critic will take in the observation and privileged information.

Q5: Minor: Line 147, V(s,a) -> V(s,z); Line 188: missing citation.

Thank you, we will update it in the next version of the manuscript.

[1] Shang, Jinghuan, and Michael S. Ryoo. "Active vision reinforcement learning under limited visual observability." Advances in Neural Information Processing Systems 36 (2023): 10316-10338.

[2] Rahbar, Arman, Linus Aronsson, and Morteza Haghir Chehreghani. "A Survey on Active Feature Acquisition Strategies." arXiv preprint arXiv:2502.11067 (2025).

[3] Antonova, Rika, and Jeannette Bohg. "Learning to be multimodal: co-evolving sensory modalities and sensor properties." Conference on Robot Learning. PMLR, 2022.

Thank you for the feedback!

novelty may be incremental relative to AWR?

As we argued around line 160 in the paper, Advantage Weighted Regression (AWR) was originally formulated for fully observed settings. One contribution of ours is showing that its naive application to POMDPs would result in suboptimal learning, see lines starting from 160 for an intuitive explanation and full theorem in Appendix D. We then show both theoretically and empirically why AAWR is the right instantiation of AWR for partially observed tasks

AAWR is a mechanistically simple modification of AWR, but it is precisely the simplicity of implementation that makes the AAWR recipe easy to evaluate on many diverse tasks, including with real-world robots, and alongside state-of-the-art generalist robot policies.

There is a lack of comparison with other approaches that use privileged information as mentioned in related work

Our task setup involves learning directly on a real world robot platform, with no access to a simulator. Due to the slow data collection time of real robots, the sample budget is very small (e.g. two hundred offline trials, and 50 online trials). Furthermore, the privileged observations are estimated using noisy, real world sensors.

True, we did indeed mention other privileged RL approaches in related work, but as we have argued there (Lines 82-95), these approaches are designed for sim-to-real RL, and (1) sim-to-real RL is not easily applicable to active perception due to the difficulty of effectively simulating most sensors, and (2) the privileged RL approaches designed for sim-to-real do not easily transfer to our settings, because they assume access to millions to billions of episodes of training data, and exploit low-dimensional ground truth simulator state as privileged information. As such, these methods are not empirically comparable to ours.

Do the authors have a sense of how "clean" the privileged information has to be?

We have some empirical sense of the robustness of our method to noise in the privileged information. In our real world experiments, we show that AAWR is able to learn from privileged observations, which have some irreducible noise due to imperfect sensing and environmental constraints.

For example, in the active vision pi0 experiments, we used off-the-shelf object detectors [1,2] to get privileged segmentation masks of the target objects (see figure 10 in appendix). These detectors would sometimes misclassify objects (e.g. classifying distractor objects with similar appearances as the target class), with a non-trivial misclassification rate of ~22%. We will include more discussion and details of the privileged sensor noise in the experimental detail sections (appendix F and G).

Next, in the Blind Pick experiment, we used a depth sensor (Realsense D435i) to get 3D position estimates of the object. On average, the estimates were off by 0.5-1cm, which is non-trivial considering the small size of the object (around 1.5cm) and the workspace (around 10cm). There are also cases when the 3D position estimates are invalid, such as when the robot gripper occludes the object.

It may also be interesting to try to theoretically characterize the sensitivity / robustness of AAWR to noise in the privileged observations. We will look into this in future work.

…does the type of privileged information impact policies?

Due to our limited resources on running real world RL experiments, we did not test different types of privileged observations for a given task. But we can give a few guidelines for choosing privileged information.

First, for the privileged critic, the privileged information should aid in predicting returns. This implication is drawn from appendix C and D, where we show the advantage is determined by both the agent state and environment state. In our active perception tasks, the location of the hidden object is an important privileged feature since the return is dependent on the object location.

Next, one should consider how easy it is to learn from the privileged sensor. For example, high dimensional sensors like cameras and depth sensors would require higher capacity encoders, which in turn demand more data samples to learn good representations.

Do the failure modes … stem more from the demonstration noise or privileged information noise?

We observe that in the offline case, the policy is more dependent on the particularities of the demonstrated behaviors, since it cannot improve from on-policy trajectories.

In the online RL phase, demonstration noise is no longer a problem since the agent is learning from its own on-policy trajectories. Here, we believe privileged information noise may impact the performance more, since noisy privileged information would result in less accurate advantage estimates, which would affect the policy gradient.

Dear reviewer u7RD,

Thanks for your reviewing efforts so far. Please respond to the authors' response.

Thanks, Your AC

We thank the reviewer for their helpful feedback, and discuss their concerns below. The reviewer asked “why AAWR would be better than search-based heuristics in …your tasks?”. They raise the point that non-learning-based approaches like a “hard-coded exploratory policy or a straightforward exhaustive search”, could plausibly do well, without requiring data collection and training costs.

Indeed, active and interactive perception tasks are search problems, and all search problems do have brute force “solutions” that will eventually succeed. The measure of good active and interactive perception strategies is how efficiently they can search. This is reflected in our performance metrics, where we measure not just if the policy succeeds at finding the object, but also the number of steps it takes.

AAWR is a learning based approach for acquiring active perception policies, and we seek to learn policies rather than engineer them, for the following compelling reasons.

1. Unlike a fixed, engineered policy, AAWR, like any other RL algorithm, can improve over time from its own experience towards an optimal policy. In contrast, a heuristic strategy may solve the task, but is likely inefficient. For example, we find that our engineered exhaustive search baseline, and a VLM-powered heuristic search baseline, both take over 3 times longer than AAWR to find the objects.

2. Engineered solutions often require more expansive sensor setups than what we afford to AAWR in our experimental setups. For example, [1] uses depth sensors and [2] uses calibrated cameras. In contrast, AAWR only uses a single uncalibrated camera in the $\pi_0$ active perception task, and only proprioception encoders in the Blind Pick task.

3. Engineered heuristic solutions tend to be very specific for each task and sensor setup, and may need to be completely reimplemented if anything changes. AAWR policies are acquired using the same recipe across different robots, sensors, and tasks, through collecting data and optimizing the AAWR objective.

Next, the reviewer is concerned about the scope of our experiments, in that “the experimental tasks are relatively simple or toy problems”. Our tasks in sim and in real are relatively hard compared to recent prior work on learning active perception policies [3,4,5], and are thus meaningfully advancing the state of the art. However, as mentioned in the paper, the field as a whole has struggled to produce useful active perception behaviors for robotic manipulation, and this is reflected in the complexity of tasks that are commonly investigated.

AAWR offers an improved technique and meaningfully advances the complexity of active perception behaviors that can be learned for real robotic setups, even showing the capacity to improve the behavior of a state-of-the-art generalist robot policy. Even so, we agree that there is much more room for growth, and hope that our paper will be an important stepping stone in this direction!

We will aim to add this context into the paper, thank you again for the thoughtful queries.

[1] Krainin, Michael, Brian Curless, and Dieter Fox. "Autonomous generation of complete 3D object models using next best view manipulation planning." 2011 IEEE international conference on robotics and automation. IEEE, 2011.

[2] Liu, Yang, Pansiyu Sun, and Akio Namiki. "Target tracking of moving and rotating object by high-speed monocular active vision." IEEE Sensors Journal 20.12 (2020): 6727-6744.

[3] Grimes, Matthew Koichi, et al. "Learning to look by self-prediction." Transactions on Machine Learning Research (2023).

[4] Cheng, Ricson, Arpit Agarwal, and Katerina Fragkiadaki. "Reinforcement learning of active vision for manipulating objects under occlusions." Conference on robot learning. PMLR, 2018.

[5] Dass, Shivin, et al. "Learning to Look: Seeking Information for Decision Making via Policy Factorization."Conference on robot learning. PMLR, 2024.

We thank the PC for the comments, and address the confusion below.

Availability of Privileged Observations during Training vs. Deployment

To be clear, privileged observations are only available during training time, and are not available during policy deployment. The training phase includes both offline and online phases. After training, the policy is deployed to the environment where only partial observations are available. Therefore a privileged policy cannot be run in our deployment setting. To make this clear, we have improved the description of training and deployment in Implementation Details and algorithm blocks in Figure 3.

Details on $s, o^+$, and policy input $z$

In practice, the privileged observation $o^+=(o, o^p)$ serves as an improved estimate of the state: $\mathcal{H}(s \mid o^+) << \mathcal{H}(s \mid o)$ , as it reduces the advantage estimation error. In some cases, we can get perfect (e.g. sim) estimates of the state, or noisier estimates in the real world. Noisier estimates of the state are still useful as privileged inputs, as it reduces the advantage estimation error, and we have verified this through our real world experiments. We leave exact characterization of how privileged information noise affects AAWR to future work. We expect this to be relatively straightforward by leveraging prior theoretical results in asymmetric learning [1].

$z=f(o_1, \ldots o_t)$, the observation history representation for the policy is first defined in Section 3's POMDP description. It is then described in section 3.2, where we introduce the AAWR loss: "We aim to train a policy ... conditioned on the agent state $z_t$ (equivalent to history $h_t=(o_1 \ldots o_t)$".

1. Lambrechts, Gaspard, Damien Ernst, and Aditya Mahajan. "A Theoretical Justification for Asymmetric Actor-Critic Algorithms." ICML (2025).