The Value of Sensory Information to a Robot

We thank the reviewer for their detailed review!

The paper aims to provide insights for robotics tasks, yet all results are obtained only in simulation. The characteristics of simulated benchmarks are clearly very different from real-world robotics tasks..

We ask that you evaluate this paper not as introducing an immediately applicable engineering solution, but as the first empirical approach to address a core scientific question in robotics: “How critical is sensory information at each moment to a pre-trained policy?” See our global response above for further discussion on the relevance of our findings to real robotic tasks.

Furthermore, it seems challenging to conduct a similar analysis on real robots …

It is true that the VoSI measure defined in Eq(1) is not easy to implement on real robots. However, our interaction-free metrics (L473-502 + Appendix C) provide an alternative, often highly correlated method of implementing VoSI measurements, that we believe may be easier starting points for studies with real robots.

Reward-free and interaction-free VoSI alternatives are not very practical. In TD-MPC2, can we use deviation between predicted rollouts and true rollouts?

The plan disagreement metric proposed in the paper has a similar flavor to the reviewer’s proposal. In this metric we are measuring the deviation between the planned rollouts and the true rollout by computing the expected cumulative action disagreement. But, if the reviewer is suggesting capturing a notion of dynamics uncertainty, we believe TD-MPC2 directly does not support this investigation as the architecture assumes that the dynamics model is deterministic. Please let us know if we have misinterpreted your suggestion.

The sensing frequency adaptation scheme was only tested on a gridworld task. I assume optimal sensing strategy is significantly close to fixed-rate to periodically reduce error accumulation. … Unsure whether a strategy to adapt replanning frequency makes sense in typical real robotics applications.

We disagree that robotics environments would mainly only require fixed-rate sensing. The rate of accumulation of error is typically dramatically different during some phases of a task (e.g. contact-rich manipulation interactions or walking on rocky terrain) than during others (e.g. free space arm motion or walking on flat carpeted floors). For example, insights from our investigation also suggest similar characteristics in the finger-spin task: in the initial phase of the task the agent might need to sense more frequently (Figure 7-2b), but once the spinner is already spinning above a desired velocity the sensing rate can be lower (Figure 7-2c).

On the relevance of adaptive replanning, many prior works have indeed pursued this line of thinking for computational efficiency in real practical systems that either have real-time requirements or operate with limited computational resources. For instance, in distributed networked systems where the communication layer imposes bandwidth and latency constraints – knowing when information is valuable to process and transmit for replanning can be very beneficial to characterize for deployment of these policies.

Finally, note that the sensing frequency adaptation shown in Fig 11 is not the main focus of the paper. Our algorithm there is a naive greedy approach that only scales to small environments like grid-world. This is only intended as a proof-of-concept.

Yet, I think in most robotics tasks the main problem is still an accumulation of error ... the object might move slightly different than expected due to friction at every step. This is very different from the sudden discrete errors for the gridworld tasks …

Error does indeed accumulate, but not at the same rate, and there are jumps and discontinuities in how it accumulates. Consider a spinning cube dropped onto the floor: the contact event when it makes contact with the floor can lead to many wildly divergent outcomes arising from minute differences in the configuration at the time of contact. Contact-rich tasks involving repeated making and breaking of contacts are difficult precisely because of these phenomena. This is connected to our examples in our previous response — the free space motion of an arm usually incurs very little accumulation of error before and after a grasp, but it is very difficult to predict exactly how an object such as a spoon might be oriented in the gripper after the grasp – all the error accumulation happens in the grasping phase, and it is more important to frequently sense the world during that phase (if the object orientation is important to get right for the task).

This example is still not very clear to me. I assume in most robotics applications the bandwidth / latency of the communication suitable to provide sensor readings at a fixed frequency. In which scenarios would it be beneficial to skip reading sensor measurements in some steps?

Sorry, we should have cited the literature that studies these problems, and perhaps it would have been clearer. Thank you for checking back with us to give us an opportunity to clarify. The setup in our previous response exemplifies the kind of problem studied in the networked control systems literature [1-3]. We explain more clearly here: In networked control systems often the communication layer imposes severe bandwidth and latency constraints, e.g.: a marine robot/robots operating in remote locations[1], in such scenarios knowing when information is valuable to transmit and process can be quite beneficial to minimize the overall communication cost[2,3].

However, networked control systems only help to make very obvious why skipping sensing is of interest. Even in a single-agent setting, if the agent has limited computational resources, it is often beneficial to avoid processing the sensory inputs and replanning based on them, even when the sensory inputs themselves are available at each moment in time. This is in fact common practice today with lookahead policies such as diffusion policies and action-chunking transformers that require non-trivial computation and time, even when they are typically run on well-equipped desktop computers connected to a robot. On more constrained robots (e.g. imagine a small UAV robot operating with onboard compute), these constraints are even more real, even for simpler policies.

I still do not think that the statement "agents [...] exposed to more training data usually degrade less at lower sensing rates" is reflected in the data.

We will modify this claim to say – “appear to” as the claim clearly holds for the finger-spin task but cannot be crisply captured by a single metric in the push-T task. Relative drop as a performance metric is indicative but not the final word: it requires some interpretation in the context of the specific reward function for each task, and the baseline rewards of the closed-loop controller relative to random behavior etc. We have provided more context near the results in Fig 5 to reflect this.

This notation seems non-standard and had me confused a bit. Please consider revising it.

Thank you for the suggestion. We have fixed this in the revised manuscript.

References

[1] Zolich, Artur, et al. "Survey on communication and networks for autonomous marine systems." Journal of Intelligent & Robotic Systems 95 (2019): 789-813.

[2] Solowjow, Friedrich, et al. "Event-triggered learning for resource-efficient networked control." 2018 Annual American Control Conference (ACC). IEEE, 2018.

[3] Kepler, Michael E., and Daniel J. Stilwell. "An approach to reduce communication for multi-agent mapping applications." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.

there is - in my eyes - no supporting evidence in the case of Push-T for this statement (more training data leads to lower sensitivity to sensing rates). Please consider either stating explicitly that you observed this phenomenon (only) for finger-spin or changing the claim to reflect more accurately the observations that you have made on more than one task.

This is fair, we can no longer edit the paper, but we will add qualitative evidence (policy rollout visualizations) to Appendix G in our next revision / camera-ready that will add further evidence for this statement in Push-T. However, we agree overall that the evidence for Push-T is not as clear-cut as for finger-spin. So we will revise Line 291 to say: “In line with the arguments above, agents with high model-capacity have lower normalized regret curves, i.e. they degrade less at lower sensing rate. This is also true for agents trained with more data, particularly on finger-spin. Push-T training data results are less conclusive (see Appendix G).”

Thank you once again for continuing to invest your effort into improving our paper. We believe our paper has grown significantly stronger through the process of responding to your comments. Please let us know if any concerns remain.

Thank you for such a detailed review! We provide responses to all your concerns below.

The paper studies entirely the wrong setting [for robots]… only an academic exercise with no practical use … tasks are not “robotic” … robots have imperfect sensing / stochastic dynamics …

We ask that you evaluate this paper not as introducing an immediately applicable engineering solution, but as the first empirical approach to address a core scientific question in robotics: “How critical is sensory information at each moment to a pre-trained policy?” See our global response above for further discussion on the relevance of our findings to real robotic tasks.

The findings within the “perfect world” assumption are unsurprising: in a perfect world, with perfect models, sensing etc. all tasks can be solved open-loop.

We agree that this is indeed not surprising, this is why we explain this concisely in Lines 243-258, and use that as motivation to study settings with imperfect models (or policies), due to limited policy capacity (see Ln 263 onwards). As such, almost none of our tasks can be truly solved open-loop, and the findings are only non-trivial because of this condition. To recap, our key findings are: In most tasks sensing only improves agent actions significantly at some rare moments during task execution, the reliance on environment feedback is similar for high-performing policies synthesized with different architectures, and the policy performance is inversely correlated with sensor dependence. Finally, note again that our experiments in this paper largely only study one axis of deviation (imperfect models) from the “perfect world (deterministic dynamics), perfect models, perfect sensing” base case, because these experiments are expensive to run, and we had to pick our battles. Do note in our global response we propose to study “imperfect sensing” in one task, to show that VoSI can be used to answer other questions beyond what we have demonstrated in this paper.

The tackled problem is only really relevant for real robots, usefulness limited in other domains. Even for robots, how much of a need for parsimonious sensing there really is, is debatable

We disagree. While we ourselves are motivated most by robotics applications, sensing often carries even more explicit costs in non-robotic decision-making settings. For example, when a government makes policy decisions about how best to serve its populace, it must perform a variety of expensive and time-consuming studies (such as censuses and surveys) to obtain “sensory observations” that determine its optimal policy decisions. Indeed, questions about the value of sensed information are studied outside of robotics too: for example, see our extended related work discussion section in Appendix F.

Coming to robots, parsimonious sensing is often of value because it entails a series of other resource costs (not only communication, but also computation, time, energy, …) in the robot, and robots are often resource-scarce, as dictated by size, weight, and power constraints. As the reviewer themselves points out, the “event-triggered control” community, as well as many of the works pointed out in Related Work Sec 2, study this question for this reason.

And finally, while we highlight the utility of VoSI characterization for efficient sensing, this is not its only use case. One might even view our approach as an extension to control of a kind of “interpretability” or “attribution” studies in other disciplines like computer vision and language. Just like they ask: “which parts of the image led the neural network to think that this is a cat / a spam email?” [1,2], one might use VoSI to ask “which sensory inputs most influenced the robot’s task performance?”

In the RL setting, using a policy that was trained closed-loop is problematic as you point out in l. 339. When instead training the policy with the parsimonious sensing, the policy will potentially change drastically to account for that. So the analysis is at best a proxy of what could be achieved under that setting.

Sec 2 lines 69-77 point out how we are different from methods that aim to generate policies under sensing constraints. We aim to characterize the value of sensory information for any learned controller, such that it might be relevant to executing the controller efficiently under any resource cost models or constraints. As such, we are not aiming to generate new behaviors that permit less sensing, just reproduce existing ones. Our proposed method reveals interesting insights into the properties of performant policies on various challenging robotic tasks of interest and serves as a general tool to understand the sensory requirements of any learned controller, complementing approaches that aim primarily to synthesize more efficient controllers.

Thank you for the kind words and feedback!

“Adding a few lines in the introduction on their contributions?”

Thank you, we have edited the manuscript to reflect the contributions clearly in the introduction. Further, we have provided a summary of our contributions in the global response above.

“Figure 9 is confusing”

Thank you, we did indeed find this figure particularly difficult to explain well within limited space, and will try to improve now. As you will recall, Figs 7 and 8 show VoSI(s,h) for a single state s while varying the horizon h. In Fig 9, we wanted to simultaneously vary not just the open-loop horizon h (from 1 to 100) but also the state s (from starting state $s_0$ to ending state $s_T$ of a closed loop policy rollout).

• This means we are now mapping a scalar function VoSI($s_{t_1}$, $h$) of two arguments (time index $t_1$ of last-observed state $s_{t_1}$, and open-loop horizon $h$). We use colors to indicate the VoSI values, and the two arguments are on the x and y axis.
• In particular, the x-axis of the figure represents the time index $t_1$ of the last-observed state $s_{t_1}$. The y-axis represents the offset open-loop horizon $t_1+h$.
• Each vertical column of colors corresponds to the VoSI profile for a particular state at varied horizons: this is a 1-D representation of the same kind of VoSI profiles depicted in Figs 7 and 8. To draw the VoSI profile for a particular last-observed state, say, $s_{t_1=10}$, one could look up the column for $t_1=10$. For example, for $h=1$, we can look up the color for $y=t_1+h=10+1=11$, and match it to the color chart on the right.
• The choice of adding the last-observed time offset to the y-axis i.e. using $t_1+h$ rather than just $h$ helps to highlight the structure in the evolution of VoSI over time. Tracking along a horizontal row, say $t_1+h=40$, you can read off how important re-sensing before timestep 40 was, at various prior moments in time: i.e. it was very important to re-sense if $t<20$ i.e., the agent had not yet observed t=20, corresponding to “high” colors. However, for $t>20$, there was no need to re-sense before $t+h$ = 40, corresponding to “low” colors.
• With this interpretation, we can now observe from Fig 9 that for all last-observed states $t_1<20$, VoSI profiles are consistent: they are low up to $t_2=20$ (no task-relevant information from sensing before timestep 20), and high afterwards (suggesting critical information at timestep 20). Likewise, VoSI profiles are also consistent for all last-observed states $t_1>20$: there is no more value to sensing any more, for ever after.
• Upon inspection, we find that timestep 20 in the closed-loop trajectory corresponds to a tipping point after which the ball is destined to fall into the cup (near the lip and with appropriate velocity), which means the task will be complete.

Statistical Rigor and plots with confidence interval(CI)s

Operating within space constraints, we dropped the confidence intervals to maintain legibility while making our plots smaller. We have now added larger plots with CIs in Appendix G (Figures 17-20). The trends pointed out in the paper remain robust.

As with any new scientific directions, all the practical use cases will only be established through many follow-up efforts (including our own that are already in progress), and our aim in this paper is mainly to establish VoSI measurements as a new probing tool capable of offering insights to researchers and practitioners. The experiments conducted in the paper are at once (1) extensive (they span more tasks and more state-of-the-art policy classes than in any prior related works, and even just the plots reported in the paper now required over 600 A40 GPU hours of compute), and (2) still limited relative to all potential uses for VoSI. In particular, we largely only explore one axis of deviation from the trivial “deterministic world, perfect models, perfect sensing” base case (where no sensing is required): where the models are imperfect. However, we fully expect that VoSI is also applicable in sensing-limited and stochastic-dynamics settings. As proof of concept, we had already applied such a noisy dynamics setup in the simple gridworld task in Sec 4.1. We had also argued there that limited policy expressivity for all practical purposes is equivalent to such noise, which is why we need to still sense the environment at many instances within our studies. We acknowledge these limitations already in Sec 6 in the paper.

(in progress) A new experiment with sensing noise: Finally, to further demonstrate that our analysis can be extended to settings with realistic sensing noise, we plan to report experiments with a policy that has a CNN-based visual state estimator operating from image observations for the cartpole swingup task.

Key Changes in the Paper:

• We have added a more thorough extended related works section (Appendix F) to discuss the adjacent literature.
• We also present in Appendix G our initial efforts in addressing reviewers’ suggestions and also include placeholder sections with the planned experiments that we are currently conducting.

Planned Experiments:

1. In response to the suggestion from reviewer Reqd, we redid the grid world experiments, corresponding to the how dynamics error necessitates more sensing (Figure 4 (right)) and the demonstration of the performance of a greedy-strategy informed by our analysis (Figure 11) , with an error model that sometimes moves the agent to a wrong but neighboring cell, rather than teleport it to any cell over the full grid. We show our results in Appendix G.2 in Figure 19 and Figure 20. All the performance trends reported in the paper continue to hold. We will move this into the main paper with suitable changes.
2. In response to the suggestion from reviewer 6eFu, we propose to conduct a proof-of-concept study demonstrating the applicability of VoSI in environments with realistic sensing noise. We plan to replicate our analysis with a policy that uses a CNN-based visual state estimator operating from image observations on the cartpole swingup task.
3. In response to the suggestion from reviewer Tq4f, we propose to characterize an event-triggered control inspired baseline that reduces sensing for control beyond the demonstrated simplistic greedy strategy leveraging VoSI profiles. We plan to define heuristic event-triggers based on course measurements of the state (e.g. a sensor that detects changes in sub-grids the agent is in) to define when high-fidelity state measurement is made to query for a new plan. However, note that different from our proposed method, this baseline relies on some amount of coarse sensory measurement at every timestep to trigger events.

References:

[1] Majumdar, Anirudha, Zhiting Mei, and Vincent Pacelli. "Fundamental limits for sensor-based robot control." The International Journal of Robotics Research 42.12 (2023): 1051-1069.

[2] Erdmann, Michael. "Understanding action and sensing by designing action-based sensors." The International journal of robotics research 14.5 (1995): 483-509.

[3] Dasari, Sudeep, et al. "RB2: Robotic manipulation benchmarking with a twist." arXiv preprint arXiv:2203.08098 (2022).

[4] Raffin, Antonin, et al. "A Simple Open-Loop Baseline for Reinforcement Learning Locomotion Tasks." arXiv preprint arXiv:2310.05808 (2023).

[5] Singh, Sumeet, et al. "Multiscale sensor fusion and continuous control with neural CDEs." 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.

[6] Mandlekar, Ajay, et al. "Mimicgen: A data generation system for scalable robot learning using human demonstrations." arXiv preprint arXiv:2310.17596 (2023).

[7] Nasiriany, Soroush, et al. "RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots." arXiv preprint arXiv:2406.02523 (2024).