Enhancing Tactile-based Reinforcement Learning for Robotic Control

We thank Xsn5 for taking the time to review our paper and we appreciate the helpful feedback.

[Xsn5-1] This work “introduces two dynamics objectives: tailored reconstruction and multi-step forward dynamics with auxiliary memory.“

To clarify, we introduce three objectives for training tactile-based agents (Sec 3.2):

1. Tactile reconstruction: decode the tactile state from the multimodal representation through binary classification.
2. Full dynamics: multi-step forward dynamics on the multimodal representation.
3. Tactile dynamics: multi-step forward dynamics on the multimodal representation and decode tactile state from each predicted future representation.

Tactile reconstruction (1) and tactile dynamics (3) are novel objectives, and the application of full dynamics (2) to tactile-based agents in model-free RL is novel.

[Xsn5-2] The proposed solutions for the hypotheses are not validated in the work… how does tactile reconstruction prevent premature convergence… how does dynamics help learn from sharp transitions?”

Our results (Figs 4, 5, 6) strongly demonstrate the efficacy of our methods, but we refrained from making definitive causal claims about the precise mechanisms underlying their success. For example, in Fig. 6 we observe that the vanilla PPO model significantly underperforms on the real-world metrics, perhaps lending some support to the theory that it has “converged prematurely”. In the same figure, we also observe that the addition of tactile signal and dynamics further improves performance on our benchmark tasks which feature “sharp transitions” between interaction states. We also attempted to provide additional insight into the learned representations through visualising future tactile states (Figs A5, A6) and performing latent trajectory analysis (Sec J) in the supplementary PDF.

To further validate these hypotheses we can provide the following analysis in the revised paper. Note, the inclusion or omission of these results do not take away from our central contributions, but instead serve to provide additional insight into the unique challenges in this underexplored problem setting. Thanks for the suggestion!

H1: Preventing premature convergence through tactile reconstruction

• Hypothesis: The tactile reconstruction objective forces the network to maintain a meaningful representation of tactile input throughout training, preventing it from being “down-weighted” prematurely in favour of other modalities.
• Experiment: We will track the L2-norm of the first layer weight vectors corresponding to proprioceptive and tactile inputs throughout training for each task and methodology. After normalising for input size, a larger contribution from the tactile input portion in agents trained with tactile reconstruction compared to RL-only agents will indicate that the model is extracting tactile features more effectively and from an earlier stage, preventing premature reliance on proprioception.

H2: Improving learning from discontinuous data through dynamics

• Hypothesis: By forcing the encoder to predict future state representations, the multi-step dynamics objective creates a more continuous learning signal, regularising learning from sparse and discontinuous tactile inputs.
• Experiment: We will track the gradient norms of the input data with respect to the policy and value function. Firstly, we will establish if training with tactile data does indeed introduce greater gradient instabilities compared to proprioceptive agents. Secondly, we will evaluate if our dynamics-based objectives lead to enhanced optimisation (evidenced by reduced magnitude or frequency in gradient spikes). In addition, we will provide visualisations of the loss landscape of each agent to support our quantitative results on optimisation stability.

[Xsn5-3] Clarity of Fig. 1 and its caption.

We thank the Xsn5 for highlighting these clarity issues, and apologise if our figure was difficult to follow. In light of the comments, we will update Fig. 1 to resolve any potential ambiguities (e.g., adding missing arrows between components). We will add a legend to clarify that rectangular blocks denote feed-forward networks or layers, squares denote vectors, text denotes scalars, variables with “hats” denote model predictions, and those without are observed. Finally, we will update the caption to the following:

(Top left): Each timestep, proprioceptive (prop) and tactile (tact) readings are concatenated to form the timestep observation $o_t$. A stack of the $k$ last observations forms the current state $s_t$. The encoder $e$ learns the state representation $z_t=e(s_t)$ which the policy and value function are conditioned on. During learning, the self-supervised loss is added to the RL losses and backpropagated in a single pass. The self-supervised loss optimises the encoder $e$ and its own task-specific networks ($d$, $f$, $p$).
(Top right): The tactile reconstruction objective optimises the encoder $e$ (and decoder $d$) to preserve the tactile state $s_t^{tact}$ in the representation $z_t$ by implementing reconstruction as binary classification with a binary cross-entropy loss $L_{BCE}$.
(Bottom right): The multi-step forward dynamics objective optimises the encoder $e$ (and forward model $f$ and projector $p$) to extract information from the state $s_t$ that will aid in predicting representations $n_f$ timesteps into the future.
(Bottom left): To further enhance learning, we propose to separate the RL memory and the auxiliary memory, which stores the last $N_{rollouts}$ of the RL memory.

We confirm Xsn5's understanding that the feed-forward direction of the blocks/squares on the right side of Fig. 1 is from left-to-right. To clarify, in tactile reconstruction $L_{BCE}$ is a binary cross-entropy loss (see Eqn 1) between the observed/ground truth tactile state (lower) and decoded/reconstructed tactile state (upper), not a MSE loss. We confirm that the dynamics loss is MSE (see Eqn 2), and we add that it is cumulatively summed for each timestep into the future: $L_1 + L_2 + … + L_{n_f}$. The computation of this loss is somewhat complex due to the need for a target network and projector (see L159-172 for full details), but we will strive to communicate this more clearly in the text and updated figure.

[Xsn5-4] Are the decoder, forward model, and the predictor used during inference?

These additional components are just used to assist with learning the encoder. Every network except the encoder and policy can be discarded for deployment.

[Xsn5-5] Potential fallibility of policy and value networks running into the same problem as encoder.

We confirm Xsn5’s understanding that yes, the self-supervised objectives are applied solely to the encoder (and relevant auxiliary networks). The policy and value networks are only optimised with the RL loss. The motivation of our approach is that through self-supervision, the encoder learns to generate a richer and more robust latent representation $z_t$ of the agent’s state. This “pre-processed” representation is then readily leverageable by the downstream policy and value function.

Empirical evidence for this enhanced representation is In Sec J of the supplementary PDF, where we show the PCA reduction of the learned 256-dim state representation $z_t$ across an episode rollout. Taking Fig. A13 as the example, the latent trajectory of the self-supervised agents is much richer and diffuse between timesteps compared to the RL-only agents. This distinctive latent space provides evidence that self-supervision has a strong influence on the learned representation, allowing the policy and value networks to operate in a space less susceptible to the challenges of sensory data.

Dear reviewer Xsn5,

Thank you for your reviewing efforts so far. Please engage with the authors' response.

Thanks, Your AC

We thank gp4F for taking the time to review our paper and for the great questions.

[gp4F-1] Paper summary and the entanglement of proprioceptive and tactile representations.

We appreciate gp4F’s summary of our work, but we would like to clarify a key potential misunderstanding. The summary states that we posit entanglement of proprioceptive and tactile representations as the problem. However, our premise is precisely the opposite, i.e., we hypothesise that the tactile observation often fails to adequately entangle or contribute meaningfully to the overall multimodal representation (see discussion on L31-40). Thus, the challenge we address is not the difficulty in attributing the benefit of tactile input, but rather the observed lack of expected performance improvement when tactile observations are concatenated to the observation.

Furthermore, while our tactile dynamics and tactile reconstruction objectives are designed to encourage the tactile state to be more salient within the multimodal representation, the key finding was that the most successful approach was the multi-step forward dynamics, which we denoted as “full dynamics” (see results in Fig 4).

[gp4F-2] Thorough hyperparameter search is performed.

We appreciate gp4F’s recognition of our thorough hyperparameter search. We fully agree that the common omission of such rigorous tuning in many RL for robotics works can undermine the reliability of reported results. To help address this challenge, we plan to release our integrated hyperparameter optimisation code with Isaac Lab (where it is not available) alongside our benchmark, which we hope will encourage and facilitate the adoption of best practices within the research community.

[gp4F-3] Additional related works.

We thank gp4F for highlighting the omission of these two tactile-based dynamics works (RoboPack and SwingBot). Our related work section primarily focuses on model-free RL with self-supervised learning. We had originally omitted SwingBot because the focus of that paper is about generalisation to unseen physical properties and it does not use RL, whereas we focus only on in-domain expertise with RL. Similarly, RoboPack uses trajectory optimisation after the dynamics model is learned, whereas we explore improving online policy optimisation with online dynamics learning. However, we appreciate that a discussion of achievements in learning tactile dynamics within other communities, particularly model-based policy learning (e.g., deep tactile MPC [A]), would indeed be valuable and provide a more comprehensive context for the reader. We will revise our related work section to include a discussion of these additional references.

[A] Tian et al., Manipulation by Feel: Touch-Based Control with Deep Predictive Models, ICRA 2019

[gp4F-4] Lack of sim2real experiments on real robots for dexterous manipulation.

We acknowledge this point regarding the absence of sim2real experiments, which was also a focus of some of our discussion with reviewer WBY5 (please see response [WBY5-[1-3]] for further context). We believe the absence of real-world experiments does not detract from our work’s core contributions for the following reasons:

1. Establishing a simulation baseline for novel tactile-based robotics. We address an important but largely unexplored domain, i..e, tactile-based robotic agents operating without visual input. Given the scarcity of prior work (see L41-50), our paper provides a crucial foundational benchmark for future research in this novel setting.

2. Solving a fundamental problem of neural network-based RL agents. Our paper's central contribution is a new methodology that markedly improves how agents leverage binary tactile observations in simulation (see Fig. 6). As demonstrated in the supplementary videos and latent trajectory analysis in Sec J of the supplementary PDF, agents using our method discover unique, effective strategies for complex, partially observable control tasks that traditional RL agents do not.

3. Robustness across diverse dynamics. To counter concerns about simulation-specific gains, we validated our method across three tasks with distinct contact dynamics and different simulated robotics hardware (Find: sparse, Bounce: intermittent, Baoding: continuous). The consistent high performance across these varied settings and robot morphologies (see Figs. 4 and 5), strongly indicates a genuine advancement in learning capabilities, rather than a mere simulation artifact.

4. Designed for sim2real transfer. Our design choices, particularly the use of binary contact activations, are deliberate steps to facilitate future real-world transfer. Prior work [61, 63] has demonstrated robust sim2real transfer with binary tactile activations even in dexterous manipulation tasks. Furthermore, our method's focus solely on modifying network weights simplifies real-world deployment. We contend that a superior simulated policy, assuming the sim2real gap is otherwise bridged for a tactile-based RL agent, should indeed transfer, and our validation across diverse tasks suggests our policies are not unduly reliant on unrealistic simulation physics as asked by WBY5.

We believe that our comprehensive empirical study in controlled simulated environments provides substantial findings and novel insights that will be of immediate interest and benefit to researchers working at the intersection of machine learning and robotics. Specifically, we demonstrate the proven efficacy of binary tactile activations for complex dexterous manipulation, addressing a key open question in the tactile sensing community. We posit that it is important to first establish the efficacy of our findings in simulation, via a repeatable benchmark that can be used by others, before moving on to real world robotics experiments in future work.

[gp4F-5] Losses depicted in Fig 1.

We apologise for the confusion caused by the notation in Figure 1 (right) regarding the multi-step forward dynamics. gp4F's assumption about it being a typo is understandable. To clarify, the subscripts L1​, L2​,…​ do not refer to different types of loss (e.g., L1-norm vs. L2-norm), but rather denote the loss calculated for the prediction at the first step (i.e., L1​), second step (i.e., L2​), and subsequent steps into the future from the current state. The specific loss function used for all these steps is consistently the L2 loss, as correctly identified by the reviewer and detailed in the multi-step forward dynamics subsection. We recognise that this notation is ambiguous and will revise Figure 1 and its caption (see response [Xsn5-3]) in the camera-ready version to explicitly clarify this.

[gp4F-6] Is any prioritized sampling used to reduce the imbalance present in the tactile state transitions for the forward dynamics model?

This is an interesting question. While we did employ positive weighting for tactile reconstruction to address sparsity (see L288), we confirm that no prioritised sampling mechanism was used for the dynamics learning. We will clarify this in the revised text.

During initial investigations, we experimented with various sampling approaches, such as rule-based strategies and exponential weighting. However, we did not observe significant performance gains in our specific experimental setup. We wish to emphasise that this outcome does not diminish the potential value of prioritised sampling in tactile dynamics learning. On the contrary, we think that it might be a critical component for handling tactile sparsity and imbalance. The exploration of prioritised sampling, particularly in conjunction with a separated auxiliary memory for transitions, represents a potentially interesting direction for future research.

[gp4F-7] Can the authors perform an ablation study on $n_f$?

We agree with gp4F that the impact of the number of forward prediction steps, $n_f​$, is highly interesting and critical for performance. To ensure robustness and account for potential interactions with other hyperparameters (e.g., learning rate), we treated $n_f​$ as an optimized hyperparameter within our search space, sweeping over values of {2, 3, 4, 10}. This approach is detailed in Appendix I, Table A3, and we will ensure this optimisation strategy is clearly stated in the main paper body.

Our findings revealed task-specific optimal values for $n_f​$. For the 'Find' and 'Baoding' tasks, optimal performance was achieved with $n_f$​ values between [2, 4]. Interestingly, for the 'Bounce' task, a longer sequence length of $n_f​=10$ proved most effective, likely due to its ability to capture a full bounce trajectory, as illustrated in Fig. A5.

Dear reviewer gp4F,

Thanks for your reviewing efforts so far. Please respond to the authors' response.

Thanks, Your AC

We thank WBY5 for their careful review and for the highly insightful comments!

[WBY5-1] Are the superhuman policies realistic?

We agree that any results for a dynamic contact-rich task in simulation must be critically examined. Based on our analysis, we do not believe our superhuman performance is due to exploiting simulation inaccuracies. We have several pieces of evidence for this:

1. Oscillatory impulses masquerading as bouncing: The maximum bounce count was 88 over a 600-timestep episode, which translates to approximately seven timesteps between each bounce. By slowing down the provided videos to 0.25x speed, it is evident that the SSL-optimised agent has learned a sophisticated strategy involving both distinct bounces (see timestamp 00:09) and a secondary, vibration-like behaviour (see timestamp 00:07), which together enable the superhuman performance. This strategy contrasts with the RL-only agent’s repetitive and less effective motions, suggesting our method has learned a superior behaviour rather than exploiting a simulation artifact.
2. Similar improvements in static tasks: Our method also shows significant performance improvements in the “Find” task, which is a completely static, non-dynamic environment. In this task, our agents found the target object on average 1.5 seconds faster than the baselines (see Fig. A1 in the supplementary PDF). This demonstrates that our method's benefits are not exclusive to dynamic environments, reinforcing the assertion that our core contribution is a more effective learning strategy rather than the exploitation of dynamic physics.

[WBY5-2] Is there value in optimising for superhuman performance in simulation?

We believe there is great value. See L275 to 278 for a discussion of our performance in the context of human baselines. A key tenet of modern reinforcement learning research, from Atari to AlphaGo, has been to push beyond human capabilities in simulation as a means of discovering novel and superior strategies. In our work, we were initially focused on the fundamental challenge of getting a "blind" agent to perform any meaningful action at all. The discovery that our methodology enables superhuman capabilities was an unexpected but powerful outcome, attesting to its ability to unlock creative solutions for challenging, partially observable control tasks.

[WBY5-3] What is the likelihood of sim2real transfer?

We believe that the benefits of our method should translate to the real world, assuming a successful sim2real gap can be bridged for the base tactile agent. This is because our core contribution is a method for improving the learning process and generating superior policies. If the physical assumptions are generally valid, these strategies should be transferable. Our deliberate choice of binary contact signals and a methodology that only modifies network weights were made to specifically facilitate this future transfer. Please see response [gp4F-4] for more discussion of sim2real.

[WBY5-4] Binary contact signals are limiting for more practical tasks.

The choice of binary signals is a deliberate one, driven by practical constraints and the state of sim2real research. While richer sensory data is theoretically advantageous, it incurs substantial computational costs (e.g., see discussion on L305) and faces a significant sim2real gap, even for continuous signals due to the difficulty of accurately simulating contact physics.

Binary tactile sensors, in contrast, are cheap, robust, and require minimal calibration (e.g. can employ a low pass filter and then threshold the voltage [A]). Our work demonstrates, for the first time, that it is possible to achieve superhuman performance using only these simple measurements. This finding not only challenges the perception that binary signals are limiting but also establishes a highly feasible and effective baseline for real-world tactile learning.

[A] Lee et al., DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity, IEEE RAL

[WBY5-5] Large variances for complex task policies.

There is indeed a large variance observed in the policies for the complex Baoding task (see Fig. 4). We attribute this variance to the difficult nature of this partially observable control task and the relatively sparse reward signal (Sec D3 in the supplementary PDF). A single successful rotation provides a large, sparse bonus (+10 compared to distance reward within [0, 0.1]), making agents vulnerable to this high-variance behavior. The ability to mitigate this challenge is a key contribution of our work. As shown in Fig. A3 (bottom left) of the supplementary PDF, the RL-only agents and the full dynamics agents both exhibit significant performance variance, with multiple seeds failing to achieve a successful rotation. However, our tactile reconstruction agent (i.e., “+tactile recon”) is the only method that achieves consistent, low-variance returns, with all seeds successfully learning the task. The ability of our approach to significantly reduce variance in this difficult setting is a notable and primary result.

[WBY5-6] No unified hyper-parameter setting for all tasks indicates challenges in generalization.

This is an interesting question. However, we believe that diverse hyperparameter settings across tasks are a necessary and a sound methodological choice. The tasks in our study (see Fig. 2) are intentionally diverse, encompassing distinct contact dynamics, devices, and control objectives. As is standard practice in reinforcement learning research, it is highly unlikely that a single set of hyperparameters would be optimal across such a varied set of problems. Thus, we followed the best practice (e.g., as in [B]) of performing a hyperparameter search for each method on each task.

This approach is crucial for validating the true performance of our method. The fact that our methods consistently outperform the baselines after both are optimally tuned for each specific task provides stronger evidence for its efficacy. We will release our training code, including the hyperparameter search configurations and final values to facilitate reproducibility and future research. We will also clarify the hyperparameter settings in the revised text.

[B] Eimer et al., “Hyperparameters in Reinforcement Learning and How To Tune Them”, ICML 2023

Dear reviewer WBY5,

Thanks for your reviewing efforts so far. Please respond to the authors' response.

Thanks, Your AC

We thank zxcc for their comments and constructive feedback. As a reminder, we introduce a new approach that improves sensory-driven robotic agents for complex manipulation tasks that does not make idealised assumptions about the quality of tactile state information available. In addition, we introduce three new environments for benchmarking tactile-based manipulation methods.

zxcc’s concerns primarily relate to the structure of the paper, missing details, and the lack of experiments on real robots. With the exception of the last one, which we attempt to justify why its absence does not impact our contributions, the other concerns can be addressed by updating the final camera ready text.

[zxcc-1] Organization and presentation of the paper.

We thank the reviewer for highlighting these clarity issues, and will make the following changes in light of zxcc’s comments:

• Notation: While all symbols are sequentially introduced in the text, we recognise that a central reference would be helpful. We will add a summary of the most important notation at the beginning of the Method section to serve as a quick reference for the reader.
• Figure 1 caption: We will update the caption for Fig. 1 (see response [Xsn5-3]) so that it is more self-contained and will expand it to provide a more detailed step-by-step description of our method's pipeline in the camera-ready version.

[zxcc-2] Training objective and role of self-supervised learning.

The total training objective is given in L147: “The self-supervised loss is added to the policy, value, and entropy loss and backpropagated in a single pass. The auxiliary loss optimises the encoder e and its own task-specific networks.” We did not include this in the original figure because it is the default optimisation strategy in self-supervised RL, and is not a novel contribution for our work. In the camera-ready version, we can explicitly add the loss summation for improved clarity.

[zxcc-3] Quantitative results are presented as figure instead of table.

In the main body of the paper we chose to display results as a figure of the agent’s mean evaluation return throughout learning, because we are interested in understanding the impact of incorporating self-supervised objectives throughout the optimisation process, and figures provide more information on sample efficiency and stability. For a more direct comparison, we provided a bar chart in Fig. 6 to show how different methodologies correspond to improvements in physical quantities. In the camera-ready version, we can add a table of the highest obtained mean evaluation returns for each methodology.

[zxcc-4] Missing implementation details about the proposed method.

Our problem setting is outlined in Sec 3.1, the self-supervised objectives in Sec 3.2, and auxiliary memory is described in Sec 3.3. Additional implementation details are provided in Sec 4 and Sec E, F, and I of the supplementary PDF. We will revise the descriptions and more clearly point to the additional implementation details in the main paper. If there are specific aspects of the implementation details that are not clear, we are happy to discuss them during the author-reviewer discussion period - please do not hesitate to ask additional questions.

[zxcc-5] Self-supervised training has been used in previous works

Yes, we agree, and do not claim in the paper that the use of self-supervised learning in combination with reinforcement learning is novel. In the related work (Sec 2) we summarise how different self-supervised training strategies have been adopted over the past decade. Indeed, reconstruction and dynamics-based objectives have been widely adopted in previous works, because of their general-purpose nature. Please see the summary of our contributions in the introduction (Sec 1) and response [zxcc-7] for the novelties we claim.

[zxcc-6] The separate auxiliary memory is an important contribution but only briefly discussed.

We thank zxcc for highlighting the separate auxiliary memory as an important contribution. The description is brief because the method itself is intentionally simple, and we have described the implementation in its entirety in Sec 3.3. The contribution is not an unnecessarily complex new algorithm, but rather the insight that decoupling the auxiliary memory from the on-policy buffer and significantly expanding its size can be a simple yet powerful technique (see results in Fig. 5).

To be concrete, for an agent using $B$ parallel environments and a rollout length of $R$, the on-policy data has a shape of $[B, R, ...]$. Our method simply involves storing data for the auxiliary tasks in a larger, separate buffer of size $[N_{rollouts}, B, R, ...]$, where $N$ is the number of past rollouts we keep in memory.

[zxcc-7] Sim2real gap and real world experiments.

We wish to clarify that our paper does not claim the proposed method has a low sim2real gap; rather that we designed our methodology with this long-term goal in mind. As stated in our Introduction (L62), “We desire the following characteristics… (d) low sim2real gap: inspired by the low-cost setups in [61, 63], we study learning with binary tactile activations… to avoid transfer difficulties that can come with continuous measurements [61]”. We also acknowledge in our Limitations (L314) the absence of real-world experiments and express we "hope our choice to study binary tactile activations greatly minimises the potential sim2real gap”.

We believe the absence of real-world experiments does not detract from our work’s core contributions for the following reasons:

1. Establishing a simulation baseline for novel tactile-based robotics. We address an important but largely unexplored domain, i..e, tactile-based robotic agents operating without visual input. Given the scarcity of prior work (see L41-50), our paper provides a crucial foundational benchmark for future research in this novel setting.
2. Solving a fundamental problem of neural network-based RL agents. Our paper's central contribution is a new methodology that markedly improves how agents leverage binary tactile observations in simulation (see Fig. 6). As demonstrated in the supplementary videos and latent trajectory analysis in Sec J of the supplementary PDF, agents using our method discover unique, effective strategies for complex, partially observable control tasks that traditional RL agents do not.
3. Robustness across diverse dynamics. To counter concerns about simulation-specific gains, we validated our method across three tasks with distinct contact dynamics and different simulated robotics hardware (Find: sparse, Bounce: intermittent, Baoding: continuous). The consistent high performance across these varied settings and robot morphologies (see Figs. 4 and 5), strongly indicates a genuine advancement in learning capabilities, rather than a mere simulation artifact.
4. Designed for sim2real transfer. Our design choices, particularly the use of binary contact activations, are deliberate steps to facilitate future real-world transfer. Prior work [61, 63] has demonstrated robust sim2real transfer with binary tactile activations even in dexterous manipulation tasks. Furthermore, our method's focus solely on modifying network weights simplifies real-world deployment. We contend that a superior simulated policy, assuming the sim2real gap is otherwise bridged for a tactile-based RL agent, should indeed transfer, and our validation across diverse tasks suggests our policies are not unduly reliant on unrealistic simulation physics as asked by WBY5.

We believe that our comprehensive empirical study in controlled simulated environments provides substantial findings and novel insights that will be of immediate interest and benefit to researchers working at the intersection of machine learning and robotics. Specifically, we demonstrate the proven efficacy of binary tactile activations for complex dexterous manipulation, addressing a key open question in the tactile sensing community. We posit that it is important to first establish the efficacy of our findings in simulation, via a repeatable benchmark that can be used by others, before moving on to real world robotics experiments in future work.