We sincerely appreciate your positive evaluation, especially noting the intuitiveness of our approach. In response to your observations about weaknesses, below we provide answers in detail.

Weakness1. "A key question here..."

We agree that learning forward and inverse dynamics models has additional computational cost but emphasize that superior sample efficiency is a critical goal in deep RL. Like common techniques such as data augmentation or auxiliary losses, this added computation directly enables efficient learning with fewer samples and better final performance. Furthermore, our method seamlessly integrates with existing approaches that have already trained dynamics models, offering enhanced performance without incremental computational cost in such cases.

Weakness2. "I can't help but wonder..."

We appreciate your insightful suggestion about sophisticated ways of leveraging time reversal symmetry and we agree that it is an interesting topic. One possible approach is to incorporate the temporal symmetry into the neural network architecture design, similar to steerable CNNs handling spatial symmetry like rotation equivariance. Moreover, we suspect that the 2x speedup may be too optimistic. This would require idealized conditions rarely met in practice, including perfectly reversible dynamics and accurate backward policy execution.

Thank you for noting the quality and clarity of our work. We are happy that you recognize that the model design leading around our idea feels like the "shortest path". Below, we address each question thoroughly.

Q1. "While the work..."

We need to clarify that our method specifically learns an 'action inverting model' implemented via an inverse dynamics model. Unlike Barkley et al.'s approach [1] of simply negating actions in reversed transitions—which is not always valid—our model derives reverted actions through a learned inverse dynamics. However, for transitions without full time reversal symmetry, the inverse dynamics model may still generate invalid actions. To address this, we introduce a forward dynamics model to filter out such invalid transitions.

Q2. "What would the results..."

In the sparse-reward setting, exploring tabula rasa is a very challenging task for the RL agent. Consequently, agent performance without demonstrations drops substantially, which is a key reason that we do not include these experiments in the manuscript. On challenging tasks such as door opening/closing outward, nut assembly/disassembly, and block stacking/unstacking, the agents fail to make progress under sparse rewards alone. However, on other tasks like door opening/closing inward and peg insertion/removal, the IQM of success rate (shown in the tables below) reveal that the performance gap between TR-DRL and baseline methods widens significantly without demonstrations, confirming that time reversal symmetry provides guidance and partially compensates for missing demonstration data.

"Single-Task SAC": baseline; "+reversal aug": trajectory reversal augmentation with dynamics-aware filtering; "+reversal reward shaping": time reversal symmetry guided reward shaping.

[1]. Barkley et al. "An Investigation of Time Reversal Symmetry in Reinforcement Learning." https://arxiv.org/abs/2311.17008

Dear Reviewer Pawe,

We appreciate your positive feedback and increased confidence in our paper. Thank you again for your reply.

Best, Authors

Thank you for indicating our contributions and providing thoughtful feedback on our work. In this work, we focus on robotics tasks with time reversal symmetry, and evaluate our proposed method in simulation. We definitely agree that applying our approach in a real-world setting is an important step to prove the effectiveness of our method for solving those time-symmetric robotics tasks. And it is an interesting direction to learn the temporal symmetry if it does not obviously exist in tasks. We leave the real-world experiments and learning temporal symmetry as future work. Below, we answer each question in detail.

Q1. "Can the time reversal..."

We would like to clarify that our method exploits two types of symmetry: full time reversal symmetry (FTR) and partial time reversal symmetry (PTR). Peg insertion/removal and door opening/closing inward are examples of FTR and PTR task pairs respectively. In door opening/closing inward, most reversed transitions are filtered out by the dynamics-consistent filter, since the agent cannot reverse the action (i.e., from "push the door" to "grasp the handle and pull the door") without first grasping the handle. The detailed examples of partial reversibility can be found in Example 2 of Section 4.1. We design a time reversal reward shaping mechanism to handle these partially reversible transitions. A potential model is trained with successful trajectories of task A and used to guide the agent training of task B through potential-based reward shaping, as detailed in Section 4.3.

Q2. "Given that the paper..."

As stated at the beginning of the response, we admit the significance of real-world experiments and leave it as future work. Considering the difficulties of conducting RL directly under real-world setting, a common practice is to train the policy with RL in simulation and deploy the policy in real-world by solving the sim-to-real gap. Our approach can effectively facilitate the learning process during the simulation phase.

Q3. "In partially reversible tasks..."

For tasks where only subsets of states exhibit time reversal symmetry, reversed trajectories obtained from trajectory reversal augmentation often violate true environment dynamics, leading to low quality data that may even hurt agent training. Instead, our method leverages time reversal guided reward shaping to exploit the partial reversibility. Intuitively, for a high-reward trajectory from one task (door from open to closed), if a trajectory from another task can achieve the reversed object states (door from closed to open), it should likewise receive a high reward. This motivates us to employ reward shaping to guide the RL agent with these partially reversible transitions, as detailed in Section 4.3. Furthermore, we choose to use potential-based reward shaping as it preserves policy optimality and train a potential model so that we can fully utilize all successful trajectories.

Thank you for summarizing our work and providing thoughtful feedback that helps us strengthen both the presentation and technical depth of our paper. Below we answer your questions in detail.

Weakness1. "Baselines..."

As detailed in related works (Section 2), there are several other ways to integrate time-reversal information.

Some existing works exploit reversibility from goal states to enhance exploration of the agents, similar to what we would like to achieve through the time reversal guided reward shaping technique. Our method differs from them in that our proposed trajectory reversal augmentation technique leverages time reversal symmetry not only from goal states but globally for every transition in the trajectory, enabling a broader application of time symmetry across the entire state space.

Other prior works are designed to avoid irreversible actions of the agent for safer exploration. One example is "learning a reset policy alongside the normal policy to prevent agents from entering non-reversible states, ensuring safety in exploration phase and achieving better training efficiency". Unlike their reset policy that sets the initial state as the ending state of the current policy and the goal state as the task's starting point, our method treats two reversible tasks independently, with initial and goal states defined separately for each task. Moreover, our method is orthogonal to theirs and can be integrated to enhance the training of their reset policy.

Weakness2. "Time reversal examples..."

We have updated our manuscript to recall state decomposition proposed by this work when we introduce the example of partial time reversal symmetry. The answer to your question in Weakness2 is: Yes. By our definition in Section 4.1, partial time reversal symmetry indicates that a subset of the whole state is reversible. In the example of door opening/closing inward, it means that the door's trajectory can be reversed by some action.

Minor1. "existing work..."

Thank you for highlighting this important connection. We have expanded the related work to explicitly include how data augmentation methods improve noise robustness and cited these two works as key examples.

Minor2. "Line 33:..."

Yes, "the speed of action execution" is the frequency at which actions are selected. This is often controlled by a hyperparameter called "action repeat" in RL methods. A larger value of "action repeat" indicates a lower control frequency.

Minor3. "Figure 2:..."

Thanks for this constructive suggestion. We have updated our manuscript accordingly.

Minor4. "Line 105:..."

As defined in line 110, time reversal symmetry applies to probability distributions over transitions. Consequently, stochastic dynamics does not interfere with reversibility, as it can remain statistically consistent under time reversal.

Minor5. "Line 111:..."

To avoid confusion, we have explicitly defined the overhead arrow notation as time reversal operation on $\mathcal{S}$ or $\mathcal{A}$ at line 110 of our manuscript.

Minor6. "Line 115:..."

Thank you for highlighting this important point. As noted by Barkley et al. (cited at line 110), such time symmetries commonly emerge in physical systems. We have now emphasized this point explicitly near line 115 in our manuscript.

Minor7. "When defining..."

Both $f_{\mathcal S}$ and $f_{\mathcal A}$ are used in Section 3 for involutions over states and actions. We choose to stick with the current notations for inverse and forward dynamics models ($h$ and $g$) to avoid misunderstanding.

Minor8. "Figures 5 and 6..."

Due to page limit in the main text, we combine the results of all methods into a single plot. We will include separate plots for each algorithm pair (TR vs. no TR) in the appendix for better readability.

Question1. "most reversed transitions..."

We empirically validate this point through quantitative analysis. We have tested our filtering method on the agent's saved replay buffer, which confirms that only 27.27% of transitions in door closing inward task pass filtering versus 63.13% for door closing outward task. Specifically, when closing a door inward, transitions of the gripper approaching the door can be reverted, while the transitions of the gripper pushing the door fail to pass filtering due to their violations of learned environment dynamics.

Question2. "The training process alternates..."

We would like to clarify that data collection (agent interacting with environments) in our method alternates between two tasks purely for implementation convenience. It is expected to be more efficient to interact with the two environments in parallel. During agent update step, we sample transitions from replay buffers of both environments simultaneously, and update all models with these transitions.

Question3. "Do the authors..."

Time reversal augmented data differs from policy generated data in quality and utility. One advantage of policy generated data is that it is 100% realistic and obeys environment dynamics, while time reversal augmented data, even after filtering, may not be fully dynamics-consistent. Conversely, there are two advantages of time reversal augmented data. First, it could help improving agent exploration at the beginning of the training, since the trajectories from two tasks cover distinct regions of the state space. This synergy is validated by the results shown in Figure 22 of our manuscript, where doubling the amount of policy generated data can not reach the performance of time reversal augmented data in this case. Second, the agent may converge at different speeds on the two tasks. A successful trajectory from one task can effectively guide the training for the other.

Limitations. "Yes, but I think it's worth mentioning..."

We fully agree with this point and have explicitly included the following sentence in the limitations section of our manuscript. "Like other data augmentation methods, our approach relies on prior knowledge of task structures—specifically, time reversal symmetry in our case."