Response

Thank you for your valuable comments. Hope that our answers can address your concerns.

The experiment part may not fully match the motivation of the paper. Generally speaking, simulation environments such as Mujoco do not require the use of fragmentary control.

Fragmented interactions exist in many virtual scenarios. For example, remote control of NPCS: In the game, the cloud server needs to control the terminal NPC in real time. NPC stalling can significantly degrade the user experience. So we follow works with related evaluation scenarios (e.g.RTAC[1]RDAC[2]). We used mainstream virtual scenarios (Mujoco, D4RL) to simulate remote control of NPC tasks instead of using Mujoco directly. The details of constructing fragmented interactions simulation tasks are described in the experiment section Sec 4.2: In real-world tasks, an execution time for one step is about 0.05s to 0.25s. And most of the prohibited interaction duration is 0.5s to 2s. Thus, we set the interval step length as 8 to simulate the real-world setting. We will highlight the task settings in the new version. And put it at the top of the experiment section.

[1] Chris Pal, et al. "Real-time reinforcement learning." NeurIPS (2019).

[2] Ramstedt, Simon, et al. "Reinforcement learning with random delays." arXiv preprint arXiv:2010.02966 (2020).

The real-world robotic control experiment lacks important details, including the interaction pattern between the executor and the agent in the real-world experiment.

The 21-joint snake robotic uses a rolling gait. The algorithm is arranged in the cloud server, and the decision end interacts with the snake robot through the local area network. This is the interaction pattern figure.

Snake robot tracking is a periodic information acquisition task. The reason for this phenomenon is not packet loss or delay, but the unique rolling gait of the snake robot. This gait of the snake robot is the most efficient gait for outdoor work, which includes the rolling of the head sensor. However, this gait causes the next observation to be acquired only when the head sensor rolls to a position parallel to the ground. If the rolling motion is interrupted midway, the next state cannot be obtained, leading to ineffective shaking of the robot.

The snake robot needs to scroll from the starting point to the target point in a scene with a size of 5 square meters. Each observation step is 54d, and the action sequence time step is 20, so the action sequence dimension is 1.08k. The control frequency of the real interaction is 20 Hz. We describe the snake robot tracking task in detail and provide several visualization results in the new version.

Why does the method significantly outperform TD3 with advanced decision?

Explanation:

• Td3-advance decision needs to output the whole action sequence (concatenate c steps), and the long action sequence will increase the output dimension (output_dim=c * single_action_dim). This increases the difficulty of the action space exploration. Thus, the agent cannot learn the optimal policy (mentioned in the introduction).
• Our method represents diverse action sequences in low-dimensional space. Our method reduces the difficulty of exploration and performs better.
• Ours is a plug-in method and can be applied to all major reinforcement learning methods. To ensure experimental fairness, all methods use Td3. This eliminates the framework advantages that Td3 brings to advance deisicion.

Verified by experiment:

• Following results show the sensitivity of three methods to the action sequence dimension. The first table runs on Walker and the second runs on Maze-hard.
• The exploration ability of Td3-advance decision decreases significantly with the increase of sequence length.
• TD3-frameskip has better exploration ability than TD3- advanced decision. This is because frameskip reduces the spatial dimension of exploration by sacrificing action variety (repeating the same action).
• Our method has the best performance by reducing the dimension of exploration space and ensuring the diversity of action through the representation of action space.

If our reply addresses your concerns, we would appreciate it if you could kindly consider raising the score.

We think that the supplementary experiment is inadequately described. Please Allow us to analyze it again.

Why does the method significantly outperform TD3 with advanced decision?

Explanation:

• Td3-advance decision needs to output the whole action sequence (concatenate c steps), and the long action sequence will increase the output dimension (output_dim=c * single_action_dim). This increases the difficulty of the action space exploration. Thus, the agent cannot learn the optimal policy (mentioned in the introduction).
• Our method represents diverse action sequences in low-dimensional space. Our method reduces the difficulty of exploration and performs better.
• Ours is a plug-in method and can be applied to all major reinforcement learning methods. To ensure experimental fairness, all methods use Td3. This eliminates the framework advantages that Td3 brings to advance deisicion.

Verified by experiment:

• Following results show the sensitivity of three methods to the action sequence dimension. The first table runs on Walker and the second runs on Maze-hard. Walker's single-step action dimension is 3. To analyze the sensitivity of the methods to the action space dimension, we set the number of decision steps as follows: 4 (dim = $4\times3$), 8 (dim = $8\times3$), 12 (dim = $12\times3$). Maze-hard's single-step action dimension is 2. This navigational task requires the agent to change its action in a specific state (e.g., change the directionto to avoid running into a wall). Excutor with actions repeating hard to get to the end of the maze, and therefore fail to get the maximum reward resulting in an inaccurate Q-value fit. Thus, when the action sequence of the policy is internally homogeneous, long decision steps lead to training instability. We set the number of decision steps as follows: 9 (dim = $9\times2$), 18 (dim = $18\times2$), 27 (dim = $27\times3$).
• The exploration ability of Td3-advance decision decreases significantly with the increase of sequence length.
• TD3-frameskip has better exploration ability than TD3- advanced decision. This is because frameskip reduces the spatial dimension of exploration by sacrificing action variety (repeating the same action).
• Our method has the best performance by reducing the dimension of exploration space and ensuring the diversity of action through the representation of action space.

Response

Thanks for your valuable advice on robot learning. Hope that our answers will address your concerns.

Details of snake robot experiment.

Snake robot point tracking is a periodic information acquisition task. The reason for this phenomenon is not packet loss or delay, but the unique rolling gait of the snake robot. The rolling gait of the snake robot is the most efficient gait for outdoor work at present, which includes the rolling of the head sensor. However, this gait causes the next observation to be acquired only when the head sensor rolls to a position parallel to the ground. If the rolling motion is interrupted midway, the next state cannot be obtained, leading to agent-ineffective shaking. Our control system is the first to complete the rolling gait tracking task. We describe the snake robot tracking task in detail and provide several visualization results in the new version. The snake robot needs to scroll from the starting point to the target point in a scene with a size of 5 square meters. Each observation step is 54d, and the action sequence time step is 20, so the action sequence dimension is 1.08k. The control frequency of the real interaction is 20 Hz.

In addition to robot scenarios, fragmented interactions also exist in many virtual scenarios. For example, remote control of NPCS: In the game, the cloud server needs to control the terminal NPC in real-time. NPC stalling can significantly degrade the user experience. So we follow works with related evaluation scenarios (e.g.RTAC[1]RDAC[2]). We used mainstream virtual scenarios (Mujoco, D4RL) to simulate remote control of NPC tasks. Our method outperforms in these tasks.

[1] Chris Pal, et al. "Real-time reinforcement learning." Advances in neural information processing systems 32 (2019).

[2] Ramstedt, Simon, et al. "Reinforcement learning with random delays." arXiv preprint arXiv:2010.02966 (2020).

It is usually good to include demo videos.

Thanks for your advice. We are willing to provide the visualization results, we provide a demo video and several visual descriptions of the task in the new version.

If our reply addresses your concerns, we would appreciate it if you could kindly consider raising the score.

Response

Thanks for your recognition of our work. We hope our reply can address all your concerns.

The questions are also related to weakness, what are the differences between FIMDP and POMDP, or MDP with reward delays?

We compare FIMDP with reward delayed MDP and POMDP in the new version:

• The main difference between FIMDP and delay MDP: In FIMDP All environmental information is delayed (observation, action sequence, reward). Besides, agents are not allowed to pause midway. However, in reward delay MDP agents just need to address the reward delay. And reward delay MDP does not consider the harm caused by the agent stalled in the middle.
• POMDP does not involve delay, the agent gets a local observation at each step. In contrast, the FIMDP has a delay in obtaining observations, but the agent can obtain global observations

2. If we have/learned a world model for the environment, can we make the model-based predictions, like model predictive control to solve the FIMDP? (this might be the most straightforward method that first came into my mind.) How does this compare to learning the multi-step representations?

This question is valuable. In fact, we originally wanted to build a world model to achieve comprehensive environmental perception. However, the interval of FIMDP is too long and some tasks have random intervals. It is difficult to construct effective dynamics models and reward models under delayed conditions, which can be summarized as we cannot model random delays effectively. Therefore, we chose a method that is easier to train and robust to delay: construct an action latent space. To prove that our approach is more efficient than the model-based approach in FIMDP tasks. The following results show the performance of the original MBPO[1] based world model method (Average of the 10 runs. Interval is 10). In the future, we will further think about how to make Model-based methods effective in FIMDP tasks.

[1] Janner, Michael, et al. "When to trust your model: Model-based policy optimization." Advances in neural information processing systems 32 (2019).

If you think the above response addresses your concerns, we would appreciate it if you could kindly consider raising the score.

Response

Why is clustering representations of actions that have similar environmental effects better than clustering action sequences with similar values or rewards?

Explanation is as follows (highlight in new version):

• Get accurate Q value is difficult: In sparse reward environments (such as FIMDP), reward and Q are difficult to obtain and the evaluation of Q values in the early stage of training is inaccurate. In contrast, environmental dynamic is more reliable and accessible.
• Environmental dynamic contains more information: The same reward or Q value may correspond to different environmental changes, but the same environmental change must have the same reward or Q value.
• Environmental dynamic is reward-agnostic: In FIMDP, rewards are sparse. Environment dynamics do not require a per-step reward. Therefore, environmental dynamic representation is more robust in FIMDP.

Further, we compare these 3 representational learning methods. We only changed the clustering representations.

Is the number of decision steps fixed within one trial? How will MARS perform if trained with longer action sequences but executed with a shorter interval, compared with training with shorter action sequences?

The number of decision steps is fixed. The following results show that reducing the number of steps as much as possible can improve scores (Average of the 10 runs. Interval is 6). As the number of steps increases, both the sequence dimension and the reward sparsity increase. These lead to the difficulty in exploration.

How to guarantee that the actions actually executed by the executor strictly match the policy's output, do the collected trajectory samples inherently lack stationarity?

We use the physical clocks on both devices(a common method in real-time control, hightlght in the new version). If the actions are obsolete, lose them. We retain the time stamp and execution flag of each action, which makes actions executed in strict accordance with the timestamp order. When the new sequence arrives at the executor, the previous sequence will be replaced, and the execution flag of the unexecuted action will be False. Each latent space action reward is the sum of the executed action reward in the corresponding sequence. Results show the effect of the alignment method (average of the 10 runs, Interval is 6).

MARS has a more advantage over frame-skips in simple tasks than in complex tasks?

Because the VAE hyperparameters of MARS are not optimized in difficult tasks. We adjust the VAE hyperparameters while keeping the remaining hyperparameters unchanged for both methods （following table). By doing this MARS has a more pronounced advantage on complex tasks. (Old score of MARS in maze-Hard:315.2±13.9 maze-Hard:6417.3±317.3)

MARS has better stationarity.

Explanation:

• Frameskip leads to internal homogeneity of the action sequence and the inability to change the action at key states. Thus, the policy is unstable.
• Advance decision needs to output the whole action sequence (concatenate c steps), which will increase the output dimension. This increases the difficulty of the action space exploration. Thus, the agent cannot learn the optimal policy.
• Our method represents diverse action sequences in low-dimensional space. RL algorithms only need to learn policies in the latent action space. Our method reduces the difficulty of exploration and performs better.

Further, we compare the policy stability of all methods for sequence length. Ours provides high stability.

Does the decoder need to be run on the execution device? Does this mean that latent representations will also be lost?

We do not deploy the decoder to the executor. The agent makes c step decision based on each timestep information, so each time step (t+i) will receive c times in the future. Besides, our decision steps are set at maximum intervals to ensure that the executor receives the next sequence before the previous sequence is completed.

Should the "Musar" be referred to as "MARS"?

Correct this in the new version.

If our reply addresses your concerns, we would appreciate it if you could kindly consider raising the score.