DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays

[Q5] It would help readers if you briefly stated how Seq2Seq effectively handles delays.

[A5] For addressing the delay issue, the Seq2Seq model in the paper serves two purposes: encoder pretraining and policy training. In the encoder pretraining phase, the dataset from the delay-free environment is initially randomly reconstructed to create information state sets with various delay sizes and their corresponding delayed state sequence sets. Specifically, assuming time $t$ and delay $d$ where $d \in [d_I, d_I+d_M]$, with $D=d_I+d_M$, the information state is denoted as $I_t = (s_{t-d}, a_{t-d},a_{t-d+1}, \cdots, a_{t-1})$，and the delayed state sequence is $g_t=(s_{t-d+1}, \cdots, s_{t-1})$. For information states with a delay smaller than $D$, to facilitate batch training, these information states are zero-padded until their length reaches $D$. Next, a Seq2Seq encoder-decoder model is employed to predict the state based on the input information state and output the delayed state sequence. Specifically, a multi-layer perceptron encodes each element of the information state to generate corresponding embeddings. These embeddings are then fed into a GRU module to generate a hidden feature vector of a dimension determined by hyper-parameters. Finally, the Seq2Seq model is optimized using MSE loss to improve the accuracy of state sequence prediction, thereby refining the context representation of the information states. These hidden states are the ones utilized for policy learning. During the policy training phase, the agent interacts with the environment in real-time, causing the length of the information state $I_t$ to dynamically change due to delays. We employ the pretrained Seq2Seq encoder, where information states of varying lengths $I_t$ are fed into the encoder. After several iterations, the encoder extracts delayed state information and maps it to fixed-dimensional context representations. Subsequent policy training relies on these context representations rather than the original state space. Moreover, there is no need for additional adjustments to the chosen policy algorithm due to changes in delays.

[Q6] First line of page 2: What does “dynamic environments” mean?

[A6] “dynamic environments” means non-deterministic environments. The corresponding sentence is revised as: While effective in deterministic environments, their adaptability in nondeterministic environments needs further improvement.

[Q7] What the difference between d_M and D? The seem to represent the same quantity.

[A7] $d_I \in Z^{+}$ denotes the intrinsic delay value resulting from long-distance transmission or heavy data transfers. Meanwhile, $d_M \in Z^{+}$ represents the maximum number of additional dropping steps, ensuring that $d_I + d_M$ remains within the agent’s capacity. The sum of these values is defined as $D = d_I + d_M$.

[Q8] The paper mentions the computation drawback of existing approaches. How does DEER compare computationally to the baselines considered?

[A8] To address this inquiry, we conducted a comparative analysis of the time and performance of four algorithms—DEER, RLRD, DATS, and SACAS. Specifically, within the Hopper task, a fixed delay of 4 was established, with each algorithm undergoing three runs of 1 million environmental steps using different seeds. All algorithms were executed on the same NVIDIA GeForce RTX 4090 graphics card, and the time and performance comparison are illustrated in the Table.

[Q9] Modification in paper: figure 1 and 3 are essentially the same, advantage (1) seems to subsume advantage (2), the meaning of tasks involving delays” and “delay issues” in the Introduction, making “ Offline assisted Online RL” section more readable, GRU not defined at its first occurrence and last paragraph at section 4 would fit better at beginning of paragraph 2 in section.

[A9] The aforementioned content will be updated in my newly uploaded PDF.

We are glad that the reviewer found our work is especially relevant to real-world RL tasks and can be used with any RL algorithm. We hope to provide clarification to address the concerned issues. For the convenience of the reviewer, we reiterate the comments below.

[Q1] It's unclear why breaking the end-to-end decision process into an encoding phase and a decision-making phase increases interpretability. I don't believe this claim is supported by the current experiments. The paper should clarify this contribution.

[A1] Information states consist of the most recently observed states and their corresponding action sequences. The SACAS method utilizes information states as decision states to design value and policy functions. When there is significant delay, the dimensionality of the action part within the information states can be much larger than that of the states themselves. The SACAS method operates as a black-box process, disregarding the mentioned issue and completing training end-to-end, with the quality depending on the neural network's capacity and parameter settings. However, in the initial phase of our experiments, the delayed state information corresponding to delayed actions is actually known. Using this information, an encoder-decoder can map the information states into a latent space. This process, unlike SACAS's end-to-end random exploration of network parameters, is more directive, compelling the model to start from a good state representation. The improved interpretability is reflected here.

[Q2] Since DEER essentially learns a dynamics model that predicts a sequence of states given an initial state and sequence of actions, my understanding is that DEER still suffers the same drawbacks as model-based approaches.

[A2] Model-based algorithms typically involve constructing a dynamics model through real-time interactions with the environment, followed by planning using this model. Consequently, these algorithms are often vulnerable to model and iteration errors. DEER, however, leverages existing data from delay-free environments to learn an encoder-decoder for multi-step prediction. During the decision-making phase, DEER employs the encoder's representation of input states and action sequences, termed as context representation, rather than the final state from multi-step predictions. This representation extracts decision-relevant information from the most recent observed states and action sequences, thereby mitigating sensitivity to deviations from the true state caused by model and iteration errors.

[Q3] DEER seems to have an unfair advantage over the baselines in Figures 4 and 5. In particular, DEER assumes access to an offline dataset, but the baselines do not. I believe DEER could in principle train its encoder in an online fashion -- if so, then an online variant of DEER would provide a fairer comparison across algorithms.

[A3] Due to constraints in computational resources and time, our comparative experiments were exclusively performed on the Hopper and Walker2d tasks, each configured with a delay set at 4. The online version of DEER encompasses two primary modules: an encoder-decoder and a decision-making component. Initially, the encoder-decoder network is randomly initialized, and training data is gathered during real-time interactions within the environment. The encoder-decoder is updated every 300,000 steps. Regarding the decision-making aspect, the SAC framework is employed, wherein the replay buffer stores context representations derived from the encoder's information states. The entire training duration extends to 1 million steps. The comparative results are shown in the following table:

[Q4] The paper should explicitly define "augmented states. I believe "augmented state" refers the concatenation of state and action information, but this term makes me think of a synthetic data generated by a data augmentation function (e.g. a random cropping and recoloration of an image state).

[A4] Thanks for the suggestion. We replaced all "augmented states" with "information states".

Has our response sufficiently addressed your inquiries? Are there any additional problems requiring discussion? We eagerly anticipate further correspondence with you. Please let us know if there are any additional comments, questions, or concerns. Many thanks again for the invaluable hard work of all reviewers to improve our work.

[Q6] Why the performance of DEER is sensitive to dimension of latent representation? In particular, why dim=512 results in significantly worse performance in Walker2D (Figure 6)? My expectation is large dimension should work better or similar.

[A6] The context representations are obtained by encoding the information states through a pre-trained encoder, capable of depicting delayed state information, thus their quality significantly impacts the final performance of the algorithm. The dimensionality reflects the representational capacity of the context representation, making DEER sensitive to dimensions. In the literature mentioned by the reviewer, OFENet explicitly highlights that simply increasing the dimensionality of the decision space doesn’t necessarily lead to improved final performance, as performance is also influenced by network architecture design, regularization, and auxiliary task design. DEER conducts pre-training before policy training, and the encoder structure remains unchanged throughout the subsequent process, differing from OFENet's update framework. Additionally, in the walker2d task, a decrease in performance is observed with 512 dimensions, likely due to overfitting during the encoding and decoding in the pre-training phase.

[Q7] Minor: including the size of font in Figure 3, the color of linestyle in Figure 6 and Figure 7, a table summarizing the numerical reults and A typo in the title "resillient".

[A7] The aforementioned content will be updated in my newly uploaded PDF.

[Q4] How did the authos obtain the result of the baseline models? E.g., DATS used less training samples (environmental steps) than 1e6 in their paper. Also, How many steps of actions are augmented in the state in SACAS? What are the hyperparameters and how did the authors selected them?

[A4] DATS uses fewer training samples, but during training, the probabilistic ensemble network in the dynamics part and the sampling method in the planning part can lead to an increase in the wall-clock time. Moreover, from the results of experiments with random delays, there is a moderate improvement in the final performance with increased steps in the experiments. The code repository for DATS can be found at [1], using default parameters. However, to reach 1 million environment steps, adjustments were made for the number of training instances based on the maximum steps per episode for different tasks: 1000 iterations for ant, halfcheetah, hopper, swimmer, and walker2d, while reacher was set to 20000 iterations. RLRD has high sample utilization but takes more time when converting eligible offline trajectory segments into current policy segments. The code repository for RLRD can be found at [2], using default parameters. SACAS is derived from the SAC algorithm, with the code available at [3], using default parameters. In cases where the environment encounters a fixed delay $d$, we modified the input dimensions of the SAC policy network and value function network to be the sum of the dimensions of the original state space |S| and the original action space |A| scaled by the delay $d$, denoted as |S| + d|A|. When the environment encounters a random delay $z_t$, we modified the input dimensions of the SAC policy network and value function network to be the sum of the dimensions of the original state space |S| and the maximum delay |$d_I+d_M$| times the action dimensions |A|, expressed as |S| + |$d_I+d_M$||A|.

[Q5] In the end of the first paragraph, it was mentioned minor delays markedly amplifies the risk in self-driving. Is there any reference? Following this question, are there more examples (better with data) of practical scenarios that suffers heavily from delays?

[A5] In the '5G + Intelligent Transportation' series white paper (link provided below [4]), clear indicators are specified for the latency requirements in four scenarios of intelligent driving, as detailed below: Scenario Assisted Driving Remote Driving Autonomous Driving Vehicle Convoy

Additionally, in literature [5], practical observations indicate a delay of 0.2 to 0.5 seconds between steering signals from the vehicle's computer and the actual steering wheel operation. While the vehicle remains in motion, this leads to substantial deviations in path tracking, potentially resulting in a loss of control when traveling at high speeds. Furthermore, literature [6] notes that hydraulic vehicle braking systems typically require over 0.4 seconds to achieve the necessary deceleration, significantly impacting the planning and control modules of Connected and Autonomous Vehicles (CAVs). In various other domains such as control of robotic systems [7], distributed computing [8], and management of traffic networks [9], delays have demonstrated adverse effects on the performance of corresponding intelligent agents."

Reference

[1] https://github.com/baimingc/dambrl

[2] https://github.com/rmst/rlrd

[3] https://github.com/pranz24/pytorch-soft-actor-critic

[4]http://13115299.s21d-13.faiusrd.com/0/1/ABUIABA9GAAglcTHjQYojIWZswY.pdf?f=%E4%B8%AD%E5%9B%BD%E8%81%94%E9%80%9A5G+%E8%BD%A6%E8%BD%BD%E5%AE%9A%E5%88%B6%E5%8C%96%E6%A8%A1%E7%BB%84+%E7%99%BD%E7%9A%AE%E4%B9%A62019.pdf&v=1639047701

[5]赵建辉, 高洪波, 张新钰, 等. 基于时间延迟动态预测的自动驾驶控制[J]. 2018.

[6]Bayan, Fawzi P., et al. "Brake timing measurements for a tractor-semitrailer under emergency braking." SAE International Journal of Commercial Vehicles 2.2009-01-2918 (2009): 245-255.

[7]Imaida, Takashi, et al. "Ground-space bilateral teleoperation of ETS-VII robot arm by direct bilateral coupling under 7-s time delay condition." IEEE Transactions on Robotics and Automation 20.3 (2004): 499-511.

[8]Hannah, Robert, and Wotao Yin. "On unbounded delays in asynchronous parallel fixed-point algorithms." Journal of Scientific Computing 76 (2018): 299-326.

[9]Dollevoet, Twan, et al. "Delay propagation and delay management in transportation networks." Handbook of Optimization in the Railway Industry (2018): 285-317.

We are glad that the reviewer found our work important and thoght it was a good attempt. We hope to provide clarification to address the concerned issues. For the convenience of the reviewer, we reiterate the comments below.

[Q1] The novelty of DEER is moderate. From my understanding, the key novelty is a framework combining "offline-assisted online RL" and "Encoder in RL".

[A1] The reviewer has indeed grasped the core concept of the DEER method. However, summarizing DEER as a mere fusion of "offline-assisted online RL" and "Encoder in RL" is somewhat simplistic. Firstly, DEER stands as the pioneering approach that bridges data from delay-free tasks with applications in delayed tasks. Secondly, DEER doesn't necessitate modifications to subsequent policy algorithms; it seamlessly applies existing algorithms. Lastly, although DEER utilizes a simple Seq2Seq model, context representation derived from its encoder effectively extracts delay-related information, significantly bolstering decision-making capabilities.

[Q2] The definition of RDDMP is quite complex and probably unrealistic. One concern is about the random delay value $z_t$ with dropping probablity $\mu$. $z_t=d_I$ if $z_{t-1}=d_I+d_M$. which means the $d_M$ steps of delay suddenly "disappear". How can this happen in reality? My suggestion is to modify it in another formation.

[A2] The reviewer's statement is accurate. In our defined scenario, when the cumulative count of consecutive frame drops reaches $d_M$, it resets. There are two perspectives to explain this setup: 1.The probability of dropping frames, $\mu$, for each instance, compounded over $d_M$ consecutive drops, equates to $d_M^{\mu}$. This value tends to be relatively small, signifying that it is an infrequent occurrence. 2.In real-world scenarios, when an agent encounters $d_M$ consecutive instances of not receiving a state, it surpasses the agent's capacity to handle delay. Hence, it necessitates the remote end to ensure the appearance of the next state. The proposed scenario by the reviewer is indeed more reasonable but requires a slight modification: when the agent observes the returned state, the current delay reverts to $d_I$, representing the intrinsic delay value resulting from long-distance transmission or heavy data transfers. Hence, $z_t = d_I$ if $z_{t-1} > d_I$ with a probability of $\mu_{catch}$. Based on this stochastic delay setting, we compared four algorithms (DEER, RLRD, DATS, SACAS) in the Hopper and Walker2d tasks, setting $d_I=2$, $d_M=4$, and $\mu=0.2, 0.4, 0.6$. Each algorithm was experimented with three different random seeds for 1 million environment steps in each environment configuration. The comparative results are shown below.

From the table above, it's evident that even under the requested random delay settings by the reviewer, the performance of DEER significantly surpasses other algorithms (although there might be instances where it closely aligns with RLRD scores). This underscores the significance of state representation in decision-making, further emphasizing that the pre-trained encoder in the DEER algorithm indeed extracts delay-related information from the information state.

[Q3] DEER should also be compared with vanilla SAC in delay=0 case. This is important to understand the effect of pretrained encoder in deep RL and probably explain why DEER significantly outforms expert in Swimmer.

[A3] Whether in fixed or random delay settings, the legend labeled "Expert" in the graph corresponds to the performance of vanilla SAC trained with delay=0, as depicted in the Figure 4 and Figure 5 .

Has our response sufficiently addressed your inquiries? Are there any additional problems requiring discussion? We eagerly anticipate further correspondence with you. Please let us know if there are any additional comments, questions, or concerns. Many thanks again for the invaluable hard work of all reviewers to improve our work.

We hope our response has satisfactorily addressed your queries and concerns. Please do not hesitate to reach out if you have any further questions or require additional clarification. We are truly grateful for the invaluable contributions made by the reviewer, which have significantly enhanced the quality of our work.

In [A1], we revisited and elaborated on our innovations. Within [A2], under the random delay setup suggested by the reviewer, we conducted diverse algorithmic comparisons on Hopper and Walker2d. [A3] delineates the comparison between DEER and vanilla SAC. [A4] explicates the application specifics of the Baseline method. [A5] illustrates the hazards of delays in real-world scenarios. [A6] analyzes the reasons behind DEER's sensitivity to dimensions. We've also updated the graphs and charts in the newly uploaded PDF. We sincerely look forward to engaging in further discussions with you.

We thank the reviewer for the useful and detailed comments. In the following, we aim to address your questions and concerns. For the convenience of the reviewer, we reiterate the comments below.

[Q1] The proposed method relies on encoding the augmented state with embeddings. When the state space is large and high-dimensional, further augmentation can be computationally inefficient. Authors should clarify whether higher-dimension embeddings are necessary as a context representation to achieve the desired performance. It is also desirable to have experiments that assess how the performance (w.r.t. return) is affected by the size of embeddings, and whether higher-dimension yields significantly better performance. When a compact / low-dimensional embedding is used, can it still generalize well to different delay scenarios.

[A1] From our experimental results, mapping the information state to a higher-dimensional context representation resulted in better performance of the trained policy in delayed environments. Moreover, the improvement in performance does not have a direct proportional relationship with the increase in dimensions. In other words, higher dimensions do not necessarily yield better results. To clarify this conclusion further, we extended our experiments on the Hopper and Walker2d tasks by encoding the information state into dimensions similar to the original information state (32 and 64 dimensions) under a delay of 4. Subsequently, we employed the SAC framework and the same set of parameters for policy training. The final outcomes of the encoders with different dimensions on the Hopper and Walker2d tasks are presented in the table below.

Based on the data in the table above, it's evident that when the dimension of DEER's pre-trained encoder is too small, such as setting it to dim=32, the encoder's capability is too weak to effectively represent delay information within the information state, resulting in decreased performance. Conversely, when the dimension of DEER's pre-trained encoder is excessively large, for instance, with dim=512, although the encoder has numerous parameters and enhanced capabilities, there is a risk of potential overfitting to the current dataset, leading to decreased performance. Moreover, as the dimension shifts from 64 to 256, the performance improves for both tasks, indicating that the representation capability of the context representation indeed influences the final outcome.

[Q2] The current method is a direct application of the encoder-decoder framework based on Seq2Seq scheme. Algorithmically, the novelty is limited. The authors are suggested to highlight what are the technical novelties in their algorithmic framework.

[A2] The technical novelties in DEER are as follows: Firstly, DEER stands as the pioneering approach that bridges data from delay-free tasks with applications in delayed tasks. Secondly, DEER doesn't necessitate modifications to subsequent policy algorithms; it seamlessly applies existing algorithms. Lastly, although DEER utilizes a simple Seq2Seq model, context representation derived from its encoder effectively extracts delay-related information, significantly bolstering decision-making capabilities.

[Q3] The authors utilize an undelayed offline dataset to pretrain the encoder. In this case, how can it capture the delays in the embeddings of the states when deployed to the delayed environments?

[A3] The paper only discusses communication delay within the tasks, where the misalignment between the observed states by the agent and the executed actions leads to the current delay environment. We transform delay-free offline datasets to match different delay sizes within the delayed environment. Subsequently, we employ a Seq2Seq encoder-decoder model to predict states in an autoregressive manner. The encoded information state, referred to as the context representation, is then used as the decision-making state. In the delayed environment, at time $t$, the information state is defined as $I_t = (s_{t-d}, a_{t-d},a_{t-d+1}, \cdots, a_{t-1})$ with the corresponding representation as $c_t$, and the decision action as $a_t$. If the subsequent time $t+1$ observes the state $s_{t-d+1}$, the information state updates to $I_{t+1} = (s_{t-d+1}, a_{t-d+1},a_{t-d+2}, \cdots, a_{t})$ with the corresponding representation as $c_{t+1}$. However, if the state s_{t-d+1} is not observed, the information state updates to $I_{t+1} = (s_{t-d}, a_{t-d},a_{t-d+1}, \cdots, a_{t})$ with the corresponding representation as $c_{t+1}^{‘}$, and the decision action as $a_{t+1}$.

Has our response sufficiently addressed your inquiries? Are there any additional problems requiring discussion? We eagerly anticipate further correspondence with you. Please let us know if there are any additional comments, questions, or concerns. Many thanks again for the invaluable hard work of all reviewers to improve our work.

We hope our response has satisfactorily addressed your queries and concerns. Please do not hesitate to reach out if you have any further questions or require additional clarification. We are truly grateful for the invaluable contributions made by the reviewer, which have significantly enhanced the quality of our work.

We are glad that the reviewer characterized our work as “clear” and “reasonable”. We thank the reviewer for the useful and detailed comments. In the following, we aim to address your questions and concerns. For the convenience of the reviewer, we reiterate the comments below.

[Q1] Currently no baselines use the offline dataset, but DEER uses offline data to train the seq2seq model. Some offline-to-online methods should be included as baselines.

[A1] In response to the reviewer's inquiry, we undertook a comparison between the state-of-the-art algorithms PEX [1] and DEER. PEX's core approach involves initially learning a policy from an offline dataset, utilizing this learned policy as a candidate. Subsequently, another policy takes charge of further learning. Both policies interact with the environment in an adaptable manner. While training the offline policy, PEX requires reward information to guide its learning, in addition to states and actions. In contrast, DEER only relies on states and actions.

Within the MuJoCo environment, we conducted policy training using PEX across six continuous tasks (Ant, HalfCheetah, Hopper, Reacher, Swimmer, and Walker2d), incorporating delays of 1, 2, 4, 6, and 8. Each scenario underwent experimentation using three different seeds. The offline policy training dataset for PEX aligns with that of DEER. Offline policies for various tasks were trained over a day. Presently, the online training has persisted for two and a half days, set to conclude tonight. The comparative results between PEX and DEER will be updated here upon completion of the training. The training outcomes for the Ant, Hopper, Walker2d, and HalfCheetah tasks have been updated. The comparison between the training results of PEX and DEER for these three tasks is shown in the table below.

Analysis of the data in the table indicates that the final training outcome of PEX closely resembles that of an agent employing a random strategy. The subpar performance is likely associated with the utilized offline dataset, primarily constituted by a significant majority of random trajectories alongside a limited number of expert trajectories. This composition within the offline data has resulted in PEX learning a strategy leaning towards randomness.

[Q2] Why does pretrained encoder have a very specific requirement (generated by a random policy along with a few expert trajectories) on the offline trajectories? Offline dataset’s influence on the model performance is not discussed and analyzed. Offline dataset plays a critical role in DEER, so it should be analyzed.

[A2] The reason behind structuring the offline dataset for pre-training the encoder to include a substantial amount of random trajectories and a few expert trajectories in the delay-free environment is to reflect a more realistic scenario, where random trajectories are predominant while expert trajectories are comparatively scarce. To address the reviewer's inquiry regarding the impact of the offline dataset on the model, we introduced two additional types of offline datasets in the Hopper and Walker2d tasks, both with a delay of 4: one composed entirely of random trajectories and the other exclusively of expert trajectories. The subsequent policy training procedures align with DEER's methodology. The comparative outcomes from training with these three distinct offline datasets are presented in the table below.

The experimental results from the Walker2d task in the table reveal an intriguing aspect: the quantity of expert trajectories doesn't directly correlate with superior final results. Essentially, the efficacy of DEER's pre-trained encoder seems to hinge closely on the distribution of state transitions within the provided dataset.

Has our response sufficiently addressed your inquiries? Are there any additional problems requiring discussion? We eagerly anticipate further correspondence with you. Please let us know if there are any additional comments, questions, or concerns. Many thanks again for the invaluable hard work of all reviewers to improve our work.

We hope our response has satisfactorily addressed your queries and concerns. Please do not hesitate to reach out if you have any further questions or require additional clarification. We are truly grateful for the invaluable contributions made by the reviewer, which have significantly enhanced the quality of our work.