Addressing Signal Delay in Deep Reinforcement Learning

“In the beginning of Section 3, baseline DRL methods are tested in delay signal settings and they are demonstrated to struggle in such a setting. Reasons are guessed for why they fail. However, I do not think that the failure of these baselines methods are studied well enough. I would like to see more ablation study on understanding how exactly they fail, which I believe will greatly help readers understand in what way DOMDPs are different to POMDPs and what are their unique challenges. Is it really due to the signal delay? Or like in general POMDPs, the RNNs cannot come up with good representations for decision making? For example, if these methods struggle to learn representations that support effective decision making, then one can test the accuracy of using these learned representations to predict the true current state. There are possible causes mentioned in paper but I find them rather vague and are not enough to motivate the proposed algorithmic changes.“

We appreciate the reviewers' suggestion to conduct a more thorough investigation into the failure of baseline methods in DOMDP settings. In response, we have performed additional experiments to better understand these failures and their implications for distinguishing DOMDPs from POMDPs. Our findings are summarized below:

1. RNN can learn good representations. We show additional experiment results to investigate in Sec. G.2 of the Appendix. The representation loss in VRM with delay (e.g. even -1.505 with a 12-step delay) close to the -1.91 (when there is no delay), suggesting that representation learning is not the primary bottleneck.

   Delay	Walker2d	HalfCheetah	Hopper	Ant	Average
   0	-1.87 ± 0.31	-1.60 ± 0.15	-2.50 ± 0.07	-1.68 ± 0.21	-1.91
   1	-1.90 ± 0.20	-1.74 ± 0.10	-2.54 ± 0.05	-1.99 ± 0.43	-2.04
   2	-2.25 ± 0.24	-1.70 ± 0.20	-2.62 ± 0.10	-1.89 ± 0.21	-2.10
   4	-2.26 ± 0.06	-1.65 ± 0.22	-2.49 ± 0.09	-1.81 ± 0.24	-2.05
   8	-1.92 ± 0.18	-1.57 ± 0.08	-1.81 ± 0.08	-1.67 ± 0.20	-1.74
   12	-1.86 ± 0.14	-1.29 ± 0.20	-1.56 ± 0.14	-1.31 ± 0.10	-1.50

2. Redundant or insufficient information in input to the policy or critic network.

   Our new experimental results shows that the proper amount of historical information is important. Excessive history can negatively impact performance, as shown in the following table:

   Column is # of delay $\Delta t$; Row is Historical num $H$ $\downarrow$	$\Delta t=0$	$\Delta t=1$	$\Delta t=4$
   $H=0$	97.7±24.0	87.8±22.3	47.6±20.3
   $H=1$	104.5±18.4	100.2±16.5	77.7±15.4
   $H=2$	98.8±23.4	93.7±18.9	84.3±16.3
   $H=4$	86.8±18.6	81.6±23.4	80.3±17.8
   $H=8$	68.3±12.5	73.0±13.6	58.6±15.5
   $H=12$	62.6±14.3	60.9±12.8	55.0±8.3

   ** We highlight the lines in the error range of highest one, to be consistent with Tab.1.*

   We investigate this by changing, $H$, the number of past actions of input to see how this affects the performance when delay $\Delta t$ is fixed to $\{0,1,4\}$.

   Here, we present the result using both “Delay reconciled training for critic” and “State Augmentation - MLP”, but with different number of histories $H$. Note that he model is powered with all the effective tricks in our paper, with only the $H$ varies.

   We can observe that incorporating no past actions ($H=0$) leads to suboptimal performance. For instance, with a delay of $\Delta t=4$, the absence of historical actions ($H=0$) resulted in a relatively low performance score of 47.6. Introducing a moderate amount of past actions significantly enhances performance, as evidenced by the improved score of 84.3. However, it's important to note that overloading the model with an excessive history (too high $H\geq 8$) reversely impacts the efficiency. This was clear when an even larger history led to a reduced performance of 55.0 ($H=12)$. Same trend can be found in all columns that the middle performance is higher than these of two sides.

   This finding also provides a potential explanation for the limited success of RNN structures in our experiments, as RNNs inherently lack control over the extent of historical information utilized. So it brought redundant information such as state history and extra action history outside of $\Delta t$. The result also embraces our theoretical finding that only historical actions in $\Delta t$ is enough to keep the Markovian property in Theorem 4.1.

   We hope this would also support our response to your another question, “common practice of POMDP that introducing historical state and actions” can not solve DOMDPs well.

   This is an important observation, wee add new experimental results and analysis in Sec. G.3 of the Appendix.

“Then in Section 4.2, the authors propose to give the actor past actions to help it infer the state at the current step. I don't understand why is this not done by default. In my understanding, DOMDPs are POMDPs and in POMDPs, past actions and observations should always be given to the policy for optimal control. I don't see how this is an innovation.”

We agree that using observation and action history is did a common practice in POMDPs. But we want to address the following facts:

1. The “common practices” can not solve Delayed RL well: Although using historical observations and actions is a standard approach in POMDPs, our findings suggest that this strategy is less effective in DOMDPs. For instance, as demonstrated in Tab.1, the state-of-the-art POMDP method 'RNN Strong', which adheres to this common practice, using both action and state history, significantly underperforms compared to our proposed method. This discrepancy indicates that the “common practices” does not fully address the challenges posed by DOMDPs.

2. We show that Historical Actions and only historical actions in the delayed window are useful: Our research shows that not all historical information is helpful. Instead, only the actions within the delay window is useful. This is a departure from the “common practices” of incorporating all historical data. More specifically, We show that theoretically (Theorem 4.1):

   • Historical states do not improve to performance in DOMDPs.
   • Only actions within the delay window are helpful.

   This selective incorporation of historical data not only save time and space cost. More importantly, it leads to a notable improvement in performance. We provide experimental evidence to support this claim, which also addresses your subsequent question about the RNN model so we defer there.

The method that is proposed in Section 4.3 has also been explored for the general POMDP settings. For example, in [4,5,6], people have explored auxiliary losses (to predict the next observation, state, reward) for improving the representation learning of RNNs when learning optimal policies in POMDPs.

Thanks for pointing these related work out. Here, we do not intend to claim this is an innovation, since, as you said, this is has been explored for the general POMDP settings. We just want to do some investigations here to help future studies, as part of our paper.

We list it as a special Section because we think auxiliary predictive loss is an intuitive and important part of DOMDPs. It is intuitive solution to predict the delayed observation to solve DOMDPs. And there naturally raised two quesions:

1. Choice of Representation: Is it more effective to use encoded features directly or to rely on a predictive head?
2. Network Architecture and Supervision: Is it preferable to have a distinct prediction network, or should we RL loss for more effective supervision?

We try to figure out when and where we can do it in Tab.1 and Tab.2 in Appendix G.1.1, which holds importance as reference for future studies.

Thank you for your insightful comments and for raising the concern regarding our paper. We appreciate the opportunity to clarify our contribution in this regard.

“My biggest concern on this paper is that many of the proposed techniques, if not all, are not new, and the paper misses a large body of closely relevant works.

We apologize if our paper gave the impression that these techniques were newly proposed by us. We recognize the importance of acknowledging previous work and have updated to properly reference earlier studies where these techniques were first introduced and applied (see Sec.4.1 last paragraph, Sec.4.3 first paragraph).

Our main contribution is not in creating brand new RL algorithms, but in systematically investigate effective framework for RL with delays, which can be applied to solve DOMPDs as a plug-in.

For your concerns about every technique you mentioned below, we are not simply use it, but either provide theoretical guaranteed effective modifications or comprehensive study of possible variants. We believe our work our work contribute to the research in the field of singal delay in RL, providing valuable insights and practical solutions.

In the following, we are going to address your detailed concerns about these techniques one by one to explain how we contribute to each one.

In Section 4.1, the authors propose to condition the critic on the true current state (without delay) to improve learning. The idea of using priviledged information during training by conditioning the critic on the ground-truth state in actor critic methods has been widely explored by both the POMDP and MARL communities. For example, see [1] and [2] for the former case and [3] for the latter case. Section 4.1 basically describes a trivial application of this idea to DOMDPs, which are special cases of POMDPs.

It is true that using “ground-truth state” is a widely explored We address the following facts:

1. There has been extensive studies about delayed RL while they all miss to use this “ground-truth state” technique. We are the first to deploy it in delayed RL and the show that the performance improvement is significant. The techniques itself may be not novel but investigating it and reveals the robustness is an important step for Delay DRL study. Investigating and reveals this comprehensively is of a small part of our investigation of techniques which is potential to solve delayed RL.
2. Existing SOTA POMDP methods can not solve delayed RL well. We has conduct experiment with “RNN Strong” and VRM, both are SOTA methods in POMDP studies. But their performance improvement are both marginal. So, simply treating DOMDP as a special case of POMDP and using existing methods can not be a solve the problem perfectly.

Furthermore, the authors mention that the efficay of such asymmetric architecture has been demonstrated by past previous studies. However, in [2], it has been shown that conditioning the critic (only) on the true state is fundamentally questionable as it leads to biased gradients.”

This is a great point. Conditioning the critic (only) on the true state is asymmetric and could lead to bad performance.

To address this, in our paper, we not only implemented the "Oracle Guiding Critic-MLP" but also conducted an ablation study with the "Symmetric-MLP" approach, using observations and historical actions (see Tab.1, final paragraph of Section 4.1).

We recognize there's a trade-off where the oracle critic can provide valuable information but also introduce bias. In this way, experimental investigation under delayed RL becomes crucial.

Our experimental results, as shown in the table below, indicate that the benefits of the information provided by the oracle critic generally outweigh the potential bias. We can see that “Oracle-Critic -MLP” consistently outperform the “Symmetric-MLP”, and the performance gap is not trivial, which is 38.6 v.s. 75.9 on average.

These findings are consistent and we believe they will be valuable for future applications and research in this area.

We recognize that our paper's initial presentation is not clear. To address this, we've revised Section 4.1's last paragraph for better accuracy and included relevant paper you mentioned.

“I also find lots of experimental details are missing from the paper, such as the architectures of networks, and the hyperparameters. Regarding hyperparameters, it would also be useful to know how are the hyperparameters tuned for each baseline and the proposed method to ensure fair comparison.”

Thank you for your feedback. We've addressed these concerns in Appendix Sec. [TODO] and summarize key points here:

• Network Architectures: For all proposed algorithm variants, we maintained a consistent structure. We compared different architectures like RNNs, MLPs, and Transformers, ensuring similar parameter counts. For baselines, we strictly followed the architectures from their original source code.
• Hyperparameters: We conducted extensive sweeps to ensure fair comparison. For each (algorithm, delay) pair, we select the best-performing hyperparameters.

“Also, for the baselines, such as DDPG, TD3, SAC, are they also RNN based? It would be strange to use a non-recurrent baseline in a POMDP environment. And are the past actions passed to the actors in these baseline methods? “

In Table 1, the versions of Vanilla DDPG, TD3, and SAC we used are not RNN-based. These vanilla implementations serve primarily to show the performance decline in DOMDP environment if nothing is done for the mainstream RL algorithms, rather than as competitive baselines.

While we did provide the RNN based version, which is “RNN Strong Baseline [1]” (line 4th in Table.1). The author proposed to use RNN to beat other SOTA methods in POMDPs. So this implementation of RNN is very well tuned, thus capable to serve as a confident baseline of RNN version of SAC.

[1] Recurrent model-free RL can be a strong baseline for many POMDP.

“In Figure 3, I find it weird to argue that baseline methods are failing by saying that there is a significant drop in performance from no delay to one-step delay. With different delay steps, we essentially have different POMDPs, which may inherently have different difficulty levels and different optimal performance. As such, the drop in performance I think is not sufficient to say the baseline methods are failing. However, a comparison to the optimal performance or performance from a better method would work.”

We acknowledge the points you've raised and have repainted the Fig.3:

1. Consistent Delay Comparison: We now display the performance of all methods under the same delay step in a single figure. This approach ensures a more equitable comparison by eliminating the variable of different delay steps, which, as you noted, can introduce varying levels of difficulty.
2. New Performance Benchmark: To provide a clearer evaluation of the baseline methods, we have added a benchmark line (dashed black) indicating the performance of our proposed method ("Ours") under the same delay. This serves as a reference for understanding the relative performance levels and the impact of the delay. We also mark the relative performance drop related to Ours.

Please refer to the updated Figure 3 in the revised manuscript.

We appreciate again for your comments, which help to improve the manuscript significantly.

My main concern is the real-world application. Delayed systems investigation is motivated from the real systems, so I think the approach/theory developed for delayed systems should be examined through real-world experiments such as robotic, autonomous vehicles rather than just artificial simulations.

We value your emphasis on real-world applicability and acknowledge the importance of testing our approach in practical scenarios such as robotics and autonomous vehicles. However, due to constraints in time and available equipment, conducting experiments with actual robots and vehicles was not feasible within the rebuttal stage timeframe.

To address this limitation and align our research closer to real-world condition, recognizing a key difference between MoJoCo's deterministic nature and the inherent randomness of real-world settings, we have introduced elements of randomness into our simulations. This adjustment aims to bridge the gap between our controlled environment and the unpredictability of real-world applications.”

$^*$ We choose “State Augmentation - MLP” in Tab.1 as an example implementation of “Ours”.

** In our results table, each cell contains {Value}({Relative%}), where {Value} is normalized in the same manner as in Tab.1 of our paper. The {Relative%} indicates the relative reward compared to the highest value of each line.

We can see that the performance of Vanilla SAC significantly decreases with the introduction of delay, while our method substantially improves performance in delayed scenarios.

We understand that this is not a real-world environment, while we did make progress to make our experiment can be close to real-world.

1. We study both fixed and unfixed random delay (Tab.1, Tab.2)
2. We study probabilistic environment. e.g. noisy and sticky actions (G.1.3), and noisy version of Mujoco above (have been add to G.1.2).

We would like to emphasize that all results presented in our paper, as well as those highlighted in the response above, consistently demonstrate that our performance is significantly superior to other baselines.

We hope these updates and clarifications will address your concerns to some extent. We remain committed to improve our work by considering your valuable suggestions.

“ I would suggest the authors to survey and (briefly) compare more control papers in this field because control theory and reinforcement learning are highly related and share similar concerns.”

We agree that the control related work worth investigation in the DOMDP topic, we will take time to do a comprehensive survey and add discussion in the paper.

Thanks for your comments!

One potential weakness of the paper lies in its handling of experiments in a probabilistic state transition setting. While the study explores action space perturbation, such as sticky action and noisy action, it could benefit from a more comprehensive consideration of the stochasticity within the state space. This might involve scenarios with dynamic background images or robots with joint angular positions sampled from a normal distribution with high variance. Incorporating these additional sources of uncertainty would provide a more realistic and robust evaluation of the proposed algorithm's performance under varied conditions.

Thank you for your insightful suggestion regarding the importance of investigating probabilistic state transitions to enhance the comprehensiveness and realism of our study.

In response, we have modified the Mujoco environment to introduce diagonal Gaussian noise, denoted as $\mathcal{N}(0,\sigma^2)$, to the state at each step.

*“Ours” is “State Augmentation - MLP” in Tab.1

** {Value}({Relative%}), where {Value} is normalized in the same manner as in Tab.1 of our paper. The {Relative%} indicates the relative reward compared to the highest value of each line.

The results comparing Ours$^*$ with the Vanilla SAC are presented below. It is important to note that when delay=0, Vanilla SAC is equivalent to Ours$^*$. Key observations from our experiments are as follows:

1. The performance of Vanilla SAC significantly decreases with the introduction of delay, dropping from 100% to 23.78%, 26.67%, 32.39% and 44.25% when the $\sigma$ is 0.05, 0.1, 0.2, 0.4, respectively.
2. Our method substantially improves performance in delayed scenarios. e.g. when $\Delta t=12$, the performance increase is 23.78%→79.57%, 26.67%→83.14%, 32.39%→92.25%, 44.25%→98.23%, when the $\sigma$ is 0.05, 0.1, 0.2, 0.4, respectively.

These observations are consistent with those in the original Mujoco environment, demonstrating that our improvements are both consistent and significant.

While we acknowledge that this may not encompass all probabilistic environments, we wish to highlight:

1. The significant and consistent improvement demonstrated in the above table.
2. The inclusion of two additional probabilistic environments discussed in Section G.1.2 of our paper.

These consistent results strongly support the efficacy of our method in probabilistic environments.

Furthermore, it is important to note that our theoretical framework consider general probabilistic scenario as in Theorem 2.1 and Theorem 4.1.

We hope these experimental and theoretical results adequately address your concerns.

We appreciate your suggestion and have included the Gaussian probabilistic experiment in Section G.1 of the paper.

Could the performance of the method be further enhanced by considering advanced world models, like the one developed for model-based RL in Dreamer[1], as an alternative for the architecture of the state augmentation and prediction module?

Good point! Actually, we did preliminary experiments about using advanced world models in this research, in particular the recurrent state space model (RSSM) used in Dreamer [1]. The problem of using RSSM is that since it models single-step state-transition, the prediction error may accumulate (especially due to a stochastic nature of RSSM) in the scenario of multi-steps delay. The empirical results also failed to support the effectiveness of using RSSM. Rather, MLP network is simple and good enough. Nonetheless, we appreciate this comment and will consider more powerful world models in future exploration.

[1] Mastering diverse domains through world models.

1.) Aren't delayed rewards already accounted for via the value function? Consider removing this wording in the introduction or clarify that your research looks at delays in states, as we could not find a discussion on addressing delays in received rewards.

Thanks for your kind advise. We removed “reward” part in the introduction.

2.) The transition probability in section 2.2 is confusing. The next state only transitions if $s’^{-t+ = s^{-t+1}}$ and otherwise would return 0 if any states don't match. Is this correct, or a misunderstanding?

Thanks for pointing this out this potential confusing point. The superscript $(-t)$ on the state $s$ is the relative timestep to the current one. And the transition probability function discribes “one step shift” of all the $s$ in $\sigma$ for each step transition. This is why $\mathbb{I}({s'^{(-t)} = s^{(-t+1)}}) = 1$ only if $s'$ is the next state of $s$ and $=0$ otherwise. This means exact one-step shift of for every $s^\{(-t)\}$ in $\sigma$.

3.) What were the parameters used in the experiments of Section 3? Including experimental details is reasonable for reproducibility. If they are in the appendix, please mention this in the section

We add it to Sec.E in the appendix and reference them in the Sec.5.1, the start of the experiment results.

4.) What about including the time since the last received sample as a state feature?

Thanks for your suggestion.

We think that this will take effect under case that the delay is unfixed, append time with the sample is promise to improve the performance.

1. According to our reply to your first question, we know that the when historical number $H$ is equal to delay $\Delta t$, the performance is the best. So it is promising that timestamp can help to focus on the historical in $\Delta t$ range, even when the delay is unfixed.
2. Especially, when the state is from multiple source, e.g. robot with multiple sensors, financial agent with multiple network information source, losing the time stamping information can make it hard to make decisions.

However, it's important to note that the effectiveness of this method may vary depending on the specific environment. For instance, certain sensors might not support time stamping, or network messages may lack time-related data. This make the trick is not a general solution for all cases

We create a new section Sec.D in the appendix to discuss this in detail.

One issue with the paper is the proposed use of previous actions in the augmented state representation. Some previous work suggests, particularly for sequence models [1], that this can be problematic, and it seems the results may mean this is the case as the Transformer and RNN results seem to benefit less from the state augmentation approach, especially as the horizons increase. We consider this a future work that the authors might consider discussing. Theorem 4.1 might require a few additional words to emphasize whether the provided proof is from previous work, mainly as the authors cite a paper when talking about this theorem.

Thank you for your valuable feedback. We appreciate the opportunity to look into the concerns you've raised.

You mentioned the issue of using past actions in our model. We understand that this can be a double-edged sword:

1. Good Point: Including past actions helps make our model's environment more predictable and easier to understand.
2. Downside: As pointed out in [1], this approach can sometimes cause problems.

To explore this further, we experimented with changing, $H$, the number of past actions our model considers. We wanted to see how this affects the performance when delay $\Delta t$ is fixed to $\{0,4\}$ . Here's a summary of what we found:

ps. We highlight the lines in the error range of highest one, to be consistent with Tab.1.

ps. We use State Augmentation - MLP in Tab.1 as method for above table.

• When we don't consider any past actions ($H=0$), and there's no delay in the data ($Δt=0$), the model performs quite well. But if we start adding too many past actions (like $H=4,8,12$), the performance drops significantly, result in 62.6% performance when $H=12$.
• In scenarios where there's a delay in the data ($Δt>0$), we observed that not incorporating any past actions ($H=0$) leads to suboptimal performance. For instance, with a delay of $\Delta t=4$, the absence of historical actions ($H=0$) resulted in a relatively low performance score of 47.6. Introducing a moderate amount of past actions significantly enhances performance, as evidenced by the improved score of 84.3. However, it's important to note that overloading the model with an excessive history ($H$ too high) reversely impacts the efficiency. This was clear when an even larger history led to a reduced performance of 55.0. This pattern underscores the necessity of a balanced approach in incorporating historical actions, especially in the context of delayed data.

In summary, our findings indicate that optimal performance is generally achieved when the number of past actions considered, $H$, closely aligns with the delay in the data, $\Delta t$. RNN

This finding also provides a potential explanation for the limited success of RNN structures in our experiments, as RNNs inherently lack control over the extent of historical information utilized, it just takes all historical states and actions into account. So it brought redundant information such as state history and extra action history outside of $\Delta t$, distract the model from values information.

We have added the results and discussion to Sec.G.3. Thanks for your comments.

[1] Shaking the foundations: delusions in sequence models for interaction and control. Ortega et al. 2021.