Delay-Aware Reinforcement Learning: Insights From Delay Distributional Perspective

The following are the latest comparison evaluation results.

Comparative evaluation in gamma delay distribution

Comparative evaluation in double gaussian delay distribution

Comparative evaluation in uniform delay distribution

References

[1]Bouteiller, Y., Ramstedt, S., Beltrame, G., Pal, C., and Binas, J. Reinforcement learning with random delays. In International conference on learning representations, 2021.

[2]Liotet, P., Maran, D., Bisi, L., and Restelli, M. Delayed reinforcement learning by imitation. In International Conference on Machine Learning, pp. 13528–13556. PMLR, 2022.

[3]Qingyuan Wu, Simon Sinong Zhan, Yixuan Wang, Yuhui Wang, Chung-Wei Lin, Chen Lv, Qi Zhu, Jürgen Schmidhuber, and Chao Huang. Boosting reinforcement learning with strongly delayed feedback through auxiliary short delays. In Forty-first International Conference on Machine Learning(2024).

[4]Qingyuan Wu, Simon Sinong Zhan, Yixuan Wang, Yuhui Wang, Chung-Wei Lin, Chen Lv, Qi Zhu, and Chao Huang. Variational delayed policy optimization. In Advances in Neural Information Processing Systems 37 (2024).

[5]Jangwon Kim, Hangyeol Kim, Jiwook Kang, Jongchan Baek, and Soohee Han. Belief projection based reinforcement learning for environments with delayed feedback. Advances in Neural Information Processing Systems, 36:678–696, 2023.

[6]Wei Wang, Dongqi Han, Xufang Luo, and Dongsheng Li. Addressing signal delay in deep reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.

[7] Simon Ramstedt, et al. Real-time reinforcement learning. In NeurIPS, 2019.

We greatly appreciate your thoughtful and detailed review of our work. The following responses address your concerns in detail.

• [W1] Writing

We sincerely appreciate your thorough reading and careful review of our paper. In future revisions, we will address and resolve the issues you raised to ensure a more polished and unified presentation.

• [W2] Theorem

For Theorem 1, we will investigate the performance of the proposed operator at the fixed point in the future version of this paper. For Theorem 2, we focused on validating the applicability of the equivalence between different types of delays within the framework of our method, where delays are assumed to follow a specific probability distribution.

• [W3] Experiment

1. Doubts about the performance of BPQL[5] and State-Aug-MLP[6]

Experiments with BPQL[5] in gamma delay distribution at different delay settings

Experiments with BPQL[5] in double gaussian delay distribution at different delay settings

In random delay environments, we provide the mean of the delay distribution rather than its maximum value to BPQL[5], as BPQL fundamentally relies on a state augmentation approach. We believe that the augmented state represents a potential expression of the ideal, delay-free observation. For example, if we were to provide BPQL with the maximum delay value—such as the gamma distribution with a range of 1-6 (mean 2) used in this paper—BPQL would always select actions based on the state 6 time steps ahead. However, in reality, the probability of a delay of 6 is extremely low. This setup would likely make the algorithm overly farsighted. Providing BPQL with the mean delay instead allows it to consistently select actions in an averaged manner. To validate this, we added experiments where the maximum delay value was provided to BPQL. As expected, this configuration further degraded BPQL's performance.

For the State-Aug-MLP[6] method, the original paper explicitly specifies the use of the maximum value of random delays in non-fixed delay environments. Therefore, we followed this setup and provided the maximum random delay value to State-Aug-MLP.

Regarding the significant performance decline of BPQL in our experiments, we believe this is inherently caused by the differences between fixed-delay and random-delay environments. In random-delay environments, observations or actions can experience time misalignment, repetition, or omission. These factors significantly impair the ability of state augmentation methods to accurately represent the ideal delay-free observation, further disrupting the Markov property and severely affecting the algorithm's performance. This highlights the primary challenge in addressing delay issues in random-delay environments. Lastly, we assure that the experimental data presented in our results are genuine and accurate, and we have released the complete source code of our method.

2&3. Additional baselines

Among the additional baselines mentioned by the reviewers, DC/AC[1] is specifically designed for addressing random delay. However, due to its computational inefficiency, we decided not to include comparisons with it, and its limitation is also discussed in [2]. Based on our investigation, other baselines are primarily designed for fixed-delay environments, although some methods briefly evaluate their performance in random-delay environments.

For instance, RTAC[7] is designed to handle one-step action delays, which makes it less suitable for comparison with delay-aware RL methods. Additionally, DIDA[2] has not released its source code. Due to time constraints, we conducted limited tests to include comparisons with VDPO[4], which is likely one of the most recent methods. Notably, VDPO outperforms AD-RL[3], DIDA[2], DC/AC[1] in most scenarios according to its reported experimental results.

The following are the latest comparison evaluation results.

Comparative evaluation in gamma delay distribution

Comparative evaluation in double gaussian delay distribution

Comparative evaluation in uniform delay distribution

References

[1]Bouteiller, Y., Ramstedt, S., Beltrame, G., Pal, C., and Binas, J. Reinforcement learning with random delays. In International conference on learning representations, 2021.

[2]Liotet, P., Maran, D., Bisi, L., and Restelli, M. Delayed reinforcement learning by imitation. In International Conference on Machine Learning, pp. 13528–13556. PMLR, 2022.

[3]Qingyuan Wu, Simon Sinong Zhan, Yixuan Wang, Yuhui Wang, Chung-Wei Lin, Chen Lv, Qi Zhu, Jürgen Schmidhuber, and Chao Huang. Boosting reinforcement learning with strongly delayed feedback through auxiliary short delays. In Forty-first International Conference on Machine Learning (2024).

[4]Qingyuan Wu, Simon Sinong Zhan, Yixuan Wang, Yuhui Wang, Chung-Wei Lin, Chen Lv, Qi Zhu, and Chao Huang. Variational delayed policy optimization. In Advances in Neural Information Processing Systems 37 (2024).

[5]Jangwon Kim, Hangyeol Kim, Jiwook Kang, Jongchan Baek, and Soohee Han. Belief projection based reinforcement learning for environments with delayed feedback. Advances in Neural Information Processing Systems, 36:678–696, 2023.

[6]Wei Wang, Dongqi Han, Xufang Luo, and Dongsheng Li. Addressing signal delay in deep reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.

[7]Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao. Delay-aware model-based reinforcement learning for continuous control. Neurocomputing, 450:119–128, 2021.

[8]Somjit Nath, Mayank Baranwal, and Harshad Khadilkar. Revisiting state augmentation methods for reinforcement learning with stochastic delays. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1346–1355, 2021.

[9]Krasniqi, F., Elias, J., Leguay, J., & Redondi, A. E. End-to-end delay prediction based on traffic matrix sampling. In IEEE INFOCOM 2020-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) (pp. 774-779). IEEE.

[10]Wang, Y., Vuran, M. C., & Goddard, S. Cross-layer analysis of the end-to-end delay distribution in wireless sensor networks. IEEE/ACM Transactions on networking, 20(1), 305-318.

[11]Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE transactions on neural networks and learning systems, 33(11):6584–6598, 2021.

[12]Guillo-Sansano, E., Syed, M. H., Roscoe, A. J., Burt, G. M., & Coffele, F.. Characterization of time delay in power hardware in the loop setups. IEEE Transactions on Industrial Electronics, 2020, 68(3), 2703-2713.

We sincerely thank you for your meticulous review and valuable suggestions. The following are our responses to your comments.

• [Q1] Concerns about delay distribution

Our method assumes that the delay distribution is known as prior knowledge and that it possesses stationary properties. In recent related research, most methods typically rely on exact delay information as prior knowledge, such as real-time true delay values [1,2,3,4,5,7,8], and most of them are designed for the environment with fixed delays. However, this type of prior knowledge is often difficult to obtain in practical applications and the delays are usually non-fixed in practical systems. This study aims to bring delay-aware reinforcement learning in random delay environments closer to real-world applications.

Some existing work also assumes prior knowledge of the maximum value of random delays [6]，but we found that many real-world delays often follow specific statistical patterns [6,9,10,12], such as network delays. By leveraging these statistical characteristics as prior knowledge for the delay distribution, our method becomes more suitable for real-world scenarios and outperforms the method with prior knowledge of maximum delay. The analysis of Wi-Fi network delays in [1] shows that the statistical results closely resemble a gamma distribution; in [9], the authors used Gaussian-related distributions to simulate delays. These studies inspired us to design gamma and double Gaussian distributions to simulate various delay environments.

Theoretically, as long as the delay distribution is known, our method can effectively address delay problems, regardless of the specific type of delay distribution. To further validate the adaptability of our method across different delay environments, we have included experimental results based on a uniform distribution with values ranging from 1 to 13.

• [Q2] How the delay distribution affects the return distribution?

Distributional reinforcement learning aims to more accurately represent the return in the form of a distribution, and we refer to [11] and estimate the return using a Gaussian distribution. In our method, the critic learns the mean and standard deviation of this return distribution to represent the final return of a state-action pair, accounting for the impact of the delay distribution. During the policy improvement phase, we use the mean of the return distribution to assist in policy updates.

• [Q3] Why Eq. (4) is not reduced to a unified form.

As the reviewer pointed out, existing work has demonstrated the equivalence of observation delay and action delay. In Theorem 2, we also prove that our method can equivalently handle both types of delays. In the presentation of this paper, we extended the consideration from only one type of delay to simultaneously considering both types. In fact, equation (3) is a simplified version of equation (4), which only considers one type of delay, as long as we assume that $p_i$ represents a probability distribution equivalent to the cumulative effect of both types of delays. However, in practical applications, if the delay distributions for observation and action delays are different, such as the observation delay following a gamma distribution and the action delay following a double Gaussian distribution, the probability distribution equivalent to the cumulative effect of these delay distributions is difficult to compute accurately. Therefore, we retain the formulation that simultaneously considers both types of delays for ease of understanding and calculation.

• [W1] Related Works

We will include the missing recent benchmarks in future versions of the paper.

• [W2] Section 4

We sincerely thank the reviewers for their writing suggestions and for pointing out the issues with inconsistent definitions. We will carefully consider the reviewers' concerns regarding the proof of Theorem 1 and potential extensions, and we will make the necessary revisions in future versions of the paper.

• [W3] Experiments

Due to time constraints, we only added comparisons with VDPO [4], which is one of the most recent methods, and VDPO [4] outperforms all the other methods including DIDA[2], AD-RL[3], DC/AC[1] in most scenarios. (Additionally, DIDA did not released its source code. We initially considered comparing with DC/AC[1]; however, its computational inefficiency led us to abandon this comparison, and its limitation is also discussed in [2].) In future versions of the paper, we plan to further refine the experimental results and incorporate additional baselines.

Thanks for author’s rebuttal, and some answers do address my concern. Here are some remaining concerns:

• All the addressed writing issues etc seems not be addressed in the updated pdf.
• Under what kind of scenarios we have both action delay and state delay with different prior distribution? And do the system still possess markovian property? If answer is yes, then I can understand the not unified version of Eq 4, but corresponding proof need to provided.
• I think DC/AC and AD-RL as the well-known works in the field should appear as benchmarks for comparison, though VDPO seems to be the recently appeared work. It only address the sample efficiency issue in previous works with similar convergence point guarantee, and in their experiments there is also no statement or proof for exceeding final performance.
• The problem of Thm 1 is still not resolved.

Thus, I have decided to remain my current scores and confidence if no further improvements have been made.

Thank you for your detailed review and valuable insights. Below please find our response to your concerns.

• [Q1 & W2] Concerns about value correction mechanism

Firstly, in our method, distributional returns and the value correction mechanism are complementary and interdependent. Random delays may introduce issues such as missing, duplicate, or out-of-order observations and actions, resulting in sequential feedback from the agent-environment interaction that is inconsistent with the feedback stored in the replay buffer. Due to this inconsistency, SAC+Value Correction suffers from inaccurate Q-value estimation based on sampled data from the replay buffer, thereby reducing the effectiveness of the value correction mechanism.

In contrast, distributional returns can effectively model the uncertainty in delayed environments, enhancing the robustness and multimodal expressiveness of the value function estimation. Consequently, the DSAC method inherently demonstrates a certain level of adaptability to delayed environments, achieving performance comparable to SAC+Value Correction. Furthermore, leveraging the advantage of distributional returns in significantly reducing value function estimation errors, the value correction mechanism becomes more effective. This synergy enables our proposed DADAC method to achieve superior performance in random delay environments.

• [Q2 & Q4] Concerns about the settings of delay distribution

In equation (3), $p_i$ represents the probability of the delay of i, which is typically considered a fixed decimal value between 0 and 1. This delay distribution acts as prior knowledge for the value correction mechanism, with the assumption that the delay distribution is stationary. If $p_i$ = 1, it indicates a constant delay environment with the delay of i. In this case, equation (3) can be written as

$Z(s_{t},a_{t})=r_{t+i}(s_{t},a_{t},s_{t+i})+\gamma^{i}\cdot\sum_{a_{t+i}\sim\pi(\cdot|s_{t+i})}Z(s_{t+i},a_{t+i})$

This is not the conventional Bellman equation, as the right-hand side represents the return starting from time t+i. The variant form of equation (3) is precisely how our method handles fixed delay environments. The following are the experimental results of our method in constant delay environment with limited tests. The results demonstrate that DADAC effectively handles constant delay environments.

Comparative evaluation with constant delay of 5

• [Q3 & W2] Questions about delay distributions in experiments

Our method assumes that the delay distribution is known as prior knowledge and that it possesses stationary properties. In recent related research, most methods typically rely on exact delay information as prior knowledge, such as real-time true delay values [1,2,3,4,5,7,8], and most of them are designed for the enviornment with fixed delays. However, this type of prior knowledge is often difficult to obtain in practical applications and the delays are usually non-fixed in practical systems. This study aims to bring delay-aware reinforcement learning in random delay environments closer to real-world applications.

Some existing work also assumes prior knowledge of the maximum value of random delays [6]，but we found that many real-world delays often follow specific statistical patterns [6,9,10,11], such as network delays. By leveraging these statistical characteristics as prior knowledge for the delay distribution, our method becomes more suitable for real-world scenarios and outperforms the method with prior knowledge of maximum delay. The analysis of Wi-Fi network delays in [1] shows that the statistical results closely resemble a gamma distribution; in [9], the authors used Gaussian-related distributions to simulate delays. These studies inspired us to design gamma and double Gaussian distributions to simulate various delay environments.

Theoretically, as long as the delay distribution is known, our method can effectively address delay problems, regardless of the specific type of delay distribution. To further validate the adaptability of our method across different delay environments, we have included experimental results based on a uniform distribution with values ranging from 1 to 13.

• [W3] The experiments are insufficient

The two methods we compared, BPQL[5] and State Augmentation-MLP[6], are already state-of-the-art methods. While we could compare with other methods, their performance is inferior to these two SOTA baselines. Additionally, we identified a newly published method, VDPO[4], which represents the latest work in this area. To further enrich our experiments, we added comparisons with VDPO[4] and included environments where the delay distribution follows a uniform distribution ranging from 1 to 13. Due to time constraints, we conducted a limited set of experimental tests. In future work, we plan to refine the experimental results further and consider incorporating more baselines and diverse delay environments.

Comparative evaluation in gamma delay distribution

Comparative evaluation in double gaussian delay distribution

Comparative evaluation in uniform delay distribution

• [Q5] How does DADAC compare to a delay-free algorithm like robust RL?

Consistent with most of the related work, we did not consider comparisons with robust reinforcement learning. In my view, if the application scenario (e.g., drone navigation) involves significant delays, selecting DADAC offers a clear advantage, as it more precisely addresses state and control errors caused by delays. On the other hand, if the primary focus is on environmental uncertainties (e.g., parameter perturbations or noise variations) and delay effects are minimal, Robust RL may be preferable for its simplicity and broad adaptability. Nevertheless, I believe that some optimization strategies from Robust RL could be leveraged to enhance the adaptability of delay-aware reinforcement learning.

• [W1] Lack of relevant experiments where both types of delays occur simultaneously

Most related works focus on experiments conducted in environments with observation delays only, and adapting them directly to environments with action delays is often challenging. Therefore, for a fair comparison, we limited our experiments to scenarios with observation delays. To further validate that our method can handle scenarios where both types of delays coexist, we plan to include an additional set of experiments. These will demonstrate that our method can equivalently address environments with a single type of delay as well as those with both types simultaneously.

For fair comparison, delays in single-delay scenarios followed a uniform distribution from 1 to 10, while in dual-delay scenarios (observation and action delays), both followed a uniform distribution from 1 to 5—equivalent to a single delay type with a range of 1 to 10. Due to time constraints, we conducted a limited number of tests, which may lead to performance instability. Nonetheless, current results demonstrate that the proposed DADAC effectively handles environments with both observation and action delays.

Comparison experiments with different types of delays.

The following are the latest comparison evaluation results.

Comparative evaluation in gamma delay distribution

Comparative evaluation in double gaussian delay distribution

Comparative evaluation in uniform delay distribution

References

[1] Bellemare, Marc G., Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International conference on machine learning. PMLR, 2017.

[2] Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE transactions on neural networks and learning systems, 33(11):6584–6598, 2021.

[3] Nam, D. W., Kim, Y., & Park, C. Y. . GMAC: A distributional perspective on actor-critic framework. In International Conference on Machine Learning (pp. 7927-7936). PMLR.

[4] G. Barth-Maron et al. Distributed distributional deterministic policy gradients. In International conference on learning representations, 2018.

[5] Jangwon Kim, Hangyeol Kim, Jiwook Kang, Jongchan Baek, and Soohee Han. Belief projection based reinforcement learning for environments with delayed feedback. Advances in Neural Information Processing Systems, 36:678–696, 2023.

[6] Bouteiller, Y., Ramstedt, S., Beltrame, G., Pal, C., and Binas, J. Reinforcement learning with random delays. In International conference on learning representations, 2021.

[7] Wei Wang, Dongqi Han, Xufang Luo, and Dongsheng Li. Addressing signal delay in deep reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.

[8] Qingyuan Wu, Simon Sinong Zhan, Yixuan Wang, Yuhui Wang, Chung-Wei Lin, Chen Lv, Qi Zhu, Jürgen Schmidhuber, and Chao Huang. Boosting reinforcement learning with strongly delayed feedback through auxiliary short delays. In Forty-first International Conference on Machine Learning.

[9] Qingyuan Wu, Simon Sinong Zhan, Yixuan Wang, Yuhui Wang, Chung-Wei Lin, Chen Lv, Qi Zhu, and Chao Huang. Variational delayed policy optimization. In Advances in Neural Information Processing Systems 37 (2024).

[10] Liotet, P., Maran, D., Bisi, L., and Restelli, M. Delayed reinforcement learning by imitation. In International Conference on Machine Learning, pp. 13528–13556. PMLR, 2022.

[11] Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao. Delay-aware model-based reinforcement learning for continuous control. Neurocomputing, 450:119–128, 2021.

[12] Somjit Nath, Mayank Baranwal, and Harshad Khadilkar. Revisiting state augmentation methods for reinforcement learning with stochastic delays. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1346–1355, 2021.

[13] Krasniqi, F., Elias, J., Leguay, J., & Redondi, A. E.. End-to-end delay prediction based on traffic matrix sampling. In IEEE INFOCOM 2020-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) (pp. 774-779). IEEE.

[14]Wang, Y., Vuran, M. C., & Goddard, S. (2011). Cross-layer analysis of the end-to-end delay distribution in wireless sensor networks. IEEE/ACM Transactions on networking, 20(1), 305-318.

[15]Guillo-Sansano, E., Syed, M. H., Roscoe, A. J., Burt, G. M., & Coffele, F.. Characterization of time delay in power hardware in the loop setups. IEEE Transactions on Industrial Electronics, 2020, 68(3), 2703-2713.

We deeply appreciate your thoughtful review and constructive comments. We have addressed your concerns as follows.

• [Q1] Concerns about inconsistencies in the form of γ

The notation for $\gamma$ in equation (3) is correct; however, due to a typographical error, we omitted the exponential form of $\gamma$ in equation (5). We will address and correct this issue in future versions of the paper.

• [Q2 & W1] Theorem 1

In [1], it has been proven that the distributional Bellman operator is a contraction under the Wasserstein distance. However, the computational complexity of the Wasserstein distance makes it impractical for real-world applications. In distributional reinforcement learning, it is a common setting to approximate the true return distribution using a specific distribution and to define the loss function accordingly [1,2,3,4]. Following the methodology outlined in [2], we adopt Gaussian distribution to approximate the return distribution and employ the KL divergence during the update process. In future versions of this paper, we will enhance the explanation of Theorem 1 and further investigate the convergence properties of our method when employing the KL divergence variant as the distance metric.

• [Q3 & W2] Concerns about ablation studies

Firstly, in our method, distributional returns and the value correction mechanism are complementary and interdependent. Random delays may introduce issues such as missing, duplicate, or out-of-order observations and actions, resulting in sequential feedback from the agent-environment interaction that is inconsistent with the feedback stored in the replay buffer. Due to this inconsistency, SAC+Value Correction suffers from inaccurate Q-value estimation based on sampled data from the replay buffer, thereby reducing the effectiveness of the value correction mechanism.

In contrast, distributional returns can effectively model the uncertainty in delayed environments, enhancing the robustness and multimodal expressiveness of the value function estimation. Consequently, the DSAC method inherently demonstrates a certain level of adaptability to delayed environments, achieving performance comparable to SAC+Value Correction. Furthermore, leveraging the advantage of distributional returns in significantly reducing value function estimation errors, the value correction mechanism becomes more effective. This synergy enables our proposed DADAC method to achieve superior performance in random delay environments.

• [Q4] The number of random seed

In recent related works, we noticed that the number of experimental test runs varies, such as 5 runs (BPQL[5]), 6 runs (DC/AC[6]), 8 runs (State Augmentation-MLP[7]), and 10 runs (AD-RL[8] and VDPO[9]). Due to time constraints, we opted for a moderate number of 8 random seeds and did not exclude any outlier seeds. We have released the complete source code and will include additional experiments in the next version of the paper.

• [W3] Concerns about delay distribution

Our method assumes that the delay distribution is known as prior knowledge and that it possesses stationary properties. In recent related research, most methods typically rely on exact delay information as prior knowledge, such as real-time true delay values [5,6,8,9,10,11,12], and most of them are designed for the enviornment with fixed delays. However, this type of prior knowledge is often difficult to obtain in practical applications and the delays are usually non-fixed in practical systems. This study aims to bring delay-aware reinforcement learning in random delay environments closer to real-world applications.

Some existing work also assumes prior knowledge of the maximum value of random delays [7]，but we found that many real-world delays often follow specific statistical patterns [6,13,14,15], such as network delays. By leveraging these statistical characteristics as prior knowledge for the delay distribution, our method becomes more suitable for real-world scenarios and outperforms the method with prior knowledge of maximum delay. The analysis of Wi-Fi network delays in [6] shows that the statistical results closely resemble a gamma distribution; in [13], the authors used Gaussian-related distributions to simulate delays. These studies inspired us to design gamma and double Gaussian distributions to simulate various delay environments.

Theoretically, as long as the delay distribution is known, our method can effectively address delay problems, regardless of the specific type of delay distribution. To further validate the adaptability of our method across different delay environments, we have included experimental results based on a uniform distribution with values ranging from 1 to 13.