DiffLight: A Partial Rewards Conditioned Diffusion Model for Traffic Signal Control with Missing Data

We are delighted that the reviewer found our motivation interesting and reasonable. Thank you for your positive and insightful comments. We respond to each of the points below:

[W1 & Q1] Explanation of Diffusion Communication Mechanism

Thank you for your comments. We have made a brief introduction in Section 3.2. We apologize for our unclear expression and appreciate the opportunity to clarify this. Diffusion Communication Mechanism (DCM) is designed to enhance information communication among multiple intersections. In DCM, the states of neighboring intersection $x^k(\tau^i)$ diffused by diffusion model at diffusion step $k$ would be used to calculate $x^0(\tau^i)$ and then $x^0(\tau^i)$ would be sent to its current intersections. The agent in the current intersection would take $x^0(\tau^i)$ as the condition to generate the states of the current intersection $x^{k-1}(\tau)$ at diffusion step $k-1$.

[W2] Missing patterns of data

Thank you for highlighting the complex missing patterns of data in the real world. Undoubtedly, the missing patterns of data in the real world are far more complex than the random and Kriging missing patterns mentioned. We completely agree with this. In future work, we will follow your suggestion to further explore traffic signal control methods under mixed data-missing scenarios.

[W3] Symbols and metrics

Thank you for your detailed inspection of the symbol definitions. We will carefully correct these incorrect symbol definitions to improve the paper. Additionally, regarding the explanation of average travel time (ATT), it is the most commonly used metric in the field of traffic signal control. This metric represents the average travel time of all vehicles from entering the road network to leaving the intersection, and it effectively reflects the performance of traffic signal control. The calculation formula is as follows:

$$\text{ATT}=\frac{1}{N} \sum_{i=1}^{N} \left ( t_{i}^{l}-t_{i}^{e} \right ),$$

where $N$ is the total number of vehicles entering the road network, $t_{i}^{e}$ and $t_{i}^{l}$ are the entering time and leaving time for the $i$-th vehicle respectively.

We thank the reviewer for their thoughtful and insightful comments. We respond to each of the points as follows:

[W1] Selection for datasets

Thank you for your comment. We appreciate the opportunity to elaborate on this. We would like to point out that DiffLight is designed to address the traffic signal control problem under data-missing scenarios, instead of focusing on scalability. Meanwhile, the selected datasets are commonly used in most of the literature [1, 2, 3, 4, 5, 6]. Besides, due to the limitation of computing resources and time, it is difficult to conduct experiments on large-scale datasets with thousands of intersections.

However, there is no denying that scalability and cooperative performance play an essential role in applications. To further validate the scalability and cooperative performance of our approach, we conduct experiments on a larger dataset New York, which includes 48 intersections.

Random Missing:

Kriging Missing:

In the new experiment on the New York dataset, DiffLight achieves the best performance in most of scenarios, demonstrating its ability to deal with complex traffic scenarios and control traffic signals in a larger scale traffic network. In contrast, the performance of most baselines drops rapidly, due to the cumulative effect of errors in state imputation and decision-making at more intersections.

[W2] Benchmarks and metrics

Thank you for pointing out the lack of benchmarks and metrics. We apologize for our oversight and appreciate the opportunity to make a further explanation.

1. Concerning comparison benchmarks, we have included RL-based methods in experiments, including CQL and TD3+BC. To our knowledge, there is no GNN-based method proposed for the traffic singal control problem in the offline setting. Therefore, we choose CoLight [3], a GAT-based method widely used in the TSC task, as an additional baseline, to replace the DQN algorithm with the CQL algorithm, and conduct experiments on Hangzhou$_1$ and Jinan$_1$. However, due to the gap between offline RL and online RL, CoLight in the offline setting does not converge stably.
2. Concerning metrics, average travel time, queue length and throughput do reflect the performance of methods in different aspects. Whereas average travel time is widely used in most of the literature [1, 2, 3, 4, 6] and is a more important metric for improving traffic efficiency. We apologize for our oversight and we will take queue length and throughput into consideration in future work in order to evaluate method performance more comprehensively.

[W3] Lack of explanations for notations, formulas and conclusions

Thank you for your feedback. We will revisit our paper carefully and incorporate specific contextual explanations for these notations, formulas and conclusions.

[Q1] Boundary conditions of the method

Thank you for highlighting the boundary conditions of the method. These are indeed important for the application of our approach and I'm pleased to make the following discussion:

1. Limits on missing proportions: To further explore limits on missing proportions, we conduct experiments on the selected datasets in random missing with missing rates of 70% and 90%.

In the new experiment, DiffLight remains acceptable performance at the missing rate of 70%. When the missing rate rises to 90%, the performance of DiffLight drops rapidly, which shows that the limit for the missing rate is around 70%.

2. Conflicts with neighboring traffic: In the traffic network, all agents in the intersections make decisions at the same time with observations of the current intersection and neighboring intersections as input. Thus, there is no conflict in the order of decision-making. However, under data-missing scenarios, especially when the current intersection and neighboring intersections are all unobservable, agents' input could be unavailable, leading to conflict which could reduce the performance of the method. We design Diffusion Communication Mechanism (DCM) to alleviate this problem and conduct experiments in Appendix G.3.

Reference:

[1] Mei, Hao, et al. "Reinforcement learning approaches for traffic signal control under missing data." IJCAI. 2023.

[2] Ye, Yutong, et al. "InitLight: initial model generation for traffic signal control using adversarial inverse reinforcement learning." IJCAI. 2023.

[3] Wei, Hua, et al. "Colight: Learning network-level cooperation for traffic signal control." CIKM. 2019.

[4] Zang, Xinshi, et al. "Metalight: Value-based meta-reinforcement learning for traffic signal control." AAAI. 2020.

[5] Yu, Zhengxu, et al. "MaCAR: Urban traffic light control via active multi-agent communication and action rectification." IJCAI. 2021.

[6] Zhang, Liang, et al. "Expression might be enough: representing pressure and demand for reinforcement learning based traffic signal control." ICML. PMLR, 2022.

We highly appreciate your high-quality and valuable suggestions. We provide the point-by-point response as follows:

[W1 & Q1] Comparative analysis

Thank you for emphasizing the gap in the comparative analysis. We apologize for our unclear expression. Actually, all the baselines were implemented with reference to the SDQN-SDQN (transferred) method in MissLight [1]. We replaced the DQN algorithm with different algorithms in the offline setting and imputed the states with the SFM model.

To further evaluate the efficacy of the proposed approach, we employ an offline version of SDQN-SDQN (model-based) in MissLight and conduct experiments on Hangzhou$_1$ and Jinan$_1$. To distinguish the baseline of CQL in Section 4.2, it is named CQL (model-based) while CQL in Section 4.2 is named CQL (transferred).

Random Missing:

Kriging Missing:

In the new experiment, DiffLight achieves competitive performance compared with CQL (transferred) and CQL (model-based). The possible reason why DiffLight has better performance is that CQL (model-based) suffers from error accumulation caused by the reward imputation model while DiffLight can directly make decisions with Partial Rewards Conditioned Diffusion (PRCD).

Besides, Diffuser [2] and Decision Diffuser [3] are the most representative work in the field of diffusion model for RL, which are included as baselines in much literature [9, 10, 11]. Thus, we choose them as our baselines rather than other diffusion-based methods.

[W2&Q2] Additional datasets

Thank you for highlighting the discussion on the selection of datasets. Actually, Hangzhou, Jinan and New York datasets are all commonly used in recent literature [1, 4, 5, 6, 7, 8]. We conducted extensive experiments on Hangzhou and Jinan datasets with different data-missing scenarios. To further evaluate the efficacy and validate the performance of our approach, we conduct experiments on the New York dataset, which includes 48 intersections.

Random Missing:

Kriging Missing:

In the new experiment on the New York dataset, DiffLight achieves the best performance in most of scenarios, demonstrating its ability to deal with complex traffic scenarios and control traffic signals in a larger scale traffic network. In contrast, the performance of most baselines drops rapidly, due to the cumulative effect of errors in state imputation and decision-making at more intersections.

Reference:

[1] Mei, Hao, et al. "Reinforcement learning approaches for traffic signal control under missing data." Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023.

[2] Janner, Michael, et al. "Planning with Diffusion for Flexible Behavior Synthesis." International Conference on Machine Learning. PMLR, 2022.

[3] Ajay, Anurag, et al. "Is Conditional Generative Modeling all you need for Decision Making?." The Eleventh International Conference on Learning Representations.

[4] Ye, Yutong, et al. "InitLight: initial model generation for traffic signal control using adversarial inverse reinforcement learning." Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023.

[5] Wei, Hua, et al. "Colight: Learning network-level cooperation for traffic signal control." Proceedings of the 28th ACM international conference on information and knowledge management. 2019.

[6] Zang, Xinshi, et al. "Metalight: Value-based meta-reinforcement learning for traffic signal control." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 01. 2020.

[7] Yu, Zhengxu, et al. "MaCAR: Urban traffic light control via active multi-agent communication and action rectification." Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 2021.

[8] Zhang, Liang, et al. "Expression might be enough: representing pressure and demand for reinforcement learning based traffic signal control." International Conference on Machine Learning. PMLR, 2022.

[9] Li, Wenhao, et al. "Hierarchical diffusion for offline decision making." International Conference on Machine Learning. PMLR, 2023.

[10] Venkatraman, Siddarth, et al. "Reasoning with Latent Diffusion in Offline Reinforcement Learning." The Twelfth International Conference on Learning Representations.

[11] Zhu, Zhengbang, et al. "Madiff: Offline multi-agent learning with diffusion models." arXiv preprint arXiv:2305.17330 (2023).

We appreciate the reviewer recognizing the importance of our work and thank you for the detailed comments. Please find the point-by-point responses to the reviewer's comments below.

[W1] Discussion on MissLight

Thank you for highlighting the importance of a direct discussion between DiffLight and MissLight [1]. We apologize for our unclear expression and appreciate the opportunity to have a further discussion. We have made a brief discussion in the part of the Introduction in lines 40-44. To better clarify the core differences between our approach and MissLight, we compare them from the following two aspects:

1. Model training: MissLight is an online method with a state imputation model and a reward imputation model, which means that interaction with the environment is necessary. In the online setting, if the method is trained in the physical environment, safety problems must be taken into consideration. If the method is trained in a simulated environment, the difference between the physical environment and the simulated environment could affect the performance of the method to some extent when the online method is going to be employed in the physical environment. In contrast, our approach, DiffLight, is an offline method based on the diffusion model. In the offline setting, our method is trained using the collected dataset without interaction with the environment, which avoids the problems mentioned above.
2. Model composition: MissLight is a two-stage method. In the first stage, state imputation and reward imputation models are used to fill in the missing data. In the second stage, the DQN algorithm is employed to complete the training process based on the imputed data. This approach suffers from the problem of error accumulation during the training process. However, our proposed DiffLight model, which incorporates both a diffusion model and an inverse dynamics model, can simultaneously train on missing data and collaboratively achieve traffic signal control with missing data.

Additionally, it should be noted that MissLight as an online method can't be compared with DiffLight directly. To better compare with MissLight, we implement the SDQN-SDQN (model-based) in MissLight and replace the DQN algorithm with the CQL algorithm in order to adapt the offline setting. Note that all the baselines were implemented with reference to the SDQN-SDQN (transferred) method in MissLight. We replaced the DQN algorithm with different algorithms in the offline setting and imputed the states with the SFM model. To distinguish the baseline of CQL implemented in Section 4.2, the new baseline is named CQL (model-based) while CQL in Section 4.2 is named CQL (transferred).

Random Missing:

Kriging Missing:

DiffLight achieves competitive performance compared with CQL (transferred) and CQL (model-based). The possible reason why DiffLight has better performance is that CQL (model-based) suffers from error accumulation caused by the reward imputation model while DiffLight can directly make decisions with Partial Rewards Conditioned Diffusion (PRCD).

[Q1] Word and grammatical errors

Thank you for your feedback. We will correct the word and grammatical errors you mentioned and check for other possible errors.

[Q2] Code reproducibility

Thank you for your suggestion. We update a requirement into our code in the anonymous link. To enhance the reproducibility, we will include a readme file in the code. We apologize for our oversight of the readability and reproducibility.

Reference:

[1] Mei, Hao, et al. "Reinforcement learning approaches for traffic signal control under missing data." Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023.

Dear Junxian,

Thank you for your comment! MissLight really inspires us a lot. Hope to have further discussions with you in the future!

Best regards,

Hanyang Chen