OfflineLight: An Offline Reinforcement Learning Model for Traffic Signal Control

The authors express gratitude for the valuable comments and will address them individually.

• Q1：What is the reward function?

  Response: The reward function in our decentralized MARL approach focuses on maximizing individual rewards for each traffic signal controller (TSC), as detailed in Equation 2. The metric used is the vehicle queue length (QL), with shorter queues indicating higher efficiency and rewards.

• Q2: What is the meaning of "the Critic is applied to address overestimated values of the Critic"?

  Response: Apologies for the misunderstanding due to the inappropriate terminology used. The correct term should be 'overestimated values of the Q value,' not 'overestimated values of the Critic.' We have corrected for this in the revision.

• Q3: In 4.3.1: " where $\pi_{\theta'}$ is the target policy network by constraining the policy."

  Response: Apologies for the lack of clarity in our description within Section 4.3.1 of the 'BEAR' paper, particularly regarding the target policy network. To clarify, the implementation of the target policy network in our paper is indeed similar to that in BEAR. We integrate noise into the learning policy to expand the range of action selection. Specifically, we implement this by calculating the KL divergence. We will clarify in the revision when the paper is accepted.

• Q4: Why applying PSR to overfitting to narrow peaks?

  Response: Firstly, the primary function of PSR in our Actor implementation is not to directly address overfitting to narrow peaks in the value estimate, but rather to enhance the policy's robustness and generalization. The reference to ‘narrow peaks’ is in the context of the value function landscape. In some scenarios, the value function might exhibit sharp peaks, indicating a high estimated reward for very specific actions or action sequences. Without appropriate regularization, the Actor might overfit to these narrow peaks, leading to a brittle policy that performs well in the training environment but fails to generalize to slightly different or unseen situations.

• Q5: What are the new insights?

  Response: Our method introduces a unique computational paradigm that simultaneously considers constraints on policy (action) and value, focusing on the trade-off between these two aspects. This approach is particularly insightful for addressing the OOD challenges in offline RL. By balancing the constraints on policy and value, our method effectively mitigates the common pitfalls associated with distributional shifts, which are prevalent in offline RL scenarios. OfflineLight, under this paradigm, has set a new benchmark in traffic signal control (TSC) within offline RL.

• Q6: How many seeds do you use in composing Table 1?

  Response: To ensure a fair and consistent comparison across all experimental runs, we employed the same seed for each iteration of our experiments. Specifically, we conducted a total of 80 rounds of training, with each round followed by a testing phase. Additionally, our code has been made publicly available to facilitate transparency and reproducibility.

• W1: Several math formulas are incorrect.

  Response: We apologize for our carelessness and thank you for your attention. There are several typos in the formula. In Eq.1, $\gamma^{n}$ should be $\gamma^{t}$, and $r^{n}$ of $Q(s,a)$ in Section 4.2.2 should be $r^{t^{'}}$.

• W2: What is the "offline Actor-Critic framework"'s contribution?

  Response: We believe that the main contribution of this paper is the innovation of proposing a new framework. It actually encourages novel approaches and breakthroughs in optimizing and balancing these constraints, which is a significant aspect of our work. Our OfflineLight is an example algorithm for offline Actor-Critic framework, which lies in integrating and leveraging these known techniques to push the boundaries of performance in TSC.

• W3: The paper is hard to follow.

  Response: We have carefully reviewed and polished our manuscript, making structural adjustments as per your suggestions.

• W4: CQL has similar overall performance.

  Response: Our analysis across seven datasets demonstrates that OfflineLight outperforms CQL in five out of these seven datasets. In some cases, the improvements brought by OfflineLight involve significant reductions in Average Travel Time (ATT) by several tens of seconds. While in other datasets, the reduction in ATT may appear marginal (a few seconds), hence it is important to consider the broader impact in urban traffic systems. When translated into real-world implications, these time savings can result in considerable reductions in fuel consumption and subsequent decreases in air pollution.

• W5: The authors may remove/reduce Section 5 and Section 6.

  Response: Based on feedback, Sections 5 and 6 will be reduced, with essential results from Sections 6.6 & 6.7 integrated into the main paper for coherence.

We thank you for your insightful reviews and address your concerns as follows.

• Q1&W2：Describe Offline Dataset Collection and Training Procedure?

  Response:

  1.Data Collection Method: Our approach to data collection prioritizes diversity to enhance the model's generalization capabilities. We used a variety of states and rewards, acknowledging that the optimal choices in existing works are still ambiguous. Our offline dataset comprises interactions generated by different policies, akin to the data buffers in off-policy reinforcement learning methods like Q-learning or DDPG. We engage various RL methods to interact with the CityFlow traffic simulator, recording diverse interaction data. This process encompasses different frameworks, traffic states, and rewards, detailed in Table 3 in Appendix A.2.

  2.States of RL Agents: The RL agents employed in our study were trained from scratch. This approach ensures that the distribution of rewards does not converge prematurely, allowing us to discern the impacts of different actions more effectively. If agents had been well trained beforehand, the reward distribution would have stabilized, making it difficult to assess the relative merits of their actions.

  3.Training on Datasets: Our training was conducted on a limited portion of the available datasets. For example, OfflineLight-JN1 was trained on only 20% of all data from TSC-OID. Despite this limited training, our model demonstrates robustness, as it can be effectively deployed on various other traffic topologies and flows without extensive retraining.

• Q2:In the appendix Figure 4. The comparison seems not convincing.

  Response: It is essential to clarify the interpretation of the transfer ratio used in our study. The transfer ratio, defined as $t_{transfer}/t_{train}$ , inversely correlates with performance: The higher the ratio, the lower the performance. The MPLight has shown the best transferability performance. But the comparison demonstrates that advanced MPLight does not outperform OfflineLight in Figure 4. The seemingly low transfer ratio for MPLight does not indicate superior performance but rather the opposite. This is a crucial aspect of our analysis, highlighting the effectiveness of OfflineLight in comparison to MPLight under the given metric.

  To further solidify our argument about the generalization capabilities of OfflineLight, we plan to include several classic RL methods in our analysis.

• Q3:In section 4.2.1, should the action and state subscribe with n and t?

  Response: It is crucial to note that the Q value in our formulation indeed corresponds to each time step t in the $n^{\text{th}}$ batch. While in Equation (3), for the sake of simplicity, the Q value is not explicitly subscripted with n and t , the calculations preceding this formulation have taken these variables into account. The decision to present the Q value without direct reference to n and t in the equation was a deliberate choice aimed at simplifying the representation and making the equation more readable. However, this should not be misconstrued as an oversight or a simplification in the actual computational process. We will certainly clarify this in the revision.

• Q4:“Why is the Offline-AC the abstract method?”.

  Response:

  1.Offline-AC (Actor-Critic) should be recognized as a computational paradigm within offline RL, uniquely integrating constraints on actions and values. This paradigm is not merely a specific algorithm or technique but represents a broader conceptual approach.

  2.Methods like CQL and BEAR can be viewed as specific instances of the Offline-AC framework. OfflineLight is also implemented under this paradigm. These examples demonstrate the versatility and applicability of the Offline-AC approach in various contexts.

• W1:The novelty is not clear for this work.

  Response:

  1. Innovation of Offline-AC: Offline-AC is a computational paradigm within offline RL that emphasizes innovation in the constraints on actions and values. This framework is not only applying existing methods; instead, it actually encourages novel approaches and breakthroughs in optimizing and balancing these constraints, which is a significant aspect of our work.

  2. Innovation in OfflineLight:Our model, OfflineLight, utilizing established concepts such as PSR and Conservative Q values, achieves SOTA results in offline RL, yielding impressive outcomes. This achievement itself signifies novelty. In parallel, methods such as TD3+BC [1], which combines the existing TD3 and BC approaches, have led to surprising and noteworthy results. Similarly, OfflineLight's innovation lies in integrating and leveraging these known techniques to push the boundaries of performance in TSC. Moreover, our method can be evidenced in our real implementation at some cities.

  [1] A Minimalist Approach to Offline Reinforcement Learning.

Thank you for the insightful comments. We herein address them one by one.

• Q1&W1：Definitions of State and Reward.

  Response:

  We appreciate your suggestion regarding the need of more detailed information about the state definition in our methodology. Our approach is grounded in a decentralized MARL framework, operating within the confines of a POMDP. This is an integral aspect of our research design. In Section 3.3 of our paper, we provided a comprehensive formulation that delineates the essential elements of our approach, particularly as they pertain to the POMDP context. This section is crafted to offer clarity and depth to our methodological framework.

  Each agent in our model concentrates on local traffic information regarding state representation. This includes NV, NV-segments, QL, EP, ERV, and TMP for training process.

  All these state variables are integrated into our OfflineLight model. However, in practical applications, we primarily utilize the queue length (QL) as the state input. This decision is based on our assessment of QL's relevance and effectiveness in capturing the traffic state's critical aspects for our model's objectives. This can be evidenced in our real implementation at some cities.

• Q2&W3:Describe Offline Dataset Collection Procedure.

  Response: We appreciate your comment and would like to provide clarifications as follows:

  1. Only Three Epochs of Training: (1) Our dataset is unique as it incorporates real-world traffic topology and flow data. This approach ensures that each epoch encompasses a significantly large volume of data. The substantial data volume per epoch compensates for what may initially appear as a limited number of epochs. (2) The initial training epochs are particularly crucial for our analysis. In these stages, the reward distribution exhibits a wide range of variability, which is fundamental for algorithmic guidance. This concept is somewhat similar to the Advantage mechanism in Actor-Critic methods like A2C, where the difference between Q-values and value functions is most pronounced in the early stages of learning.

  2. RL Methods Used: To address your query regarding the specific RL methods used for dataset generation in each scenario, we would direct your attention to Appendix Table 5 of our documentation.

  3. Reward Distribution: We show the distribution of reward of our offline dataset in an anonymous URL : https://anonymous.4open.science/r/OfflineLight_rebuttal-3BA9/README.md.

• Q3:Provide more details about the training and evaluation procedures in your experiments?

  Response: We actually detailed the OfflineLight training process in Appendix A. It can be found that OfflineLight quickly converges in the first three training episodes and shows exceptionally stable and uniformly smooth learning curves across five real-world datasets. One of our main contributions is the generalization of the model. Although OfflineLight-JN1 trained in 20% of all data from TSC-OID, it can be deployed directly on other traffic topologies and flows. Intuitively pleasing but with little surprise, OfflineLight-JN1 has almost achieved a similar performance to the SOTA RL methods. Even if the New York's topology and flows have not been collected to generate the offline dataset, we can directly deploy to its scenario. Because the OfflineLight is decentralized MARL with individually reward, the difference of traffic topology has little influence on the final performance.

• W2:OfflineLight does not take into account the interactions between multiple traffic lights.

  Response:

  We acknowledge the potential benefits of integrating approaches that consider interactions between multiple agents. In our next phase of development, we aim to explore the incorporation of offline multi-agent interactions to further improve the results in TSC. However, the current OfflineLight has the advantages as follows:

  1. Our OfflineLight has shown superior performance in offline RL scenarios. Notably, it outperforms methods like CoLight and AttendLight (baseline methods in this paper), which do take into account the interactions between various intersections. This outcome indicates that, in certain contexts, the decentralized approach of OfflineLight can be more effective than interaction-centric models. Moreover, OfflineLight is decentralized MARL with individual reward, and the difference of traffic topology has little influence on the final performance.

  2. Comparison with CTDE Methods: We have conducted comparative analyses with methods such as CTDE. Interestingly, OfflineLight exhibited better performance even in these comparisons. MPLight and CoLight are typical CTDE methods, but our OfflineLight performs better. Additionally, a significant advantage of OfflineLight is that it does not require interaction with the environment, which can be a crucial factor in real-world applicability.