A GPU-accelerated Large-scale Simulator for Transportation System Optimization Benchmarking

Thank you for your review.

I will respond to your questions as follows:

To Weakness about Scenario Validation and Motivation

From our benchmarking results, optimization of signal control leads to the most significant results, followed by congestion pricing. There are two possible explanations for this. First, these two scenarios have received more attention and researchers have proposed some well-designed algorithms, while the other scenarios lack targeted optimization algorithms. One of the main contributions of our work is to propose these scenarios and provide benchmark datasets and simulators to boost the research works on optimization algorithms for these scenarios. Second, these approaches may also be more efficient means of managing and optimizing the transportation system. Our simulator can also be used to test the effects of combining multiple optimization methods. In addition, the development of research on transportation system optimization from computer simulation to real-world applications requires a lot of efforts, including but not limited to algorithmic innovations and real-world scenario validation. An important implication of our research work is to support interested researchers to make more innovations in transportation system optimization algorithms. We believe that only sufficient improvement in effectiveness will allow policymakers to start the long and tedious process of real-world scenario validation.

To Weakness about Lack of Realism

The simplifications, such as excluding intersection overlap, pedestrians, bicycles, and public transport, are consistent with established practices in microscopic traffic simulation, as seen in tools like SUMO, CityFlow, and CBLab. These choices are informed by prior research and are aimed at maintaining a balance between computational efficiency and realism. Although these simplifications do bring about a certain degree of realism reduction, in the results (Figure A3 in Appendix D) of the realism experiments given in this paper, we find that the road traffic status obtained from the simulations are similar to the real ones. We therefore believe that these simplifications are less likely to affect the results of the traffic simulation. On the other side, if reasonable traffic signal settings are provided in the real world (e.g., reasonable phase settings and clearance times), the percentage of intersection overlaps that severely impact vehicle movement in the intersections will occur will be small, and pedestrians and bicyclists will have a minimal impact on vehicle movement. This is the primary role of traffic signal. For public transportation, since this simulator mainly focuses on the movement of vehicles, buses can be considered as vehicles as well, while subways can be simply ignored. Overall, we consider the impact of these simplifications on realism to be within acceptable limits based on experimental results and logical reasoning based on common sense. This conclusion is also consistent with previous works.

To Weakness about Insufficient Long-Term Scenario Evaluation

The main contribution of this paper is to provide high-performance simulators capable of supporting large-scale traffic simulation simulations and to present several research-useful scenarios for optimizing transportation systems. The long-term effects of optimizing the urban transportation system using specific algorithms, such as changes in the travel patterns of residents, are not the subject of this paper. Such content can be investigated by interested researchers by combining the simulator with other techniques (e.g., historical data-based extrapolation, citizen decision-making simulation based on large model intelligences, etc.).

To Weakness about Code Reproducibility Issues

The dataset is now available at https://iclr7274.obs.cn-north-4.myhuaweicloud.com/data.tar.gz (8.57GB). For code, we provide an automated build process based on Github Action in a non-anonymized version and release Python packages on PyPI, and provide documentation with a zero-code wizard via a non-anonymized web service. Due to ICLR's double-blind policy, we are unable to provide the relevant content here. If the article is accepted, we will add the appropriate description in the Camera-ready version.

(Page 1/2)

To Question 1

The primary purpose of including these experimental results in the main text is to demonstrate that our simulator is indeed capable of supporting the five proposed traffic simulation optimization scenarios. As a work that includes benchmarking, and in line with the writing style of previous articles of this kind, we are obliged to provide benchmarking results for the reference of subsequent researchers who conduct further research on the same datasets and simulators in order to minimize their meaningless time invested in repeating the experiments and to accelerate the forward progress of the research in the related fields.

To Question 2

• About Aimsum: We feel very sad because we don't own enough money and the city to get commercial services. So, based on the content of the introduction on Aimsum's homepage, we guess that its business services are implemented based on AI prediction methods, which may include some optimization algorithms as well. Our work provides a free algorithmic research environment for poor researchers like us to support their further innovations in the field of transportation system optimization.
• About Cities Skylines: This game is a very good reproduction of urban planning, urban transportation (including congestion caused by poor signal management), and is a great primer for researchers optimizing transportation systems. However, in the game, you can't precisely control all the details, especially the speed of the traffic simulation. In addition, manually constructing a road network in the game similar to the real world, setting up travel routes for everyone, and developing tools for parsing game logs will be a huge pain for researchers. Therefore, specialized, high-performance, and code-friendly microscopic traffic simulators are a better choice for researchers in related fields.

To Question 3

Our research team has long focused on the application of AI technology to urban transportation problems. When we wished to optimize urban traffic systems using reinforcement learning algorithms, we found that existing simulators had such poor computational efficiency that simulations consumed unacceptable amounts of time when co-optimizing scenarios at multiple intersections. As a result, the idea of designing a large-scale high-performance microscopic traffic simulator arose. We learned the basic logic of running a microscopic traffic simulation from prior work in this area, including the construction of road networks, vehicle following models, vehicle lane changing models, and Python API interfaces. On the basis of these, we independently analyze the computational characteristics of microscopic traffic simulations and design corresponding solutions to achieve GPU computational acceleration.

To Question 4

If you only have 1 GPU, the AI algorithms will compete with the simulator for graphics memory. But if you have 2 GPUs, they can work on different GPUs and not interfere with each other. We don't think this is a serious problem, and we believe that most research teams can afford the extra investment of one GPU in the era of large language models.

To Additional Comments

Thank you very much for checking the references in detail. The DYNASMART reference can be found in the Google Scholar. We will update the citation of Hebo and delete the duplicated references.

To Ethics Concerns

The datasets we use are from open source data or previous works. The dataset for performance evaluation is from CBLab. The dataset for realism evaluation is from [1]. The datasets we use for benchmarks are built from OpenStreetMap and publicly available data represented by satellite imagery. No data on personal privacy is released in this work. We strongly agree with what you say about ethical considerations. If this paper is accepted, we will discuss it in the final version.

[1] Yu, Fudan, et al. "City-scale vehicle trajectory data from traffic camera videos." Scientific data 10.1 (2023): 711.

(Page 2/2)

To Weakness 3 and Question 4

First and foremost, the Arxiv paper on CityFlowER does not provide any evaluation of realism; it only suggests that its methodology can replicate the driving behaviors observed in CityFlow and SUMO. Consequently, we do not consider CityFlowER to inherently offer a more realistic vehicle model. While CityFlowER indeed proposes a viable path towards achieving more realistic vehicle models, the ultimate realism is highly contingent upon the quality and scope of the training data used. Regrettably, the dataset we employed for evaluating realism lacked sufficient vehicle trajectory data necessary for effective training. This limitation precluded us from conducting a comprehensive evaluation. We acknowledge the importance of such comparisons and plan to incorporate advanced driving models in future work to enhance the realism of our simulator further.

To Weakness 4

For the performance evaluation, we directly utilized datasets from CBLab. Specifically, Roadnet-S corresponds to eff_1e2inter from CBLab's datasets, Roadnet-M corresponds to Shanghai, and Roadnet-L corresponds to eff_1e6veh_1e4.5inter. For the hardware configuration, we mentioned in the main text that the relevant information is provided in Appendix C. The CPU we use is Intel(R) Xeon(R) Platinum 8462Y CPU (64 threads). We do not comment on why the performance comparison between CBLab and CityFlow is inconsistent with CBLab's paper. First, the difference in performance between the two is not relevant to the innovation of this paper, and this paper is under no duty to reproduce the results reported by CBLab. Second, the computational performance of a software system is affected by a number of factors, such as the version of the compiler used, compilation options, the number of CPU cores, CPU cache, other loads running on the machine, etc. All we can do is to run baselines as fast as we can and report the results honestly.

To Weakness 5

We apologize for submitting the paper in an effort to find a suitable way to present the data without violating the double-blind principle. Now you can download the data at https://iclr7274.obs.cn-north-4.myhuaweicloud.com/data.tar.gz (8.57GB). If the paper is accepted, a great deal of content that is not publicly available due to the double-blind principle will be accessible, including the data structures we use, documentation, online services, and more.

To Question 2

For a software system, more computation means that it takes longer for the computer hardware to process, which reduces operational efficiency. As the number of vehicles in traffic simulation increases with the size of the road network, the increase in computational time is unavoidable. Our mention of “large-scale” implies that our proposed simulator is capable of performing computational tasks in city-scale scenarios with small time consumption, rather than implying that the scale of the simulation can be scaled infinitely without increasing the time spent.

To Question 3

The detailed explanation of the scenarios has been provided in Appendix F. The detailed explanation of the metrics has been provided in Section 3.3.

(Page 2/2)

Really thank you for your review. We will answer your questions as follows:

To Weakness 1

Thank you very much for bringing to our attention the recent advancements in transportation simulation. Due to the earlier submission date of our manuscript, we were unable to incorporate these recent works from Arxiv or other publications. We appreciate your effort in highlighting these studies, which indeed show that this field is gaining significant attention, contrary to what reviewer 6yoX09 suggested.

After thoroughly reviewing the manuscript and the open-source code of the relevant literature, here are our responses to your inquiries:

1. GPUDrive: This simulator focuses on autonomous driving scenarios, which significantly differs from our work in three key aspects:

   • Vehicle Movement: In GPUDrive, vehicle movement is determined by input acceleration and angular velocity, with the road network serving merely as a benchmark for assessing the operational strategy's realism. In contrast, our simulator ensures vehicles move along lane lines, with car-following and lane-changing algorithms governing acceleration and lane-switching decisions.
   • Optimization Goals: GPUDrive's external algorithms target the enhancement of individual vehicle autopilot algorithms. Our research, however, aims at optimizing city-wide traffic efficiency by modifying traffic infrastructure and operational strategies.
   • Simulation Scale: GPUDrive supports hundreds of agents, whereas our simulator can handle over a million vehicles. Thus, our work belongs more to the domain of automated driving simulation rather than microscopic traffic simulation.

2. CityFlowER: This work seeks to enhance simulation realism by utilizing neural networks for car-following and lane-changing models. While this method holds potential for increased realism with sufficient real-world training data, such data is not reported in their study. Our focus is on achieving large-scale simulation with reasonable realism. According to the CityFlowER paper, it performs significantly worse than CityFlow and does not demonstrate higher realism, making it less significant in the context of related work. As for comparing the realism of our simulator with CityFlowER, the realism test data we have does not have detailed vehicle movement processes, only vehicle routes and road conditions, and therefore it is not possible to train the CityFlowER model to compare results. Nevertheless, we acknowledge CityFlowER's promise in microscopic traffic simulation.

3. GEMSim: This is a GPU-based mesoscopic traffic simulator, not a microscopic one. Mesoscopic simulators typically simplify vehicle behavior, akin to the waiting queue model used by MATSim, as mentioned in our paper. Our primary contribution is implementing large-scale parallel computational acceleration using a microscopic traffic simulation model to simulate traffic scenarios with high realism. This work can be included in the mesoscopic traffic simulation section of Related Works.

4. Large scale multi-GPU based parallel traffic simulation for accelerated traffic assignment and propagation: Although this paper claims to be a GPU-based microscopic traffic simulator, its model simplification approach — dividing roads into discrete 1-meter spaces — is uncommon. The paper does not address realism differences in simulation results. In contrast, our work achieves GPU acceleration without model simplification, ensuring realism.

In summary, compared to these works, our work uniquely achieves large-scale microscopic traffic simulation without model simplification, maintaining the realism of the simulation results.

To Weakness 2 and Question 1

As what you said, how to manage the interaction between vehicles is an important issue to ensure realism. This is one of the core concerns as well as one of the main contributions of our work. In lines 226-228 of the original paper, we make it clear that a vehicle's sense range needs to include the vehicle in front of it in the current lane and the vehicle in front of it and behind it in the adjacent lane. To solve the problem, we implement SIMD-friendly vehicle sensing indexes and discuss the design in the lines 248-266. The simulation of individual vehicles is indeed homogenized when the sensing of the surrounding vehicles is taken into account (although there are times when the corresponding computations are not needed).

(Page 1/2)

Thank you very much for your quick reply and we will post our further thoughts in response to your question.

1. There are no further comments, and we have fully explained the reasons for not including some of the work, and the way in which some of the work was included, in the previous comment.
2. SIMD itself is not the same thing as how to properly develop software using SIMD technology. In the case of the CPU, for example, this reviewer seems to be saying that all programs working on the CPU use the standard technology of the X86 instruction set, and therefore they are all simple applications of the X86 instruction set, and there is nothing innovative about them. We are very sad that the reviewer does not appear to have any background in computer expertise (e.g. Computer Architecture).
3. No more comment.
4. We sent an email to the original author of CBLab some time ago to discuss this and got no response. We are not responsible for the realism of the data reported in the CBLab paper, we are only responsible for the data reported in our paper.

As to whether ICLR is the best place for this work, we believe that at least ICLR is the proper place for this work because this work is primarily aimed at researchers who wish to solve problems in urban transportation systems through reinforcement learning algorithms.

Thank you for your review. I will list the responses to your questions as follows:

To Weakness: Overall, the two key designs presented in Section 3.1 are major contributors to improving efficiency and performance. In other words, both designs are indispensable. They are therefore of equal importance and should be considered as a whole. According to the idea of the paper, GPUs are the hardware base for enabling computational acceleration, and the task of software system design is to best adapt to the hardware. We identify two major difficulties (read/write conflict & vehicle sensing indexes) in the adaptation process and provide solutions (two-phase parallel process & linked-list based vehicle sensing indexes) that ultimately allow the GPU's performance to be fully utilized for acceleration.

To Question 1: The answer is YES. Our simulator supports user-configurable inputs of kinematic and IDM model parameters for each vehicle, including maximum acceleration, general acceleration, maximum braking acceleration, general braking acceleration, headway, and so on. Users can build their own vehicle input data that they want to match the desired scenario, and we provide a tool chain (mosstool mentioned in Appendix A) to support such needs.

To Question 2: It is easy to develop a new car-following model by editing a few codes. As shown in the _CarFollow(Person& p, PersonNode& node, Lane* lane, float step_interval, uint& ahead_id) function, we have implemented both the IDM model and the Krauss model. (https://anonymous.4open.science/r/moss-AF45/src/entity/person/vehicle.cu: line 397) We hope that the open source code and the ICLR conference will attract more people to participate in the collaboration to provide more optional vehicle models.

Thank you very much for taking your valuable time to review our paper.

Our work, a large-scale microscopic traffic simulator, serves the fundamentals of transportation system optimization to support typical transportation system optimization scenarios. We believe that the efficiency of urban transportation systems is closely related to each individual, and that learning-based AI technologies have great potential to optimize the efficiency of transportation systems. However, as you say, there are fewer researchers focusing on this area right now. We believe this is because there is a lack of large-scale simulators and benchmarking code that can support a larger number of scenarios, leaving researchers unmotivated to investigate the application of learning methods to these problems. That is our aim in accomplishing such a non-incremental, pioneering and difficult work.

Moreover, memory usage analysis will be added to the supplement as part of the performance evaluation.