TSTTC: A Large-Scale Dataset for Time-to-Contact Estimation in Driving Scenarios

(-) The addition of NeRF-generated sequences is an interesting idea, but the exact impact of these synthetic data on the model’s generalization capabilities is not sufficiently explored. More detailed analysis of how much NeRF data improves performance, especially on smaller TTC values, would be helpful.

(-) The manuscript lacks thorough ablation studies for the proposed Pixel MSE and Deep Scale methods. It would be beneficial to break down the impact of different components (e.g., center shift, different scale bins) and analyze their contributions to the final performance.

(-) The manuscript does not address the computational complexity or real-time performance of the proposed TTC estimation methods. In real-world applications like autonomous driving, it is crucial to assess the trade-off between accuracy and computational cost, especially for resource-constrained systems.

(-) The dataset is primarily focused on highway and urban driving scenes, which limits its applicability in more complex scenarios like pedestrian-rich environments or non-vehicle objects. Including data for other road users, such as pedestrians and cyclists, would broaden the scope of the dataset.

Justification of Rating

While I do not have extensive experience in evaluating dataset papers, I can appreciate the significance of introducing the TSTTC dataset for TTC estimation in autonomous driving scenarios.

The dataset fills a notable gap by providing large-scale, real-world data for both urban and highway driving, which will undoubtedly be useful for researchers working on Time-to-Contact estimation tasks. The inclusion of NeRF-generated sequences for small TTC cases is an innovative approach to addressing the imbalance in real-world data, although a more detailed analysis of the impact of this synthetic data is needed.

The proposed TTC estimation methods, Pixel MSE, and Deep Scale, provide useful baselines for further research, but the manuscript could be improved by discussing their computational complexity and real-time performance, which are crucial factors for practical deployment in ADAS systems.

Additionally, the paper lacks thorough ablation studies and analysis of the individual contributions of different components in the methods.

Given these factors, I would leave more room for other reviewers to determine the overall contribution and novelty of this work.

1. The layout caption of Figure 2 on page 5 requires refinement.
2. It is recommended that the author provide a flowchart to illustrate further details on the continuous monocular image generation of NeRF images.
3. Lighting conditions (such as nighttime or low-light environments) or weather (such as rain or snow) affect the quality of NeRF rendering? The author could include some statistics regarding these factors in the dataset, as these issues are highly relevant in autonomous driving scenarios.
4. In real driving scenarios, the motion trajectories of objects can be highly complex, which may lead to cumulative errors in speed and depth estimation, ultimately affecting the accuracy of TTC estimation. It is recommended that the author, after the derivation of the formulas, further discuss how to reduce the impact of these errors on the final results, for example, by using filtering or other smoothing techniques.
5. In the future, it is necessary to increase the diversity of scene types in the dataset.

1. It is unclear about the quality of the ground-truth labels. The authors should conduct experiments to show the quality of labels.

2. Albeit the dataset is large, what does the dataset bring? Can other methods benefit from using the dataset? In other words, a cross-dataset experiment is needed.

3. The introduction of the Nerf dataset is somewhat unclear. Why should it be used? How realistic is the dataset?

4. The authors use the MiD metric as the main metric rather than the RTE metric by stating that "Due to the instability of TTC at larger value". This makes the reviewer wonder about the validity of the TTC metric and the correctness of the proposed approaches.

I have concerns regarding to the following 3 aspects:

1. Dataset.

• Object range. The paper highlights their dataset contributions in Table 1 and bolds the object range tag. However, from the histogram in Figure 3, the objects farther than 200m are relatively few. I wish to see a histogram comparing to existing datasets, like Argoverse 2 [1] and Cityscapes3D [2] on a relative number of distant 3D objects.

• From L186, the paper mentions all real-world data are collected from commercial trucks. I am wondering what the object size of the trucks is. Whether the dataset is available for ADAS system on commercial cars, instead of large trucks.

• NeRF scenes. From Figure 8, I can see a visual domain gap from the NeRF reconstructed scenes and the real-world scenes. Thus, I am wondering the effectiveness of introducing Nerf scenes as well. Whether it can bring a better training performance, or whether it makes the testing harder? If it makes the testing scenarios harder, whether this difficulty comes from the challenges in domain gap?

[1] Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting

[2] Cityscapes 3D: Dataset and Benchmark for 9 DoF Vehicle Detection

2. Method.

• Method details. In L287, this papers mentions the Eq. (4) is derived from Eq. (1) and Eq. (3). However, it is not obvious to me how it achieves. I suggest the author to add more details on the combination.

3. Experiments.

• Redundancy design. The paper highlights TTC estimation as a redundancy design for ADAS system. Therefore, I am wondering, whether the TTC estimation helps the planning or the ADAS system. I suggest some experimental results demonstrating TTC estimation helps planning or other components in autonomous driving, instead solely experiments on TTC estimation. This is to show TTC estimation really helps.

• Some additional experiments. As the TTC estimation can either be achieved by scale difference estimation and depth-velocity estimation as well, (also, can be achieved by motion prediction as well?), I suggest the authors to compare their methods with other alternative methods.

• Distance breakdowns are suggested. TTC estimation errors with different object distance ranges.