3D Dense Captioning beyond Nouns: A Middleware for Autonomous Driving

We would like to thank R3EY for generally positive feedback. Here we respond to two concerns:

Q1: usefullness of such a dense caption dataset.

We would like to remark on the effectiveness of the DESIGN middleware from three perspectives. Firstly, as demonstrated by the UniAD experiment in Table.4, DESIGN can significantly improve the collision rate by a relative margin of 25.8% (from 0.31 to 0.23), while achieving comparable L2 (from 1.03 to 1.05). We argue that collision rate is a more meaningful indicator than L2 for autonomous driving, so this result is significant enough to demonstrate the effectiveness of DESIGN. Secondly, we hypothesize (with all due respect) that R3EY may feel the UniAD experiment does not contain enough datapoints. In this regard, we provide another datapoint in Table.4 that directly turns the exact real numbers into natural language descriptions. This representation underperforms DESIGN. This datapoint demonstrates that the performance gain brought by DESIGN is non-trivial and serves as another evidence of DESIGN's effectiveness. Thirdly, we evaluate in another setting that shows DESIGN can also improve detection performance. The results below show that the DESIGN-former outperforms BEVFormer in mAP and NDS, suggesting that the cues from language helps to regularize the middle representation of BEV features and improve the performance of the bounding box detection. This serves as another evidence that DESIGN has some superiority over an existing middleware that is a subset of DESIGN (i.e., 3D bounding boxes).

Q2: difference compared with the original random initialized planning query.

We would like to clarify the differences between UniAD and our approach. We can see in the last example of Fig.19, a white truck in ego lane blocks the way. The UniAD tries to merge to the right lane and collides with the silver car on the right lane, while our driving model stops to avoid going against the flow of traffic and thus avoids the collision with right car. We think this introduction of reasoning ability of large language models to driving model remarks the difference between UniAD and our approach.

We thank a5o2 for these constructive feedbacks. Here we respond to five concerns one by one:

Q1: Effectiveness for downstream tasks.

We would like to remark on the effectiveness of the DESIGN middleware from three perspectives. Firstly, as demonstrated by the end-to-end driving experiment in Table.4, DESIGN can significantly decrease the collision rate by a relative margin of 25.8% (from 0.31 to 0.23), while achieving comparable L2 (from 1.03 to 1.05). We argue that collision rate is a more meaningful indicator than L2 for autonomous driving, so this result is significant enough to demonstrate the effectiveness of DESIGN. Secondly, we hypothesize (with all due respect) that a5o2 may feel the UniAD experiment does not contain enough datapoints. In this regard, we provide another datapoint in Table.4 that directly turns the exact real numbers into natural language descriptions. This representation underperforms DESIGN. This datapoint demonstrates that the performance gain brought by DESIGN is non-trivial and serves as another evidence of DESIGN's effectiveness. Thirdly, we evaluate in another setting that shows DESIGN can also improve detection performance. The results below show that the DESIGN-former outperforms BEVFormer[1] in mAP and NDS, suggesting that the cues from language helps to regularize the middle representation of BEV features and improve the performance of the bounding box detection. This serves as another evidence that DESIGN has some superiority over an existing middleware that is a subset of DESIGN (i.e., 3D bounding boxes).

Q2: Do not explicitly demonstrate the learned representations enable human-like reasoning.

We would like to respond to this concern from three perspectives: (1) The ReasonNet paper has been cited and discussed in an updated version, but annotating the ReasonNet dataset with DESIGN is not feasible in the rebuttal window. (2) Our UniAD experiment (Table.4) has a module that requires interaction with GPT-3.5, which is demonstrated in Fig.14 and Fig.16. We hypothesize (with all due respect) that a5o2 feels the cases shown in these two figures are not human-like enough, so we provide two more cases into Fig.18.

Q3: More challenging scenarios.

We thank the reviewer for this suggestion. We provide more visualization results of our driving model in challenging scenarios in Fig.19. For example, in the last example of Fig.19, a white truck in ego lane blocks the way. The UniAD tries to merge to the right lane and collides with the silver car on the right lane, while our driving model stops to avoid the collision with right car.

Q4: Uncertain intents of an object and the ability to facilitate forecasting.

To clarify, the DESIGN representation contains the moving status of 3D bounding boxes. For example, this is a ground truth caption in Fig.12: A stroller in the front left of ego car about 11 meters away is moving slowly to ego car in the crosswalk.

Here 'moving slowly to ego car' covers the moving trend of the object. As for 'uncertain intents', we think the rough descriptions of 'slowly' or 'quickly' have intrinsically modeled the uncertainty.

Q5: Comparisons with more baselines.

We thank a5o2 for pointing out these references and we have cited them in the updated paper. We compare with the newest SOTA baseline Vote2Cap-DETR in the figure below. There are two settings: the first one exploits a standard cross entropy loss (denoted as maximum likelihood estimation, MLE) and the second one exploits a self-critical sequence training (denoted as SCST) that optimizes towards a better CIDEr metric. As for 3DJCG and D3Net, since they are older baselines and involve a substantial modification to the dataloader for joint captioning and grounding, we have not yet evaluated in the rebuttal window.

We thank rzAd for these constructive feedbacks. Here we respond to ten concerns one by one:

Q0: slightly improve the performance.

We first clarify that our model outperforms previous work greatly because collision rate significantly improved. As demonstrated by the UniAD experiment in Table.4, DESIGN can significantly improve the collision rate by a relative margin of 25.8% (from 0.31 to 0.23), while achieving comparable L2 (from 1.03 to 1.05). L2 is not a good metric for safety-critical scenarios and a lower collision rate is preferred for safety over naive trajectory mimicking. So we argue this is a significant instead of slight improvement over the award-winning baseline UniAD.

Q1: (1) does not go beyond previous definitions in terms of object-level comprehensiveness.

We do use the pre-labelled 3D boxes from nuScenes, but to clarify, nuScenes annotate many boxes with the category of 'others' or 'pushable_pullable'. Our nuDESIGN dataset extends these annotations to detailed open-set entities like 'goose' or 'stroller'. So we argue that the object-level comprehensiveness is promoted.

(2) attach more meta data.

Directly attaching meta data, expressed in real numbers describing geometric entities like [-2.60m/s, 1.64m/s] or [-23.78m, 0.05m], to the caption would greatly challenge LLM since they are not natural language. We conduct experiments with this new version of captioning, where we attach meta data to the caption. The downstream planning modules follow the same protocol. This representation underperforms DESIGN as shown below. This experiment demonstrates that the performance gain brought by DESIGN is non-trivial and serves as another evidence of DESIGN's effectiveness.

Q2: (1) most words are about objects' colors .

Indeed, we also include other important information like material, gloss, status (like 'clean') and most importantly, open-set entities. See Fig.8.

(2) drops the heading angle and accurate velocity.

To clarify, the DESIGN representation contains the moving status of 3D bounding boxes. For example, this is a ground truth caption in Fig.12: A stroller in the front left of ego car about 11 meters away is moving slowly to ego car in the crosswalk. As for whether accurate descriptions or rough natural language descriptions are better, the experiment above demonstrates that the 'metadata' representation underperforms DESIGN.

Q3: over-claim of 'scalable automatic labeling pipeline.

Indeed, our labeling pipeline is not fully automatic but an auto+manual one. A fully automatic labeling pipeline would generate pseudo labels and is not what we want. We would like to clarify that our annotation method is easy to expand. With this pipeline, a worker only needs to decide what is right and what is wrong, instead of writing down every sentence. If the caption is considered right, it is included into the ground truth. If the caption is considered wrong, we just send the instance into the next round. Our annotation process takes three rounds, which are 0-20%, 20%-50%, and 50-100% of the data volume. We have counted the number of manual corrections in the three rounds, which was about 42.1%, 23.3% and 9.7% respectively.

Q4: (1) More baselines.

We compare with the newest SOTA baseline Vote2Cap-DETR in the figure below. There are two settings: the first one exploits a standard cross entropy loss (denoted as maximum likelihood estimation, MLE) and the second one exploits a self-critical sequence training (denoted as SCST) that optimizes towards a better CIDEr metric. We can see that our model outperforms Vote2cap-DETR .

(2) 2D dense captioning.

We show the results of 2D dense captioning on Tab.20.