Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection

1. Why not just predict foreground instance-level occupancy in the whole scene, instead of pursuing higher detection accuracy by using the occupancy results?

   1. To predict foreground instance-level occupancy for the entire scene, it is essential to distinguish the foreground from the background. However, separating the foreground from the background without detection results is a non-trivial task. Therefore, we leverage detection to obtain foreground objects.
   2. Pursuing higher detection accuracy is not our main objective. Our main goal is to obtain object-centric occupancy to provide a more flexible representation for downstream driving tasks. However, when we attempted to aggregate multiple frames of occupancy grids to make the shape more complete, we realized that these occupancy grids could also serve as an excellent medium for multiple frame information fusion. Thus, we designed a simple fusion network to examine the benefits for the detection task.

2. Time and memory cost

   Since object tracklets vary in length, our method's running time may also vary with different inputs. Additionally, the dimension of the decoded object-centric occupancy depends on the detected bounding box. To ensure fair testing of running time, we standardized the input length to 32 and set the number of decode queries to 4096.

   We conducted the inference with batch_size = 1 using standardized inputs on a single 3090 GPU and computed the average running costs. The results are presented in the table below. We can observe that using the shape decoder does not significantly affect the computational cost.

   model	Avg. Inference Time	Avg GPU memory Cost
   w/o shape decoder	4.08ms	2499MB
   w/ shape decoder	4.23ms	2565MB

3. Some methods, like VoxelNeXt, FSDv2, HEDNet are missing and are not compared in Table 1.

   Thanks for the suggestions. We mainly consider tracklets from GT, CenterPoint, and FSD in Table 1 because we used their tracklet proposals on the training set to train our model. Basically, our method can generalize to other detector without retraininig.

   The following table presents occupancy completion results obtained by directly applying our trained model to the tracklet proposals generated by FSDv2 on the testing data.

   Tracklet Inputs	IoU %	mIoU (track) %	mIoU (box) %
   FSD v2 (no train)	61.41	47.69	60.76
   FSD	62.84	54.12	61.58
   CenterPoint	57.99	44.94	55.10

   Due to better detection, our method with FSDv2 still outperforms the version with CenterPoint even without retraining. However, it performs slightly worse compared to using FSD tracklets, despite FSDv2 having better detection results than FSD. This indicates that significant detection improvements generally lead to better shape completion (FSDv2 vs. CenterPoint). However, for detectors with similar performance (e.g., FSD vs. FSDv2), improved detections do not necessarily guarantee better shape completion without retraining. Retraining our method using training proposals generated by FSDv2 may address this issue.

   We will add these results to the Table 1 and discuss the findings in our revised manuscript.

   Besides, we’ve conducted a detection experiment where we applied our method to 1-frame FSDv2 without retraining.

   The following table demonstrates that our method with a stronger detector continues to show detection improvement even without retraining.

   Model	Vehicle L1 mAP/mAPH	Vehicle L2 mAP/mAPH
   FSDv2	79.8/79.3	71.4/71.0
   FSDv2 +Ours	83.2/82.7	75.2/74.7

   We will include these results in our revised Table 2 We will also include VoxelNeXt, FSDv2, HEDNet for a more comprehensive comparison. In fact, our method based on single-frame FSD outperforms all these mentioned methods by noticeable margins.

4. Typos/mis-leading descriptions. For example, ‘Tab. 5.4’ on line 351 -> ‘Tab. 3’.

   Thank you for the thorough review. We will correct the typos and conduct meticulous proofreading for our revised submission.

1. Creating detailed occupancy for each object seems unnecessary. In most downstream tasks in autonomous driving, using bounding boxes (bboxes) is sufficient.

   We respectively disagree with this statement. As highlight in our introduction, using bboxes alone “fails to capture the intricate details of objects’ shape, particularly for objects with irregular geometries”. As illustrated in Fig. 1, the bbox of the crane inevitably includes unoccupied space, which is however could be the drivable space for the ego-car. Relying solely on bboxes limits the downstream planner’s ability to leverage this free space effectively. Our uploaded PDF also includes samples that showcase the advantages of object-centric occupancy over bounding boxes in representing complex shape structures.

2. The performance improvement primarily stems from temporal feature fusion, which lacks significant technical innovation.

   Indeed, our contribution does not focus on improving detection performance. As indicated in our title, we aim to take a step 'toward more flexible 3D perception,' achieved by learning an object-centric occupancy representation. This object-centric nature allows us to aggregate temporal information from a very long sequence (e.g., 200 frames) while maintaining an affordable computational cost. Compared to previous long sequence methods, our method can additionally output object occupancy within each bbox to support precise downstream driving tasks. Additionally, with a simple temporal fusion strategy, our method surpasses previous state-of-the-art approaches in online detection, demonstrating the effectiveness of our strategy.

3. It is unclear whether the loss on occ heads in Fig. 4 enhances detection performance. The authors should compare detection performance with and without occ heads after obtaining the Shape Emb. Z to determine if occ heads contribute to learning useful features, such as yaw estimation.

   Thank you for the suggestion. We removed the OCC head from our full model and trained the model using only the detection loss. The results are presented in the table below. A noticeable drop in detection performance is observed when the OCC decoder is removed.

   Model	L1 mAP/mAPH	L2 mAP/mAPH
   No Occ Dec	81.1 /80.4	73.0/ 72.3
   Ours	82.8/82.3	74.8/74.4

I have read all the reviews and authors' responses. My concern has been well addressed. so I raise my rating to 5.

1. The experimental results are only obtained on the Waymo Open Dataset. It will be nicer to conduct the experiments on nuScenes or Argoverse 2 to validate its robustness for different datasets.

   Thanks for the suggestions. Currently, we are not able to train/test our method on nuScenes or Argoverse 2 since we only prepared object-centric occupancy labels on Waymo. We will support nuScenes or Argoverse2 after our occupancy labels on Waymo is released.

2. Although the authors say it is a new task, so there are no learning baselines for shape completion, it will be interesting to compare the results with other scene occupancy methods. So that we can see the flaws of using coarse resolution quantitatively.

   Thank you for the suggestion. To the best of our knowledge, no existing scene-level occupancy method has published results on Waymo Open dataset. We will release the results when their results on Waymo are published.

3. The extrapolated results of shape completion are interesting, showing that it can achieve a performance similar to that of using GT boxes. Will it also help with 3D Object Detection results?

   Our detection already benefits from the extrapolated shape information, as the shape embedding used for occupancy decoding is also fused into the detection feature. Compared to explicitly using the generated occupancy, incorporating the feature directly ensures minimal information loss.

1. Missing related works

   Thank you for the suggestion. We will include a discussion of these works in our revised version and provide a more thorough related works section.

2. Renderings of the shape codes;

   In the uploaded PDF, we’ve included several renderings. These renderings were obtained by applying marching cubes to the decoded volumetric grids using a level of 0.5, as suggested. The renderings demonstrate that our method can complete shapes even when the current point cloud is extremely sparse.

   Due to the use of 0.2m voxel size, the resolution of our predicted occupancy may not support high-quality rendering. For example, the resolution for a typical sedan (let's assume its dimensions are 4.5m* 1.8m * 1.4m) under our voxel size is 23 * 9 * 7. In contrast, common shape completion methods typically use a resolution of 128 x 128 x 128 or higher to facilitate high-quality rendering.

   It should be noted that for our purposes, high-quality rendering is not required. Although the selected voxel size of 0.2 meters may not provide highly detailed rendering, it is sufficient for downstream driving tasks and ensures computational affordability.

3. The model takes in a series of 3D BBs and outputs one updated 3D BB - at which timestamp is this 3D BB output? The latest?

   The behavior depends on the context. During inference, we only need to output the latest one. However, during training, we can simultaneously obtain bounding box outputs at all timestamps with just one forward pass using a causal attention mask. We compute the detection loss over all these bounding boxes to facilitate training. We will clarify this in our revised version.