ODG: Occupancy Prediction Using Dual Gaussians

We thank reviewer XGZs for the constructive feedbacks. We address the reviewer's questions and comments below.

$**Visualize the scene with static and dynamic queries prediction alongside each other**$:

We have prepared such visualizations and will include them in the final version to further demonstrate the superiority of our proposed ODG. Unfortunately, this year's NeurIPS rebuttal format does not support uploading a PDF for the rebuttal and as such, we are not able to share visualization results during the rebuttal stage.

$**Metrics of predictions from each layer**$:

We evaluated predictions of all 6 layers of ODG-T against ground-truth occupancy and computed metrics. The results are summarized in Table 1 below. We can clearly see that in the beginning since the prediction is too coarse, the model behaves poorly in terms of mIoU and RayIoU. Then with our coarse-to-fine refinement, ODG starts gradually learning the scene and gives progressively better results, which validates the efficacy of our coarse-to-fine refinement design.

Table 1. Evaluation of predictions of all 6 layers of ODG-T on nuScenes.

$**Query efficiency comparison**$:

We compare with OPUS [1] which is a SotA query-based method for occupancy. The results are summarized in Table 2 below. We note that to ensure a fair comparison, the results of OPUS are directly obtained from their paper. It is clear that with the same number of total queries, ODG outperforms OPUS with significant margin, while only incurring a minimal overhead. For treating all queries as dynamic (denoted as ODG* in Table 2), we can see that even though ODG experienced a minor drop in terms of mIoU and RayIoU, it still definitively outperformed baseline OPUS. We attribute this gain to the object detection task present in our system which effectively improves dynamic object predictions, and through rendering constraint we also learn better 3D geometry.

Table 2. Query efficiency comparison.

[1] Wang, Jiabao, et al. "Opus: occupancy prediction using a sparse set." Advances in Neural Information Processing Systems 37 (2024): 119861-119885.

$**Velocity estimation evaluation**$:

Since our velocity is approximated by ego vehicle at different image timestamps and there is no velocity ground-truth, hence direct quantitative evaluation is not feasible in this case. However, we validate the effectiveness of the dynamic queries predictions through the following surrogate experiment on nuScenes: given current timestamp $t$ (keyframe), we denote occupancy ground-truth $O_t$ and predictions $O_t^p$. We denote the immediate previous timestamp $\hat{t}$ (also keyframe, where occupancy GT is available) and its occupancy GT $O_{\hat{t}}$ and predictions $O_{\hat{t}}^p$. Then we warp prediction at current time $O_t^p$ to previous time using ego-motion for static queries and velocity for dynamic queries and denote as $O_{t\rightarrow\hat{t}}^p$. We evaluation both $O_{\hat{t}}^p$ and $O_{t\rightarrow\hat{t}}^p$ against ground-truth $O_{\hat{t}}$ at time $\hat{t}$. The results are summarized in Table 3 below:

Table 3. Surrogate experiment of evaluating velocity estimation.

We can clearly see that due to motion misalignment if we directly evaluate prediction $O_{t}^p$ against GT $O_{\hat{t}}$ at previous time, there is a significant degradation in terms of mIoU and RayIoU. But if we warp current prediction $O_t^p$ to previous time $O_{t\rightarrow\hat{t}}^p$ using estimated velocities of the queries, such misalignment is corrected and we obtain significantly improved results. This demonstrates the usefulness of our estimated velocity.

We note that we will add velocity flow visualization in the final version, given during the rebuttal stage it is not feasible for us to share any type of rich media such as images and videos.

$**Ablation on only using rendering supervision**$:

We conducted such ablation experiment with ODG-T (trained for 24 epochs on nuScenes, consistent with Table 4 in main paper) and the results are summarized in Table 4 below:

Table 4. More ablation studies of ODG-T including only using rendering supervision. Trained for 24 epochs on nuScenes.

we can see that even with only rendering supervision, our method is able to achieve a modest improvement, validating the effectiveness of our design.

$**Inconsistent metric reporting on RayIoU**$:

We have corrected such inconsistency by adding $\text{RayIoU}$ as last column in Table 4 in the main paper. The changes will be reflected in the final version.

$**Stableness of improvement coming from dynamic motion compensation**$:

We launched another three separate experiments with distinctively different seeds to validate the stability of improvement. The results are summarized in Table 5 below. We can see that the metrics yielded across runs with different seeds are very stable, demonstrating the validity of the improvement from dynamic motion compensation. To facilitate reproducing, experiments were conducted using PyTorch 2.1.2, NVIDIA Driver 535.161.08 with CUDA 12.1 on A100 GPUs.

Table 5. Multiple experiments of ODG-T with different seeds to validate stableness of improvement coming from dynamic motion compensation. Trained for 24 epochs on nuScenes.

$**Writing and notation polishing**$:

We have condensed Eq.12-13 into a single-line text description and added the definition of $P$ in Eq. 16. The changes will be reflected in the final version.

We thank reviewer M77i for the constructive feedbacks. We address the reviewer's questions and comments below.

$**Contributions on dynamic and static modeling:**$

Designing an occupancy prediction network that properly considers both dynamic and static objects is not trivial and requires several careful designs. First, it is critical to create a proper learning scheme for dynamic queries, as velocity ground truth is not available. Without proper motion modeling and learning, one cannot correctly aggregate information across time steps. In addition, it is also important to introduce some form of feature interaction between dynamic and static queries in an efficient way, without which the occupancy prediction performance would not be good and/or significant additional computation cost would be incurred. Moreover, our explicit 3D representation using 3D Gaussians also makes it more efficient and feasible to incorporate motion of dynamic objects, as compared to commonly used dense representations like birds-eye-view or 3D voxels which would require expensive 2D/3D warping.

$**Comparing to OPUS and SparseOcc**$:

While OPUS [1] and SparseOcc [2] leverage some forms of coarse-to-fine design, their models miss certain important factors and/or require some cumbersome designs. For instance, OPUS is a query-based method using points as scene representation, which cannot directly be used for rendering. In addition, OPUS does not consider the motion of dynamic objects at all, which makes the modeling incomplete as there are almost always dynamic objects in a driving scene. SparseOcc employs a series of sparse voxel decoders to filter out empty grids and predict occupancy state of retained voxels, which is inefficient. In contrast, we predict 3D Gaussians in a streamlined fashion without any pre-defined locations and takes dynamic objects into account. Our proposed ODG introduces a new design offering more complete modeling, higher efficiency, as well as the possibility of leveraging additional supervision, which goes beyond incremental improvements.

[1] Wang, Jiabao, et al. "Opus: occupancy prediction using a sparse set." Advances in Neural Information Processing Systems. 2024

[2] Liu, Haisong, et al. "Fully sparse 3d occupancy prediction." European Conference on Computer Vision. 2024.

$**Difference from RenderOcc**$:

While RenderOcc also leverages neural rendering to improve occupancy prediction, it is NeRF-based which carries a significantly higher cost in terms of training time and memory required. In addition, RenderOcc adopts BEVStereo4D [1] as feature representation, which incurs a much higher inference runtime as shown in Table 1 in our paper. In contrast, in ODG our query-based representation enables a much more efficient scene representation, while also leveraging Gaussian splatting resulting in much faster rendering and significantly reduced training and inference cost.

[1] Li, Yinhao, et al. "Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 2. 2023.

$**Leveraging additional supervision signals**$:

We would like to point out that being able to efficiently leverage additional supervision signals (when available) is part of our contributions. Previous works either cannot incorporate these supervisions, e.g., OPUS [1] does not support rendering, or does it in a very expensive way, e.g., RenderOcc uses volume rendering which is both computationally and memory expensive. Moreover, many of these existing works do not model dynamic objects, making it not possible to properly apply the extra supervisions like rendering loss.

Also, we note that we do not use more ground-truth information as compared to other recent occupancy prediction methods. In order to create the occupancy labels, 3D object segmentation ground truths are used (see Occ3D [3]) which is actually more expressive than 3D bounding boxes. We obtain 2D sparse depth and semantic maps by projecting from LiDAR and the ground-truth 3D segmentation, which have also been used to create occupancy labels in existing Occ3D benchmark.

[1] Wang, Jiabao, et al. "Opus: occupancy prediction using a sparse set." Advances in Neural Information Processing Systems. 2024

[2] Liu, Haisong, et al. "Fully sparse 3d occupancy prediction." European Conference on Computer Vision. 2024.

[3] Tian, Xiaoyu, et al. "Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving." Advances in Neural Information Processing Systems. 2023

$**Whether each dynamic query corresponds to a single semantic class**$:

We do not impose such a constraint explicitly in the model. However, each dynamic query is implicitly encouraged to capture one object in our learning framework. This is because when motion compensation is applied to a dynamic query, the corresponding set of points move the same way, and we perform motion compensation for multiple temporal steps. In this way, the set of points corresponding to the same query learn to match with the same dynamic object in the scenes, as they always move together. We can provide a visualization of this in the revised paper (unfortunately this year's NeurIPS does not support uploading a PDF for rebuttal).

$**If only occupancy labels are used for supervision**$:

We conducted experiment with ODG-T and trained for 24 epochs on nuScenes where only occupancy labels are used in training. The results are listed in Table 1 below. We can see when only using occupancy labels as supervision, ODG-T experienced a minor drop in performance but is still performing respectably well, validating the robustness of our designs.

Table 1. Ablation study on different supervision signals with ODG-T on nuScenes

$**Highlighting error in Table 1**$:

We have corrected the errors and the changes will be reflected in the final version.

We thank reviewer JcRJ for the constructive feedbacks. We address the reviewer's questions and concerns below.

$**Coarse-to-fine mechanism**$:

We present our coarse-to-fine refinement scheme in Algorithm 1 below. For notation simplicity, we denote motion compensation, feature sampling, cross and self attention, along with dual query-attention presented in Figure. 1 in the main paper as a single layer, rest of the notations are the same as defined in the main paper. To further enhance clarity, as described in the main paper, since coarse-to-fine refinement doesn't apply to bounding box attributes (i.e. $K_0=1$ throughout all the layers $\hat{\mathcal{T}}_l$), we have omitted them in Algorithm 1 presented below.

We note that due to OpenReview's limited TeX support, we had to declare transient notations such as $p,\text{inputs}^s_p,\text{ and }\text{inputs}_p^d$ to facilitate algorithmic flow.

Algorithm 1: Coarse-to-fine refinement in ODG

$\text{Input}: \text{image features } **F**; \text{static Guassian queries } **G**_0^s, **Q**_0^s; \text{dynamic Guassian queries } **G**_0^d, **Q**_0^d$

$\text{Output}: \text{refined static Guassian queries } **G**_L^s, **Q**_L^s, \text{ class predictions }\mathbf{C}^s_L; \text{ refined dynamic Guassian queries } **G**_L^d, **Q**_L^d, \text{ class predictions }\mathbf{C}^d_L$

$\text{Function } \text{ODGRefine}(\mathbf{F}; **G**_0^s, **Q**_0^s; **G**_0^d, **Q**_0^d):$

$\quad\mathbf{G}^s_{:\mu,0}\in\mathbb{R}^{S\times K_0\times 3} \leftarrow \mathcal{U}(0, 1), K_0=1$

$\quad\mathbf{G}^d_{:\mu,0}\in\mathbb{R}^{D\times K_0\times 3} \leftarrow \mathcal{U}(0, 1), K_0=1$

$\quad$// initialize Gaussian means with uniform distribution; rest of the Gaussian parameters left uninitialized

$\quad\text{for }l\leftarrow 1 \text{ to } L \text{ do }$

$\quad\quad p=l-1$

$\quad\quad\text{inputs}^s_p=[\mathbf{G}^s_{\:\mu,p}, \mathbf{Q}^s_{p}, \mathbf{F}]$

$\quad\quad\mathbf{G}_{\:\mu, l}^s,\mathbf{Q}^s_l,\mathbf{C}^s_l=\hat{\mathcal{T}}_l(\text{inputs}^s_p)$

$\quad\quad\mathbf{G}^s_{\mu:,l}=\Phi(\mathbf{G}^s_{:\mu,\ell}, \mathbf{Q}^s_l)$ // rest of Gaussian params predicted by MLP $\Phi$

$\quad\quad\text{// }\mathbf{G}^s_{\:\mu, l}\in\mathbb{R}^{S\times K_l\times 3}, \mathbf{G}^s_{\mu\:, l}\in\mathbb{R}^{S\times K_l\times 10}, \mathbf{C}^s_l\in\mathbb{R}^{S\times K_l\times C}, K_p < K_l$, # coarse-to-fine

$\quad\quad\text{inputs}^d_p=[\mathbf{G}^s_{\:\mu,p}, \mathbf{Q}^d_{p}, \mathbf{F}]$

$\quad\quad\mathbf{G}_{\:\mu, l}^d, \mathbf{Q}_l^d,\mathbf{C}^d_l=\hat{\mathcal{T}}_l(\text{inputs}^d_p)$

$\quad\quad\mathbf{G}^d_{\mu:,l}=\Phi(\mathbf{G}^d_{:\mu,\ell}, \mathbf{Q}^d_\ell)$ // rest of Gaussian params predicted by MLP $\Phi$

$\quad\quad\text{// }\mathbf{G}^d_{\:\mu, l}\in\mathbb{R}^{D\times K_l\times 3}, \mathbf{G}^d_{\mu\:, l}\in\mathbb{R}^{D\times K_l\times 10}, \mathbf{C}^d_l\in\mathbb{R}^{D\times K_l\times C}, K_p < K_l$, # coarse-to-fine

$\quad\text{return }[\mathbf{G}^s_l, \mathbf{Q}^s_l, \mathbf{C}^s_l], [\mathbf{G}^d_l, \mathbf{Q}^d_l, \mathbf{C}^d_l]$

$**Efficiency analysis for ODG**$:

We profiled both ODG-L and ODG-T using DeepSpeed. The results are summarized Table 1 and 2 below.

Table. 1 ODG-L profiling.

Table. 2 ODG-T profiling.

We can see that in both ODG-L and ODG-T, the transformer part takes up most of the inference cost. We provide further profiling results on ODG-L transformer in Table 3 below. One can see that query self-attention takes up almost half the runtime. One straight-away optimization can be replacing self-attention with efficient attention schemes such as linear attention [1] or state space models (SSMs) [2], which would improve inference runtime. We will look into this as part of future work.

Table. 3 ODG-L transformer profiling.

[1] Wang, Sinong, et al. "Linformer: Self-attention with linear complexity." arXiv preprint arXiv:2006.04768 (2020).

[2] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv preprint arXiv:2312.00752 (2023).

$**Ablation studies for 100 epochs**$:

We have now conducted all ablation experiments with ODG-T and trained for 100 epochs on nuScenes. The results are listed in Table 4 below. We can see under longer training time our ODG still consistently improves, which validates the efficacy of our designs. The conclusions on the effectiveness of our proposed components do not change when training for 24 or 100 epochs.

Table 4. Ablation studies of different components of ODG-T. Trained for 100 epochs on nuScenes.

We thank reviewer cwc4 for the constructive feedbacks. We address the reviewer's questions and comments below.

$**Design of coarse-to-fine hierarchical structure**$:

We present our coarse-to-fine refinement scheme in Algorithm 1 below. For notation simplicity, we denote motion compensation, feature sampling, cross and self attention, along with dual query-attention presented in Figure. 1 in the main paper as a single layer $\hat{\mathcal{T}}_l$, rest of the notations are the same as defined in the main paper. To further enhance clarity, as described in the main paper, since coarse-to-fine refinement doesn't apply to bounding box attributes (i.e. $K_0=1$ throughout all the layers $\hat{\mathcal{T}}_l$), we have omitted them in Algorithm 1 presented below.

We note that due to OpenReview's limited TeX support, we had to declare transient notations such as $p, \text{inputs}^s_{p}, \text{ and }\text{inputs}^d_{p}$ to facilitate algorithmic flow.

Algorithm 1: Coarse-to-fine refinement in ODG

$\text{Input}: \text{image features } **F**; \text{static Guassian queries } **G**_0^s, **Q**_0^s; \text{dynamic Guassian queries } **G**_0^d, **Q**_0^d$

$\text{Output}: \text{refined static Guassian queries } **G**_L^s, **Q**_L^s, \text{ class predictions }\mathbf{C}^s_L; \text{ refined dynamic Guassian queries } **G**_L^d, **Q**_L^d, \text{ class predictions }\mathbf{C}^d_L$

$\text{Function } \text{ODGRefine}(\mathbf{F}; **G**_0^s, **Q**_0^s; **G**_0^d, **Q**_0^d):$

$\quad\mathbf{G}^s_{:\mu,0}\in\mathbb{R}^{S\times K_0\times 3} \leftarrow \mathcal{U}(0, 1), K_0=1$

$\quad\mathbf{G}^d_{:\mu,0}\in\mathbb{R}^{D\times K_0\times 3} \leftarrow \mathcal{U}(0, 1), K_0=1$

$\quad$// initialize Gaussian means with uniform distribution; rest of the Gaussian parameters left uninitialized

$\quad\text{for }l\leftarrow 1 \text{ to } L \text{ do }$

$\quad\quad p=l-1$

$\quad\quad\text{inputs}^s_p=[\mathbf{G}^s_{\:\mu,p}, \mathbf{Q}^s_{p}, \mathbf{F}]$

$\quad\quad\mathbf{G}_{\:\mu, l}^s,\mathbf{Q}^s_l,\mathbf{C}^s_l=\hat{\mathcal{T}}_l(\text{inputs}^s_p)$

$\quad\quad\mathbf{G}^s_{\mu:,l}=\Phi(\mathbf{G}^s_{:\mu,\ell}, \mathbf{Q}^s_l)$ // rest of Gaussian params predicted by MLP $\Phi$

$\quad\quad\text{// }\mathbf{G}^s_{\:\mu, l}\in\mathbb{R}^{S\times K_l\times 3}, \mathbf{G}^s_{\mu\:, l}\in\mathbb{R}^{S\times K_l\times 10}, \mathbf{C}^s_l\in\mathbb{R}^{S\times K_l\times C}, K_p < K_l$, # coarse-to-fine

$\quad\quad\text{inputs}^d_p=[\mathbf{G}^s_{\:\mu,p}, \mathbf{Q}^d_{p}, \mathbf{F}]$

$\quad\quad\mathbf{G}_{\:\mu, l}^d, \mathbf{Q}_l^d,\mathbf{C}^d_l=\hat{\mathcal{T}}_l(\text{inputs}^d_p)$

$\quad\quad\mathbf{G}^d_{\mu:,l}=\Phi(\mathbf{G}^d_{:\mu,\ell}, \mathbf{Q}^d_\ell)$ // rest of Gaussian params predicted by MLP $\Phi$

$\quad\quad\text{// }\mathbf{G}^d_{\:\mu, l}\in\mathbb{R}^{D\times K_l\times 3}, \mathbf{G}^d_{\mu\:, l}\in\mathbb{R}^{D\times K_l\times 10}, \mathbf{C}^d_l\in\mathbb{R}^{D\times K_l\times C}, K_p < K_l$, # coarse-to-fine

$\quad\text{return }[\mathbf{G}^s_l, \mathbf{Q}^s_l, \mathbf{C}^s_l], [\mathbf{G}^d_l, \mathbf{Q}^d_l, \mathbf{C}^d_l]$

$**Visual comparison with other methods**$:

We have prepared such visual comparisons and will include them in the final version like in Fig. 2 and Fig. 3 to further demonstrate the superiority of our proposed ODG. Unfortunately, this year's NeurIPS rebuttal format does not support uploading a PDF for the rebuttal and as such, we are not able to share visualization results during the rebuttal stage.