OccProphet: Pushing the Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with an Observer-Forecaster-Refiner Framework

Thanks for your positive recognition (e.g., critical for autonomous driving, more feasible for deployment, and substantial improvement in perception). We appreciate the opportunity to address your valuable feedback as follows.

Response to Weaknesses

W1. The paper does not give detailed explanations for the design of the three proposed modules.

Thanks for pointing this out. We further provide more explanations for the design of the three proposed modules in the following perspectives. The manuscript has been updated accordingly.

• Overview:

  "The Observer module efficiently and effectively aggregates spatio-temporal information within multi-frame observations (i.e., multi-frame 3D voxel features). The Observer’s output then undergoes a Forecaster, which adaptively predicts future states, ensuring flexibility across diverse traffic conditions. The Refiner module further enhances the quality of these predictions by enabling cross-frame interactions." (In Para. 1 of Sec. 3.1)

• Observer:

  Efficient 4D Aggregation (E4A): "Directly aggregating the original 4D feature $F_{motion}$ will incur a high computational cost. For efficiency, we design the Efficient 4D Aggregation (E4A) module to first produce compact features through downsampling, and then exploit spatio-temporal interactions on the compact features to achieve aggregation, followed by the upsampling process to compensate for the information loss." (In Para. 1 of Sec. 3.2.1)

  Tripling-Attention Fusion (TAF): "The tripling operation is designed to understand the 3D space from three complementary and compact perspectives, which can retain 3D scene information with fewer computational costs. Specifically, a tripling operation decomposes a 3D feature into three distinct branches: scene, height, and BEV." (In Para. 2 of Sec. 3.2.2)

• Forecaster:

  "We propose forecasting occupancy with flexibility to adapt to various traffic scenarios featuring diverse spatio-temporal complexities. To achieve this, we design a novel Forecaster module that predicts future states based on the overall environmental condition. The Forecaster comprises a Condition Generator and a Conditional Forecaster." (In Para. 2 of Sec. 3.3)

• Refiner:

  "Since the Forecaster module predicts $F_{\text{future}}$ using linear projection, it inevitably lacks cross-frame interactions. The Refiner is designed to enhance the forecasted results via further interactions between future frames, as well as incorporating historical frames as supplementary information." (In Para. 1 of Sec. 3.4)

Besides, we added some technical details, highlighted in blue in Sec. 3 of the updated paper.

Response to Questions

Q1. What is the actual frames-per-second of the model? And is it good enough for deployment?

After engineering optimization, the inference speed of Cam4DOcc is accelerated from 1.7 FPS to 8.0 FPS. In contrast, OccProphet achieves a speedup from 4.5 FPS to 21.2 FPS, which is close to deployment requirements.

Response to Questions

Q1. The current baselines lack sufficient comparison methods and evaluation metrics.

The previous benchmark in Cam4DOcc covers four camera-only occupancy forecasting tasks (see Sec. 4.2 and Appendix A.1), three datasets (nuScenes, Lyft-Level5, nuScenes-Occupancy), three accuracy metrics ($\text{IoU} _ {c}$, $\text{IoU} _ {f}$, and $\tilde{\text{IoU}} _ {f}$). In our original paper, we have added four efficiency metrics (N$_{\text{p}}$ (M), Memory (G), FLOPs (G), and FPS) for comparisons.

In the updated paper, we have added a quantitative comparison of forecasting over varying time horizons (see Tab. 8) and a qualitative comparison on dense scenarios (see Fig. 13). For the task of occupancy forecasting over varying time horizons, we use $\text{IoU}_{f}$ at different timestamps as the evaluation metric. For example, on the nuScenes and nuScenes-Occupancy datasets, we evaluate forecasting performance at 0.5, 1.0, 1.5, and 2.0 seconds; on the Lyft-Level5 dataset, we evaluate forecasting at 0.2, 0.4, 0.6, and 0.8 seconds.

Furthermore, we compared a new baseline method, BEVDet4D, on the nuScenes dataset (see the table above). There are five comparison methods totally.

In summary, by expanding the baseline methods, comparison tasks, and evaluation metrics, we provide a more comprehensive demonstration of the superiority of our proposed approach.

[1] Junjie Huang, Guan Huang. "Bevdet4d: Exploit temporal cues in multi-camera 3d object detection." arXiv preprint arXiv:2203.17054, 2022.

Thank you for your kind approval (e.g., a crucial improvement for deploying and good writing). Below, we provide detailed responses to your insightful comments.

Response to Weaknesses

W1. The paper should more clearly highlight the distinctions from the traditional encoder-decoder architecture to better emphasize the contribution of Observer, Forecaster, and Refiner.

Thanks for the constructive comment. Based on the suggestion, we have added the distinction statement in the updated paper. "The traditional encoder-decoder architecture comprises an encoder for representation extraction and a decoder for occupancy prediction, as adopted by OccNet [1] and Cam4DOcc. However, the traditional architecture either loses 3D geometry details—e.g., OccNet adopts a BEV-based encoder, or introduces high computational cost—e.g., Cam4DOcc utilizes vanilla 3D convolutional networks as the encoder and decoder.

In OccProphet, the Observer works similarly to an encoder, while the combination of Forecaster and Refiner works similarly to a decoder. Unlike the traditional encoder-decoder architecture, OccProphet pushes the efficiency frontier of 4D occupancy forecasting. To achieve this, the Observer and Refiner enable spatio-temporal interaction using the Efficient 4D Aggregation module, and the Forecaster adaptively predicts future states using a lightweight condition mechanism. Overall, our Observer-Forecaster-Refiner framework emphasizes 4D spatio-temporal interaction and conditional forecasting, meanwhile maintaining efficiency." (In Appendix A.3)

[1] Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu et al. "Scene as occupancy." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8406-8415. 2023.

W2. The tripling-attention and reduced-resolution feature aggregation may sacrifice some granularity or detail in the forecasts, possibly affecting the precision of the model in dense scenarios.

This issue does exist. The visualization of occupancy forecasting in dense scenarios is provided in Fig. 13, with the corresponding analysis in Appendix A.5.3 of the updated paper. It is obvious that both Cam4DOcc and OccProphet have errors in fine-grained forecasting. In comparison, OccProphet's results are closer to the ground truth, while our goal is to achieve high efficiency and reduce performance drop.

We consider that, although our aggregation operates on reduced-resolution features, it is multi-scale and accompanied by feature upsampling. Besides, the tripling-attention splits input features from the scene, height, and BEV perspectives. Among them, the height perspective captures vertical details of the environment, while the BEV perspective effectively obtains a large receptive field through the attention mechanism. Together, these aspects enhance the ability of the tripling-attention to capture scene details.

In Appendix A.5.3, we have claimed that fine-grained occupancy forecasting is a valuable direction for future research.

W3. The paper lacks details on how well the model performs over varying forecasting time horizons.

Thank you for the professional advice. We evaluated the performance of forecasting occupancy over varying time horizons, as shown in the above table. We can see that (1) OccProphet consistently outperforms other approaches across all time horizons on three datasets. (2) The longer the forecasting period, the lower the accuracy of all methods, indicating that forecasting becomes increasingly difficult.

We have included this experiment in Appendix A.4 of the updated paper.

Thanks for your in-depth review, endorsement of our work (e.g., extensive experiments and an efficient and effective algorithm), and constructive advice. Below, we provide detailed responses to each of your points.

Response to Weaknesses

W1. The reviewer is concerned about the effectiveness part of OCCPROPHET. The temporal-spatial interaction should also be utilized in OCFNet (Cam4DOcc). The reviewer is a bit confused about the better performance OCCPROPHET shows than that of OCFNet (Cam4DOcc).

The occupancy forecasting performance of OCFNet (Cam4DOcc), with and without temporal-spatial interaction, is presented in the table above. The integration of temporal-spatial interaction improves Cam4DOcc, yielding higher scores in $\text{IoU} _ {c}$ (accuracy of the current frame) and $\tilde{\text{IoU}}_{f}$ (average accuracy of all future frames), demonstrating the effectiveness of this interaction. In comparison, OccProphet outperforms Cam4DOcc, even with temporal-spatial interaction.

We further conduct an experiment where both OCFNet (Cam4DOcc) and OccProphet are trained for longer until convergence. The experimental results and analysis are provided in the response to Question 1 below.

Response to Questions

Q1. Could the authors show experiment results on training for longer with both OCFNet (Cam4DOcc) and OCCPROPHET to see whether the performance improvement stems from early convergence.

We train OCFNet (Cam4DOcc) and OccProphet for longer until they both converge (36 epochs).

The quantitative comparison of occupancy forecasting performance is presented in the table below. We observe that both OCFNet (Cam4DOcc) and OccProphet benefit from longer training. However, OccProphet still outperforms Cam4DOcc, indicating that the performance improvement does not stem from early convergence.

Q2. If not, is there any analysis or explanation about the performance improvement?

The performance improvement stems from the collaborative functions of the three modules within our proposed Observer-Forecaster-Refiner framework.

• Temporal-spatial Interaction Mechanism: This interaction is embedded in the Observer and Refiner modules. As analyzed in Weakness 1, this mechanism has been proven to enhance forecasting accuracy.

• Condition Mechanism: The Forecaster module predicts future states based on the overall environmental condition. This mechanism enables forecasting to adapt to various traffic scenarios.

• Independent Module Contributions: As shown in Tab. 5 and Fig. 8, 9, and 10 of our paper, each module—Observer, Forecaster, and Refiner—is important to improve occupancy forecasting accuracy.

In summary, the performance improvement is attributed to our proposed Observer-Forecaster-Refiner framework. This framework is not only lightweight but also effective in enhancing performance.

Q4. Are the 6-DoF camera parameters in the Observer used to align historical 3D features with the current frame? In lines 188-191, why does $F$ change to $F_{motion}$, resulting in an extra matrix dimension of 6×X×Y×Z?

Yes, the 6-DoF ego-vehicle poses of each frame are used to align historical 3D features into the current frame.

Furthermore, to acquire $F_{motion}$, we perform the following two steps:

1. The 6-DoF ego-vehicle poses from $T$ frames are organized into a pose tensor with the shape $T \times 6 \times X \times Y \times Z$, by expanding each 6-DoF pose into 3D and then concatenating them across frames.
2. We combine the pose tensor and the 4D feature $F$ (with the shape $T \times C \times X \times Y \times Z$) along channel axis $C$ to produce an ego-vehicle motion-aware feature $F_{motion}$ with the shape $T \times (C+6) \times X \times Y \times Z$.

Q5. In lines 206-207, how does C+6 become C?

We apply a $3\times3\times3$ 3D convolution to fuse ego-vehicle poses and environmental information within a local neighborhood, and meanwhile reducing the number of channels from $C+6$ to $C$.

Q6-(1). What is the difference between the E4A module shown in Figure 4 and the UNet structure?

There are two differences. From the data format perspective, conventional UNet structures can operate on spatial features like 3D volumes, whereas the E4A module can process spatio-temporal 4D inputs. From the functional perspective, E4A focuses on 4D interaction, by utilizing our Tripling-Attention and the spatio-temporal aggregation.

Q6-(2). Why does the FLOPs increase significantly while the number of parameters in E4A decreases? Is this due to the upsampling process?

Yes, this is due to upsampling. The upsampling process generates features with larger sizes, bringing more computations when subsequently operating on these enlarged features. The parameter decrease in E4A benefits from our lightweight design of the Tripling-Attention Fusion (TAF) module.

Q7. Would cross-attention between different scales yield better results?

Thanks for the suggestion. We performed cross-attention between different scales on our OccProphet. However, the performance did not improve, as shown in the table above. A possible reason is that the semantic gap between different scales brings difficulty for learning. How to better design cross-attention for different scales is a valuable research question worth exploring in the future.

Q8. In the Forecaster, is the condition for prediction learned from past voxels?

In the Forecaster, the condition is generated from past voxels (relative to future voxels), which consists of previous observations (i.e., frames $1$ to $T-1$) and the current observation (i.e., frame $T$).

Q9. What does the colon symbol in line 313 mean?

Sorry for a typo in line 313. We have corrected it with the proper subscript $(T+1:T+T^{\prime},:,:,:,:)$.

The colon symbol in the subscript means collecting features based on the corresponding indices. Specifically, E4A$([O_{obs},F_{future}])$ produces an output in $\mathbb{R}^{(T+T^{\prime}) \times C \times X \times Y \times Z}$. In the subscript, $(T+1:T+T^{\prime})$ means collecting features from the $T+1$ th to $T+T^{\prime}$ th frames. The subsequent colons $(:,:,:,:)$ mean that, for one frame, all features in the voxelized space of that frame are collected. As a result, we obtain a representation $V \in \mathbb{R}^{T^{\prime} \times C \times X \times Y \times Z }$, which is used for subsequent forecasting of occupancy and flow.

Q10. It would be helpful to follow the output of the 4D BEV method, like [1], with a 2D to 3D detection head for comparison. [1] Bevdet4d: Exploit temporal cues in multi-camera 3D object detection

Thanks for the comment. We have adapted BEVDet4D to forecast 4D occupancy for comparison. The experimental results on the nuScenes dataset are reported in the table above. While BEVDet4D achieves considerable forecasting performance, voxel-based approaches such as OCFNet (Cam4DOcc) and OccProphet (Ours) outperform the BEV-based BEVDet4D.

Thank you for your encouraging appreciation (e.g., well-written manuscript, clear Fig. 2, and thorough experiments). Below, we address each point in detail according to your insightful comments and constructive advice.

Response to Weaknesses

W1. Some symbols and design details of the model are unclear.

For the issues regarding unclear symbols (Question 4, 5, and 9) and unclear design (Question 6 and 8), we have provided detailed explanations. Additionally, we add more explanations for the model design in the paper. All revisions to the model design and technical details are highlighted in blue in Sec. 3 of the updated paper.

W2. The analysis of failure scenarios is lacking, but this is not a major concern.

Thanks for your advice. We have added some failure scenarios (shown in Fig. 11, 12, and 13) along with corresponding discussions in Appendix A.5.

Response to Questions

Q1-(1). What part of the Cam4DOcc model contributes to its high computational cost?

The computational cost of each component and the scores of IoU are listed below. (^{\\#}: Tested by MMDetection3D $1$ . $^{*}$: Reported from the original paper.)

The high computational cost of Cam4DOcc primarily stems from its image encoder and 2D-to-3D lifting module. To alleviate this computational burden, OccProphet reduces the input image size by half. However, this operation will decrease occupancy forecasting accuracy.

To compensate for the loss in forecasting accuracy, meanwhile maintaining efficiency, OccProphet introduces lightweight components: Observers, Forecasters, and Refiners. These components are both effective and efficient: (i) The FLOPs of the remaining modules in Cam4DOcc is 1708G. In contrast, with the same input feature size, the remaining part of OccProphet is 828G FLOPs. (ii) OccProphet achieves an $\tilde{\text{IoU}}_{f}$ of 29.15%, surpassing Cam4DOcc's 27.98%, thereby indicating higher occupancy forecasting accuracy.

$1$ : MMDetection3D Contributors. Mmdetection3d: Openmmlab next-generation platform for general 3d object detection. https://github.com/open-mmlab/mmdetection3d, 2020. 1, 4

Q1-(2). How much memory does Cam4DOcc use during training and testing?

Cam4DOcc uses 57G of memory during training (as reported in the original paper), and approximately 24G of memory during testing. In comparison, OccProphet requires 24G of memory during training, and around 8G during testing.

Q1-(3). Approximately how many GPU days are needed for full training?

The full training time of Cam4Docc is approximately 56 GPU Days on NVIDIA A100. In comparison, OccProphet takes around 19 GPU Days for training on NVIDIA A100.

Q2. The proposed Observer-Forecaster-Refiner pipeline seems to learn the dynamics of objects in the latent space without strict physical constraints. If the real world presents scenarios that are rare or unseen in the dataset, could 4D predictions fail? Are there any examples of such failures?

Thank you for the comment. We use occupancy flow labels between adjacent frames as physical constraints in training. These labels ensure that dynamic objects adhere to real-world motion principles, i.e., moving smoothly and continuously in 3D space and time, and avoiding unrealistic changes in velocity or direction.

In the updated paper, we have added several failure cases, including cross-domain forecasting for unseen scenarios, occupancy forecasting for occluded scenarios, and fine-grained forecasting for dense scenarios. The corresponding analysis is provided in Appendix A.5.

Q3. Have the labels in the 4D dataset been completed to account for occlusions? If they have, can the authors' method handle situations where any frame from the historical RGB images loses an object?

We visualize the ground truth labels and forecasted occupancy results in Fig. 12 of Appendix A.5.2. It can be observed that:

1. The occupancy labels of occluded objects remain complete over time, even when an object is nearly missing in some frames;
2. Our method can handle the occluded scenarios to a certain degree, where a historical frame loses an object. Nonetheless, the performance can still be improved in future research.