Context and Geometry Aware Voxel Transformer for Semantic Scene Completion

Q1. Parameters and performance comparison between CGFormer and Symphonize.

To compare the performance of CGFormer with Symphonize with a comparable number of parameters, we revisited the design of our CGFormer and replaced the EfficientNetB7 with ResNet50 and the Swin blocks in the TPV branch with more lightweight residual blocks. The results on the semantickitti validation set are summarized in the table below. As shown in this table, our CGFormer is robust across different backbone networks and still outperforms Symphonize with a comparable number of parameters. This highlights CGFormer's potential and robustness.

Q2. Text error in Figure 1.

Thanks for pointing out this mistake. We will correct it in the revised manuscript.

Q3. Failure cases.

Figure 2 in the uploaded PDF shows two examples of failure cases, where it is difficult to distinguish between adjacent cars. As seen in the RGB images, these objects are located in distant regions, making them challenging to differentiate. In the future, we plan to explore ways to improve the performance of our method in these far regions.

Dear Reviewer Lm1y,

Thank you for raising your score. Your suggestions are valuable to improve the quality of our paper. We will update our manuscript in the revision.

Authors of Paper ID 3532

W1. Ablation on the number of the cross-attention and self-attention layers.

We present the results of different configurations of cross-attention and self-attention layers on the semantickitti validation set in the table below. As shown in this table, the performance improves gradually with the increase of the number of the attention layers. However, once the number exceeds a certain threshold, the performance tends to stabilize. In alignment with previous methods, we set the number of cross-attention layers to $3$ and the number of self-attention layers to $2$ in the manuscript. This table also includes the results of the model without the CAQG module. Notably, using just one self-attention and one cross-attention layer with the CAQG module yields better performance than using two self-attention and three cross-attention layers without the CAQG module. This demonstrates the effectiveness of our proposed context and geometry-aware voxel transformer.

W2. Performance using only monocular image.

Following VoxFormer and Symphonize, we replace the depth estimation network with AdaBins and present the results on the semantickitti validation set in the table below. To better evaluate the performance of our CGFormer, we also include the results of VoxFormer, Symphonize, and OccFormer. Compared to the stereo-based methods when using only a monocular image (VoxFormer, Symphonize), CGFormer achieves superior performance in terms of both IoU and mIoU. Furthermore, our method also surpasses OccFormer, the state-of-the-art monocular method.

W3. More visualizations.

Figure R1 in the uploaded PDF displays two visualization examples for objects with finer details. As shown in the RGB image, the sampling points of the context-dependent queries are typically situated within the region of interest. This allows CGFormer to capture much more clear details compared to other methods, highlighting the effectiveness of our proposed module. We will also provide the image view in the revised manuscript as done in this figure.

W4. The manuscript is limited in scholarship and can be improved by incorporating semantic scene completion references.

Thanks for your suggestion. We will incorporate these semantic scene completion references in the revised manuscript.

Dear Reviewer wDiN,

Thank you again for your review. We are glad that our response has addressed the questions you raised.

Authors of Paper ID 3532

W1 (a). Performance gain of the context-aware queries.

Compared to methods that do not use temporal inputs or rely on much larger image backbone networks, CGFormer achieves the highest performance in 12 out of 19 categories on both the test and validation sets, as shown in Tables R2 and R3 of the uploaded PDF, encompassing both large-area and fine-detail objects, while the other methods excel in at most 3 to 4 categories. Although CGFormer shows limitations in several categories such as person and truck, it excels in most of the categories, including car, traffic sign, trunk, and pole. Indeed, CGFormer attains the top performance on the bicycle under the same condition.

To further validate the effectiveness of the proposed context-aware queries, we remove the CAQG module while keeping all other components unchanged. We then compare the performance of the two models across various categories. The results are presented in Table R3 of the uploaded PDF, with categories where performance improves after integrating the CAQG module highlighted in red. The CAQG module enhances performance in most categories (11 out of 19), including both large-area objects (sidewalk, terrain) and small objects (trucks, bicycles, person), while having minor effects on others. The above results demonstrate the efficacy of the proposed module.

W1 (b). More explanation of Figure 3 in the manuscript.

Thanks for your suggestion to help illustate our motivation. We will replace Figure 3 with more clear examples in the revised manuscript, as shown in the Figure R1 in the uploaded PDF. Here's a more detailed explanation for this figure. The yellow point represents the projected location of the visible voxel queries of the deformable cross-attention from 3D space onto the 2D image plane using the camera's intrinsic and extrinsic matrices. This projection aggregates information from the image features. For a voxel projected onto the image plane, the query should tend to aggregate information relevant to the semantics of the position where it lands. As shown in the left and middle columns, the yellow reference points fall onto the car and bicycle, and the sampling points of the context-aware query are mainly distributed within the regions of these two objects. In the right column, the query point projects onto the building, and the context-aware query points are primarily located on the building. In contrast, the context-independent query points are scattered in the region of the car.

W1 (c). More explanation for "geometry-aware".

The modules derived from depth information are referred to as "geometry-aware" in the title. We provide a more detailed discussion as follows.

(1) As mentioned in the manuscript (line 49-51), visible queries are projected onto the image plane to aggregate information by deformable cross-attention. However, when projecting the 3D points onto the image plane, many points may end with the same 2D position with similar sampling points on the 2D feature map, causing a crucial depth ambiguity problem. To address this issue, we extend deformable cross-attention from 2D to 3D pixel space, which allows to differentiate points with similar image coordinates by their depth coordinates, as illustrated in Figure 1 of the manuscript.

(2) Additionally, we introduce depth refinement to improve the accuracy of the estimated depth probability.

Model (a) in Table 3 provides ablation results for the 3D deformable cross-attention, and Table 5 includes ablation results for depth refinement.

W2 & Q1. More explanation for the CAQG module.

Thanks for pointing out this point, here is a revised version (line141-line144) of the manuscript.

To elaborate, the context feature $\mathbf{C}\in\mathbb{R}^{H\times W\times C}$ and depth probability $\mathbf{D}\in\mathbb{R}^{H\times W\times D}$ are first derived from the 2D image feature $\mathbf{F}^{2D}$. Taking $\mathbf{C}$ and $\mathbf{D}$ as inputs, the query generator $f$ maps them from 2D image space to 3D space to generate context-aware voxel queries $\mathbf{V_{Q}}\in\mathbb{R}^{X\times Y\times Z}$, where $(X,Y,Z)$ denotes the spatial resolution of the 3D volume. The query generator can be any explicit view transformation approach (e.g., voxel pooling, FLoSP, CaDDN). Table 4 provides ablation experiments for different methods.

Q2. More analyses for the TPV branch.

Thanks for your constructive question. We find additional insights regarding the TPV branch. We reanalyze the ablation experiments (models d, e, f, g in Table 4 of the manuscript) and discover that the performances were quite similar. To determine whether these performance gains are statistically significant across multiple training seeds, we conduct further experiments. As shown in Table R5 of the uploaded PDF, combining the outputs from both branches does enhance performance, but with some variability. We speculate that the TPV branch primarily focuses on global information, making it challenging to capture fine-grained voxel details. In contrast, the local branch enhances the fine-grained structural details of the 3D voxels. Simple addition treats all features as equally important, which may introduce side effects. Instead, weighting the more important features from each branch can enhance overall performance by leveraging their distinct strengths.

Q3. Ablation experiments for TPV branch.

Thanks for your suggestion to complete our experiments. Taking the model (b) of the table 3 in the manuscript as baseline, we present the ablation results for the TPV branch. As shown in the Table R4 in the uploaded PDF, enhancing the features with TPV branch boost the performance in terms of IoU, while minor improvement in terms of the mIoU, following the assumption that TPV focuses more on the global information.

Dear Reviewer ryRH,

Thanks for your comments, which are valuable for improving the overall quality of this manuscript.

To address your major concerns, we compared the performance of CGFormer with other methods using consistent inputs and similar image backbones. We conducted ablation experiments to evaluate the performance gains of context-aware queries across different classes. The uploaded PDF includes a more detailed analysis of the figures and additional illustrative examples. We have also provided a thorough explanation of the origin of the term 'geometry aware' as mentioned in the title. Additionally, we have included ablation studies and analyses for the TPV branch.

Could we kindly ask if our responses have addressed for your concerns and if you have any new questions? Thanks for your time and effort.

Authors of Paper ID 3532

W1. Parameter and performance comparison with Symphonize.

Thanks for your valuable suggestion. We revisit the architecture of CGFormer and discover some interesting new results. To compare its performance with Symphonize with a comparable number of parameters, we analyze the components of CGFormer, finding that replacing EfficientNetB7, used as the image backbone, and the Swin blocks, used in the TPV branch backbone, with more lightweight ResNet50 and residual blocks, respectively, can significantly reduce the number of parameters of our network. The results on the semantickitti validation set are presented in the table above. Compared to the original architecture, CGFormer maintains stable performance regardless of the backbone networks used for the image encoder and TPV branch encoder, underscoring its effectiveness, robustness, and potential.

Compared to Symphonize, lightweight CGFormer achieves an IoU of 45.86 and mIoU of 16.85, significantly surpassing Symphonize's IoU of 41.92 and mIoU of 14.89 on the SemanticKITTI validation set with a comparable number of parameters.

We retrain this lightweight model on the KITTI-360 dataset, with the detailed results of each class in Table R1 of the uploaded PDF. The lightweight version of CGFormer achieves an IoU of 47.78 and mIoU of 20.03, demonstrating a substantial improvement of 1.45 mIoU and 3.5 IoU over Symphonize. For specific categories, the original CGFormer architecture outperforms Symphonize in 9 out of 18 classes, and with the lighter backbones, it surpasses Symphonize in 10 out of 18 classes. These results further highlight the superiority of our approach.

W2. More explanation of context and geometry aware voxel transformer.

We apologize for any misunderstanding. Context and geometry aware voxel transformer is not the redesign of Symphonize. We will provide a more detailed explanation here. Existing coarse-to-fine (sparse-to-dense) methods, such as VoxFormer and MonoOcc, generally follow a pipeline that first aggregates 3D information for visible voxels using depth-based queries. These queries are defined as a set of learnable parameters, which are the same for all the input images. Subsequently, these methods complete the 3D information for non-visible regions using the reconstructed visible areas as starting points. The aggregation of information for visible voxels is accomplished through deformable cross-attention, while the completion of information in non-visible regions is handled by deformable self-attention, similar to the MAE [1].

For the proposed context and geometry-aware voxel transformer, we take into account the context of different images and introduce context-dependent queries. Instead of solely predefining a set of learnable parameters that primarily capture the overall distribution of the dataset, the context-dependent queries are related to the image content, allowing them to aggregate information from points within contextually relevant regions. Additionally, we extend deformable cross-attention from the 2D to 3D pixel space, enabling the differentiation of points with similar image coordinates based on their depth coordinates.

In contrast, Symphonize employs context-independent voxel queries for all the input images. It first condenses the image into a set of tokens with higher-level instance semantics, then the voxel queries serve as the query of the deformable cross attention while the set of tokens serve as the key and value. The original manuscript of Symphonize referred to this set of tokens as "queries", which may have led to some misunderstanding.

W3. Details of context net.

Sorry for the oversight of this module. The image encoder consists of a backbone network for extracting multi-scale features and a feature pyramid network to fuse them. We employ SECONDFPN [2] network here. It fuses the multiscale image features and output a feature map $\mathbf{F}^{2D}\in\mathbb{R}^{H\times W\times C}$, with $C$ set to $512$. The context net consists of several convolution blocks that take $\mathbf{F}^{2D}$ as input and reduce the number of channels. The generated context feature $\mathbf{C}$ has a shape of $H\times W\times 128$. We will include these details in the revised manuscript.

W4. The table font is too small to read.

Thanks for your suggestion. We will fix it in the revised manuscript. Examples of tables with larger font are provided in the uploaded PDF.

[1] Masked autoencoders are scalable vision learners, CVPR 2022

[2] SECOND: Sparsely Embedded Convolutional Detection, Sensors 2018

Dear Reviewer jg8M,

Thanks for your comments, which are valuable for improving the overall quality of this manuscript.

We've provided additional architecture analysis on our proposed method and compared it with Symphonize under a comparable number of parameters. Besides, we provided more detailed explanations for the components of the network and examples of tables with larger fonts in the uploaded PDF.

Could we kindly ask if our responses have addressed for your concerns and if you have any new questions? Thanks for your time and effort.

Authors of Paper ID 3532