Voxel Proposal Network via Multi-Frame Knowledge Distillation for Semantic Scene Completion

We sincerely thank you for your valuable comments on our paper's motivation, writing, and experiment. Below, we respond to your questions point-to-point.

Q1: At the beginning of the abstract, this paper mentioned that most previous methods use 3D/2D convolutions or attention mechanisms, having limitations in directly constructing geometry and accurately propagating features from related voxels. However, how can the proposed method tackle this problem?

Answer: The lost parts of the 3D shape lack sufficient information to infer the geometry and semantic labels. Thus, the completion model needs to regard the visible parts of the 3D shape as the cue to infer the lost parts. The existing models based on the deep networks typically use the convolution or attention scheme to learn the implicit correlation between the visible and lost parts from the training data. During the inference phase, the learned correlation helps the completion model to propagate the features of the visible parts to the underlying locations of the lost parts, whose geometry and semantic labels are inferred. Note that the lost details of the incomplete 3D shape are undetermined during the inference phase. If the above feature propagation occurs between the visible parts and the locations without object occupancy, the inference of geometry and semantic labels likely fails.

We propose the Confident Voxel Proposal (CVP) to address the limitations of existing models. A CVP can be regarded as an explicit reconstruction of the lost details of the 3D shape. What sets our approach apart from the existing methods is that we provide multiple CVPs for each frame, ensuring a more comprehensive shape reconstruction of the lost details. These CVPs facilitate feature propagation between the visible parts and the confident locations with object occupancy, thereby significantly improving the completion result. Table 3 of the rebuttal PDF compares VPNet with and without the explicit learning of CVPs. This is done by using/removing the geometry loss $\mathcal{L}_{geo}$ in Eq.(10) of the paper. VPNet, with the explicit learning of CVPs, achieves IoU and mIoU better than the counterpart without the explicit learning of CVPs (see the first and second rows of Table 3 of the rebuttal PDF).

Q2: The author employs a single branch with multiple subnetworks for single-frame input, which seems processing the input sequentially. But line117 describes it using the "branch", which can easily be misinterpreted as parallel. Besides, to my understanding, the multi-frame branch distill the knowledge of different timestamps to the results from different blocks, aiming to enhance the performance of feature propagation. It would better to give an overview before introducing the specific design of the network components.

Answer: We will follow your suggestion to clarify the branches and add an overview of each network component, briefly describing its input and output.

Q3: The com. w/o CVP and com. w/ CVP are confusing in the Table 1.

Answer: By using "com. w/o CVP", we mean the 3D branch (represented by pink blocks) in Figure 1(a) works without the subsequent CVPs, while "com. w/o CVP" means the 3D branch with CVPs. We will add this clarification to the manuscript.

Q4: Comparison with other knowledge distillation algorithms (at least three algorithms) and the proposed MFKD is necessary.

Answer: We follow your advice to compare the proposed MFKD with other knowledge distillation algorithms without changing the other parts of our framework. We compare MFKD with KD [Statistics 2015], PVKD [CVPR 2022], and DSKD [CVPR 2023] in Table 2 in the PDF along with the rebuttal. As shown in Table 2 of the rebuttal PDF, KD hurts the completion result because it only distills the features of the global scenes rather than proposals. PVKD and DSKD adopt distribution-based constraints but lack optimization before fusing voxel proposals across frames. In contrast, MFKD overcomes the above issues and achieves better completion results.

Q5: SCPNet also distills knowledge from multi-frame inputs to single-frame network. More analysis on the differences between MFKD and SCPNet is necessary.

Answer: Our MFKD contains core contributions different from those of SCPNet. The existing methods like SCPNet and MonoOcc with MFKD generally compute the visual feature of a single frame, where they fuse multiple frames’ features to capture the temporal information. They distill the knowledge of fused frames into the Student model. During the above distillation process of the existing methods, the Student model misses the opportunity to learn to infer the lost details of each frame, which means the Student model only utilizes the available information of multiple frames. We propose a novel MFKD scheme, which distills the knowledge of multiple frames and each frame’s inference of the lost details (named the Confident Voxel Proposal in our paper) into the Student model. The confident voxel proposals of different frames, which show consistent shapes, likely play as the overlapping objects across frames. They provide richer information for completing the 3D shape. Please also see our response to Q1 of Reviewer Zma5 for more details on the motivation for using MFKD in semantic scene completion.

Furthermore, a CVP explicitly reconstructs the lost details of the 3D shape. Our approach provides multiple CVPs for each frame, ensuring a more comprehensive shape reconstruction of the lost details. In contrast to other methods, CVPs facilitate feature propagation between the visible parts and the confident locations with object occupancy, thereby significantly improving the completion result. Please also see our response to Q2 of Reviewer 2ocQ for the discussion on the importance of distilling the knowledge of CVPs for semantic scene completion.

I appreciate the authors' response to my questions. Current manuscript lacks the valuable insights of the authors, which are more important for other researchers. It would better to update the overall manuscript in the revision. I'm willing to raise my score to weak accept later. Besides, I have the last question. As responded to the Reviewer Rcs7, VPNet achieves better performance than the SCPNet when using 3D convolutions. Will the authors release this part of code? (including training and evaluation)

Dear Reviewer 8T9a,

Thank you again for your review. We are pleased to see that the questions raised by you are solved.

Best,

Authors of Paper ID 4361

We sincerely thank you for your valuable comments on our paper's symbols and motivation. Below, we respond to your questions point-to-point.

Q1: For clarity, the paper uses massive symbols to elaborate technical details which is hard to follow. I think there could be some abstraction.

Answer: We recognize that complex symbols stem from multiple levels of superscripts and subscripts. We will trim the redundant superscripts and subscripts to simplify these symbols and equations.

Q2: For motivation, the paper claims that existing methods have problem with directly constructing scene geometry and accurately propagating voxel features, but I do not see why this is true and how VPNet is better in these aspects.

Answer: Please see our response to the second common question. The lost parts of the 3D shape lack sufficient information to infer the geometry and semantic labels. Thus, the completion model needs to regard the visible parts of the 3D shape as the cue to infer the lost parts. The existing models based on the deep networks typically use the convolution or attention scheme to learn the implicit correlation between the visible and lost parts from the training data. During the inference phase, the learned correlation helps the completion model to propagate the features of the visible parts to the underlying locations of the lost parts, whose geometry and semantic labels are inferred. Note that the lost details of the incomplete 3D shape are undetermined during the inference phase. If the above feature propagation occurs between the visible parts and the locations without object occupancy, the inference of geometry and semantic labels likely fails.

We propose the Confident Voxel Proposal (CVP) to address the limitations of existing models. A CVP can be regarded as an explicit reconstruction of the lost details of the 3D shape. What sets our approach apart from the existing methods is that we provide multiple CVPs for each frame, ensuring a more comprehensive shape reconstruction of the lost details. These CVPs facilitate feature propagation between the visible parts and the confident locations with object occupancy, thereby significantly improving the completion result. Table 3 of the rebuttal PDF compares VPNet with and without the explicit learning of CVPs. This is done by using/removing the geometry loss $\mathcal{L}_{geo}$ in Eq.(10) of the paper. VPNet, with the explicit learning of CVPs, achieves IoU and mIoU better than the counterpart without the explicit learning of CVPs (see the first and second rows of Table 3 of the rebuttal PDF).

Q3: The authors discuss the limitations of VPNet in the appendix but do not provide possible solutions.

Answer: In the appendix, we introduce VPNet's limitations, which stem from inaccurate semantic segmentation and offset learning. These limitations lead to unreliable feature propagation, leaving much room for improving the completion result. We can equip VPNet with a more accurate segmentation subnetwork to alleviate the above limitations. Furthermore, we can iteratively conduct the offset learning and feature propagation, where the offset learning can be based on a more refined 3D shape. The extract computation during inference brought by the iteration can be saved by knowledge distillation.

Dear Reviewer 2ocQ,

We thank you again for your valuable comments, which significantly help us to polish our paper. Could we kindly know if the responses have addressed your concerns and if further explanations or clarifications are needed? Your time and efforts in evaluating our work are appreciated greatly.

Best,

Authors of Paper ID 4361

Dear Reviewer 2ocQ,

Again, please allow us to extend our sincere thanks to you, for your time and efforts of reviewing our paper. As the deadline for the authors' response is approaching, we sincerely request your comment on our primary response. This will definitely give us a valuable chance to address the questions unsolved.

Best,

Authors of Paper ID 4361

We sincerely thank you for your valuable comments on our paper's experiments and core contribution. Below, we respond to your questions point-to-point.

Q1: Lack of Comparison with State-of-the-Art Methods: The performance of VPNet is not compared with current state-of-the-art methods such as SCPNet. Specifically, (mIoU) achieved by VPNet is far lower than that of SCPNet. Or authors can argue and clarify clearly the protocol differences that render the comparison invalid.

Answer: SCPNet utilizes many 3D convolution based subnetwork to complete the 3D shape, which makes the completion model complex while we only utilize simple convolution for refinement of CVP. And as SCPNet indicates, this improve the mIoU greatly. Though SCPNet conducts the sparse convolution to save a portion of computation, the dense occupancy of voxel-wise features still makes the computation very expensive. Table 1 of the rebuttal PDF compares SCPNet (the first row) with our VPNet (the second row) concerning the model parameters (M), GPU memory (GB), inference speed (FPS), IoU, and mIoU. VPNet outperforms SCPNet regarding IoU, requiring fewer model parameters and much faster inference time. Because IoU indicates the accuracy of predicting the object occupancy, VPNet with a higher IoU shows a more robust power of inferring the geometry of the 3D shape than SCPNet.

In contrast to IoU, we use mIoU to measure the accuracy of predicting the voxel-wise semantic labels of the 3D shape. SCPNet yields a higher mIoU than VPNet, because 3D convolution better captures the semantic context of the objects. Though using 3D convolution with expensive computation contradicts our intention of using knowledge distillation to save model parameters and computation, we equip VPNet with more 3D convolutions after extracting point features and voxelization like SCPNet for a fair comparison. Table 1 of the rebuttal PDF shows that VPNet with 3D convolution (the third row) successfully outperforms SCPNet regarding mIoU, and it keeps similar model parameters with SCPNet.

Q2: SCPNet has already implemented multi-frame distillation to enhance performance. Thus, VPNet's use of Multi-Frame Knowledge Distillation (MFKD) is not readily qualified as a core contribution in the title. Or authors can dig into the effects of multi-frame distillation in different frameworks like SCPNet and MonoOcc.

Answer: Our MFKD contains core contributions that are different from those of SCPNet and MonoOcc. The existing methods like SCPNet and MonoOcc with MFKD generally compute the visual feature of a single frame, where they fuse multiple frames’ features to capture the temporal information. They distill the knowledge of fused frames into the Student model. During the above distillation process of the existing methods, the Student model misses the opportunity to learn to infer the lost details of each frame, which means the Student model only utilizes the available information of multiple frames. We propose a novel MFKD scheme, which distills the knowledge of multiple frames and each frame’s inference of the lost details (named the Confident Voxel Proposal in our paper) into the Student model. The confident voxel proposals of different frames, which show consistent shapes, likely play as the overlapping objects across frames. They provide richer information for completing the 3D shape. Please also see our response to Q1 of Reviewer Zma5 for more details on the motivation for using MFKD in semantic scene completion.

Furthermore, a CVP explicitly reconstructs the lost details of the 3D shape. Our approach provides multiple CVPs for each frame, ensuring a more comprehensive shape reconstruction of the lost details. In contrast to other methods, CVPs facilitate feature propagation between the visible parts and the confident locations with object occupancy, thereby significantly improving the completion result. Please also see our response to Q2 of Reviewer 2ocQ for the discussion on the importance of distilling the knowledge of CVPs for semantic scene completion.

Dear Reviewer Rcs7,

We thank you again for your valuable comments, which significantly help us to polish our paper. Could we kindly know if the responses have addressed your concerns and if further explanations or clarifications are needed? Your time and efforts in evaluating our work are appreciated greatly.

Best,

Authors of Paper ID 4361

Dear Reviewer Rcs7,

We thank you again for your valuable comments, which will significantly help us to polish our paper. We will follow your suggestion to add the comparison with SCPNet to the experiment section, and add the insights of proposing MFKD and comparison with SCPNet/MonoOCC on distillation to the introduction section.

Best,

Authors of Paper ID 4361

Dear Reviewer Rcs7,

Again, please allow us to extend our sincere thanks to you, for your time and efforts of reviewing our paper. As the deadline for the authors' response is approaching, we sincerely request your further comment on our primary response. This will definitely give us a valuable chance to address the questions unsolved.

Best,

Authors of Paper ID 4361

We sincerely thank you for your valuable comments on the motivation and figure. Below, we respond to your questions point-to-point.

Q1: My biggest concern is that the author's motivation for using MFKD is not clearly explained. Why use it to predict voxel-wise labels?

Answer: Here, we introduce the motivation of MFKD for semantic scene completion. Suppose multiple frames of point clouds are available rather than a single frame only. In that case, people can leverage these sequential point clouds to complement each other and infer a complete 3D shape with semantic labels represented by voxels or point clouds. Though this strategy is a natural technical choice, using multiple frames needs a complex model for learning the knowledge, including the visual pattern of a single frame and the temporal relationship between objects across various frames, substantially increasing the model complexity and inference time. MFKD belongs to the family of knowledge distillation, where we transfer the knowledge learned by a complex Teacher model to a simpler Student model. Specifically for semantic scene completion, MFKD can distill the knowledge of multiple frames learned by the Teacher model into the Student model that only takes input as a single frame to infer the complete 3D shape quickly and accurately.

The existing methods like SCPNet and MonoOcc with MFKD generally compute the visual feature of a single frame, where they fuse multiple frames’ features to capture the temporal information. They distill the knowledge of fused frames into the Student model. During the above distillation process, the Student model misses the opportunity to learn to infer the lost details of each frame, which means the Student model only utilizes the available information of multiple frames. We propose a novel MFKD scheme, which distills the knowledge of multiple frames and each frame’s inference of the lost details (named the Confident Voxel Proposal in our paper) into the Student model. The confident voxel proposals of different frames, which show consistent shapes, likely play as the overlapping objects across frames. They provide richer information for completing the 3D shape.

Our paper follows the conventional protocol of semantic scene completion to represent the 3D shape as voxels for a fair comparison with other methods. MFKD can also help to predict point-wise labels in the 3D point cloud, which will be investigated in our future work.

Q2: Figure 3 is too complicated to understand.

Answer: We follow your comment to simplify Figure 3, the simplified figure is now included as Figure 1 of the rebuttal PDF.

Q3: Can authors open source code?

Answer: We release the code package via this anonymous Github repository https://github.com/anonymous-github-VPNet/VPNet.

Dear Reviewer Zma5,

Thank you again for your review. We are pleased to see that the questions raised by you are solved.

Best,

Authors of Paper ID 4361