Missing baselines

Thank you for your question. During the rebuttal phrase, we test the semantic segmentation performance by applying the state-of-the-art tracking method DeAOT [1] and image segmentation method Mask R-CNN [2] to the rendered images from MonoNeRF [3]. The results are shown in Table 2 and Figure 8. We also discuss the performance of these methods in the General Response. DeAOT lacks semantic understanding of 3D scenes and hence predicts inconsistent semantic labels in different views, leading to a severe performance drop as shown in Table 2. As Mask R-CNN only learns semantic information from single images, Figure 8 shows that it predicts inconsistent semantic labels of the same instances in different frames.

In addition, we also compare more Dynamic NeRF baselines by manually adding semantic segmentation heads to these methods in the General Response and Table 1 in the paper.

RGB rendering preformance

Thank you for your question. We present the PSNR, SSIM and LPIPS scores of rendered RGB images in the following table. Our model could achieve slightly better RGB color rendering quality compared to MonoNeRF [3] and DynNeRF [4], which demonstrates that our model is able to conduct the instance-level scene editing application. We also add the results to Section G.1 and Table 11 in the Appendix.

Visualization and discussion of the flow fields

In Section G.2 and Figure 6 in the Appendix, we show the visualization of the rendered images, estimated flow fields, predicted semantics and the point correspondence of dynamical foreground in 3 consecutive frames. We also discuss these results in the General Response. We agree that we do not force the flows to map the same parts of an object moving in time. However, Figure 6 shows that our model is able to predict flows that map similar parts of the dynamic foreground.

Minor issues

• Missing related work. We have inserted the Panoptic Neural Field [5] into the Related Work section in the paper, page 3.
• Ground truth flow. Thank you for your suggestion. we have corrected "the ground truth flows" to "the flows generated from the pretrained model".
• Incorrect parenthesis. Thanks. We have revised the closing parenthesis of $\boldsymbol{F}_{dy}(I;{\boldsymbol{\Gamma}})$ near the equation (7) in the paper.
• "Boundray" typo. Thank you. We have revised "boundray" to "boundary" in page 8.
• Table 2 caption. Thanks. We have revised "qualitative" to "quantitative" in Table 2 caption.

Questions

• RGB frames in semantic completion and tracking settings. We follow MonoNeRF [3] and use the same subset of frames for RGB supervision.
• Indices of the frames in Figure 3. In the completion setting, Figure 3 shows the semantic prediction results from the frame #8, which are 2-frame far from the frame #10 with the semantic labels. For the dynamic scene tracking setting, Figure 3 shows the results from the frame #10, which are 7-frame far from the frame #3 with the semantic labels.
• The input to the semantic heads. For the DynNeRF [4] and MonoNeRF [3] methods, the input is the feature vector with 256 channels obtained from a 8-layer MLP with ReLU activations.

References:

[1] Yang, Zongxin, and Yi Yang. "Decoupling features in hierarchical propagation for video object segmentation." In NIPS, pp. 36324-36336, 2022.

[2] He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. "Mask R-CNN." In ICCV, pp. 2961-2969. 2017.

[3] Tian, Fengrui, Shaoyi Du, and Yueqi Duan. "Mononerf: Learning a generalizable dynamic radiance field from monocular videos." In ICCV, 2023.

[4] Gao, Chen, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. "Dynamic view synthesis from dynamic monocular video." In ICCV, pp. 5712-5721. 2021.

[5] Kundu, Abhijit, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J. Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. "Panoptic neural fields: A semantic object-aware neural scene representation." In CVPR, pp. 12871-12881. 2022.

Thank you for the answers. The additional baseline experiments strengthen the paper, although for Mask R-CNN it would be easy to use e.g. the Hungarian algorithm to match instances in consecutive frames as is common practice for video instance segmentation. The RGB rendering performance seems in line with the baselines, although it is only reported for two sequences in the Nvidia dynamic scenes datasets so it can not be easily compared to other methods than the reported ones. The flow fields look as expected, since optical flow was used for supervision.

Thank you for waiting!

To test the performance of the Mask R-CNN + Hungarian algorithm, we choose person, truck and umbrella classes in our dataset, which overlap with the COCO dataset. We use the Mask R-CNN pretrained on the COCO dataset to predict semantic labels in each video frame and use the Hungarian algorithm to match the same instances in different frames. We present the results in the following table. Our model outperforms Mask R-CNN + Hungarian algorithm by a large margin, which demonstrates the superiority of our model that leans view-agnostic semantic labels in 3D space.

Thank you for your response!

The superiority of our model compared to image-based or video-based segmentation methods is that our model predicts view-agnostic semantic labels of each point in the 3D scenes, while image-based or video-based segmentation methods lack the semantic understanding of the 3D scenes and hence predict inconsistent semantic labels across different views, which may lead to a severe performance drop. We still agree that we should evaluate the performance of Mask R-CNN + Hungarian for a much fairer comparison. Currently we are testing the performance of Mask R-CNN + Hungarian algorithm and we will report the results as soon as possible.

The reason why we report the RGB rendering performance on the joint learning of Jumping and Skating scenes is that the examples of instance-level editing in the paper are conducted in this setting. We totally agree that we should report the performance of the entire dataset. In the following table, we report the RGB rendering performance on the entire Nvidia Dynamic Scene dataset. We follow the experimental setting of learning from multiple scenes in the MonoNeRF paper. It can be seen that our model reaches slightly better performance compared to MonoNeRF.

We also report all the details of the PSNR, SSIM and LPIPS score in the following tables.

Shortcomings

Thank you for your suggestion. We present the shortcoming of our model in Figure 10 in the Appendix. Due to the imperfect predictions of flows in the tracking setting, our model fails to estimate precise semantic labels of dynamic parts.

Occlusion and disocclusion problems

To evaluate the performance of our model in the occlusion and disocclusion situations, as shown in Section G.4 and Figure 7 in Appendix, we add an occlusion region in both the static background and dynamic foreground in the frame #6, and remove the occlusion in the frames #5 and #7. Concretely, we manually occludes the both RGB colors and semantic labels of the region in the frame #6. As Figure 7 shows, the optical flow estimation method fails to estimate the object movement due to the occlusion. We train the model with the occluded image, wrong optical flow maps and semantic labels. The occlusion may provide wrong information to train the semantics, which leads to the incorrect semantic prediction in the occlusion part. However, since our model learns the semantics from flows, it successfully predicts the semantics in disocclusion parts by incorporating correct information among the flows in non-occluded regions.

More comparisons with other related works

Thank you for your suggestion. We have tested other methods related to dynamic scene reconstruction in Table 1 in the paper and have a discussion about the performance of these methods in the General Response. Besides, we also conduct comparisons with the state-of-the-art video tracking method and the image segmentation method in Table 2 and Figure 8 in the paper and Appendix.

Semantic consistency constraint

We conduct the constraint for all pixels. We visualize the pixel correspondence in consecutive frames in Section G.2 and Figure 6 in the paper and discuss the correspondence in the General Response. Figure 6 shows that our model could generate the flows that map the similar parts of the dynamic foreground moving across time. We also discuss the occlusion problem in Section G.4 and the uncertainty of noisy flows in Table 13 and Section G.3 in the Appendix.

Outdated baselines

Thank you for your suggestion. We test the performance of NeRF [1] with time, D-NeRF [2], RoDynRF [3], NSFF [4] and discuss these methods in Table 1 in the paper and in the General Response. Besides, we also conduct comparisons with the state-of-the-art video tracking method and the image segmentation method in the General Response and in Table 2 in the paper and Figure 8 in the Appendix.

Difficult to follow the equations 4-8

Thank you for your suggestion. We have revised Figure 2 in the paper to put the mathematic notations from equations 4-8 into the figure.

Clarity

Thank you for your suggestion. In Figure 3, "25% semantic labels" in the tracking setting denotes that we use the frames #1, #2 and #3 to train the model and conduct semantic view synthesis on the frames #4-12. "25%" is determined by choosing 3 out of 12 frames to train the models in a scene. "50% semantic labels" in the completion setting denotes that we select the frames #1-3 and #10-12 to train the model and render semantic views in frames frames #4-9. "50%" is determined by choosing 6 out of 12 frames to train the models. We have added these details to the caption in Figure 3 in the paper.

Marginal accuracy improvement

Thank you for your question. We agree that the results in Table 4 cannot distinguish $\boldsymbol{\Gamma}$ & $\Delta \boldsymbol{\Gamma}$, $\Delta \boldsymbol{\Gamma}$ and $\boldsymbol{\Gamma}$. However, we demonstrate that flow displacements contribute to the semantic prediction performance according to the ablation study in Table 6 in the paper and our Semantic Flow has robustness with different flow displacements according to the Table 4. In Figure 9 in the Appendix, we visualize the semantic predictions by using $\boldsymbol{\Gamma}$ & $\Delta \boldsymbol{\Gamma}$, $\Delta \boldsymbol{\Gamma}$ and $\boldsymbol{\Gamma}$. Although Table 4 shows quantitatively comparable results by using different displacements, Figure 9 presents that the semantic prediction of using $\boldsymbol{\Gamma}$ & $\Delta \boldsymbol{\Gamma}$ is qualitatively better than using $\Delta \boldsymbol{\Gamma}$ or $\boldsymbol{\Gamma}$.

Relationship between the performance of semantic predictions and optical flows

Thank you for your question. We test how much the performance of our model depends on the predicted flows from two perspectives: choosing different flow estimation methods and manually adding noise to the flow maps.

Firstly, we choose RAFT [1] and FlowNet [2] to generate the optical maps and test the performance in the setting of semantic learning from multiple scenes. The results are presented in the following table and in Table 12 in the Appendix. According to the RAFT paper, RAFT outperforms FlowNet by a large margin in various optical flow estimation tasks. Therefore, the results show that there is about 10% performance drop in mIOU matrix by using FlowNet when jointly optimizing Jumping and Skating scenes, where the dynamic foregrounds are drastically changing and FlowNet fails to predict the correct flows. On the other hand, our model reaches comparable results when jointly optimizing Balloon1 and Balloon2 scenes where the foreground variations are relatively small and FlowNet successfuly predicts the correct flows of the foregrounds.

In addition, we manually add different percentages of noise to the predicted optical flows and test the performance of our model with noisy optical flows. The details are stated in the Section G.3 in the Appendix and the results are presented in the following table and listed in Table 13 in the Appendix. Our model could achieve comparable performance by using the optical flows with small percentages of noise (< 5%). Adding more than 10% noise to the predicted flows leads to a performance decrease in mIOU matrix, which is mainly because our model could not build accurate flow fields from noisy optical flows and hence learns inaccurate semantics from imprecise flows.

Thank you for providing the flow map visualization.

Then can I ask how adding noise (or using noisy GT) can improve the performance? As it's based on NeRF, using a better GT would contribute to better accuracy, unlike training-based approaches where noise could make models more robust by perturbing GT.

Thank you for your response!

Training Semantic Flow with the noisy optical flow maps leads to the noisy flow field predicted by $\Psi_{flow}$ in equation (3) in the paper. However, since Semantic Flow takes the flow information as input to predict semantic labels as shown in equations (6)-(9) in the paper, the noisy flows can be considered as an augmentation of flow information. In this way, although the predicted flow field is inaccurate compared to the ground truth flow map, it contributes to the semantic prediction, which is similar to adding Gaussian noise to the inputs for performance improvement in traditional training-based approaches.

Again, we appreciate the reviewer’s continuous efforts to provide helpful advice and valuable input. If there is any further information or clarification you need from us, please feel free to let us know!