To Reviewer VfkU:

1. Ablation study of the uncertainty map

Response:

Thank you for your comments. In our practical implementation, the uncertainty of the calculations (i.e., variance) is performed in the LAB color space and is normalized. Therefore, in this part, we will not explicitly define the threshold for uncertainty. If you are concerned about the uncertainty threshold referring to $\vartheta$ in Eq. (3), we investigate how different $\vartheta$ increasing strategies affect reconstruction quality. The hyperparameter $\vartheta$ is initialized by 0 and increases by 0.1 with each iteration in our paper. We evaluate five different increase ratios {0.1, 0.2, 0.3, 0.4, 0.5} and analyze their impact on reconstruction quality across iterations using our dataset. As shown in Figure 12 (b), all ratios lead to improved PSNR values over iterations. In the first two iterations, a higher increase ratio improves the reconstruction performance. However, an increase ratio above 0.1 indicates that the algorithm becomes overly confident in the inpainting areas too early, resulting in insufficient interaction of geometric and semantic information between the VISTA-GI and VISTA-CL modules, which subsequently leads to a decline in reconstruction performance in later iterations.

2. About iteration and time consumption

Response:

Thank you for the comments.

The influence of iteration number: In our setup, the number of iterations is strongly correlated with the hyperparameters involved in Figure 12. Simply increasing the number of iterations without altering the hyperparameters goes against the original intent. To evaluate the impact of different numbers of iterations, we analyzed two hyperparameters that related to the number of iterations. By examining Figures 12 (a) and (b), we can infer that fewer iterations lead to insufficient interaction of geometric and semantic information between the VISTA-GI and VISTA-CL. Conversely, excessive iterations do not improve the performance of the challenging scene and may introduce excessive computational costs.

The total time cost: To quantitatively evaluate performance and computational efficiency, we compare our method against baseline approaches (InFusion, SPIn-NeRF, and SpotLess) on the synthetic scene shown in Fig.8. This scene provides ground truth data, enabling evaluation through reference-based metrics for both rendering quality and computational efficiency during optimization. As shown in Table 9, while our method incurs additional computational overhead compared to vanilla 3DGS due to integrating iterations and diffusion models, it achieves superior rendering quality while maintaining comparable efficiency to state-of-the-art 3DGS methods (e.g., SpotLess). Furthermore, our approach demonstrates significantly better reconstruction quality while being approximately 10× faster than leading NeRF-based methods such as SPIn-NeRF.

Our method indeed introduces higher computational resource demands due to the iterative repair process. For one scenario in our dataset, the specific time comparisons are as follows:

Table 9: Quantitative results and time costs on the synthesis data in Fig.8.

Time cost for each iteration: In the aforementioned case, our method undergoes three rounds of iterations, with an average iteration time of approximately 10 minutes per iteration. By optimizing the current pipeline and reusing the reconstruction results from the previous iteration, it may be possible to further reduce the runtime to achieve real-time performance. Such acceleration efforts are still worth further investigation.

To Reviewer qvjD:

1. Ablation study of the conceptual learning

Response:

Thank you for the comments. We have conducted ablation experiments on two components of our algorithm using our dataset, and the specific results are as follows:

Our experiments demonstrate two key findings. First, attempting reconstruction using only a 2D generative model (i.e., VISTA-CL) without VISTA-GI leads to significantly degraded image quality metrics. This validates that VISTA-GI's uncertainty guidance effectively mitigates multi-view inconsistencies during 3D reconstruction, resulting in higher-quality outputs. Second, while omitting VISTA-CL maintains image quality comparable to existing methods like SplotLess and SPIn-NeRF, the lack of concept-guided learning significantly reduces CLIP-Score metrics. This indicates that without conceptual constraints, the inpainting process produces results that are visually plausible but semantically inconsistent with the scene context.

To address the concerns, we have added the discussion in Sec 5.

2. Multi-view consistency

Response:

Thank you for the comments. Given multi-view images from a static scene, the original 3DGS trains the Gaussian splats through the $L_1$ loss and SSIM loss function (i.e., Eq.(1) in the submission) to make sure the Gaussian splats can render detailed-preserved image across all perspectives. Hence, the texture consistency across views is implicitly achieved via the mix of $L_1$ loss and SSIM loss functions that are good at the reconstruction of structure and details and are validated in [A].

However, when the scene contains dynamic objects, the object can be seen in one view but cannot be observed in another view. This leads to conflicts between different view renderings and affects the training of the Gaussian splats through the original objective function.

Our visibility uncertainty map indicates the potential conflicts across different views and we use it to reweight the contributions of different pixels in Eq. (1) and get Eq. (4) in the submission. Intuitively, we assign low weights to the pixels with high visibility uncertainty. Such a strategy avoids conflicts during the training of Gaussian splat and leads to better texture consistency.

We analyzed the quality of reconstruction between the masked and non-masked areas in an underwater scene. The results are as follows:

Table 8: Quantitative results of different areas.

It can be observed that when the mask is not used, our method maintains much better texture consistency in both the masked and non-masked areas. For the restoration of the masked areas, our method also ensures texture consistency by introducing the limitations imposed by uncertainty.

[A] H. Zhao, O. Gallo, I. Frosio and J. Kautz, "Loss Functions for Image Restoration With Neural Networks," in IEEE Transactions on Computational Imaging, vol. 3, no. 1, pp. 47-57, 2017.

4. Influence of image resolution

Response:

Thank you for the comments. In our experiment setup, we use the stable diffusion v1.5 as the inpainting model and train and test the model following its default setup: if the input image has a resolution higher than 512x512, we crop the image to a new size that is both the closest to the original image size and a multiple of 8; if the input image is smaller than 512x512, we rescale the image to 512x512. To analyze the influence of the strategy on different original resolutions, given an original scene with input images having the size of 1299x974, we downsample these images to four resolutions: 64x64, 128x128, 256x256, and 512x512. Then, for each resolution, we can build a 3D model and evaluate the rendering quality. As shown in Table 2, we observe that: (1) reducing the resolution to 512x512 does not significantly impact any of the metrics, demonstrating our method's robustness to substantial resolution changes. (2) further decreasing the resolution leads to gradual degradation in reference-based metrics, while non-reference metrics remain relatively stable.

We have added the above discussion in the Appendix Sec.A.4.

Table 2: Quantitative ablation results of different resolutions.

5. Concerns about Fig. 7

Response:

We appreciate your careful review. Regarding the fish in the top left corner of Fig. 7 is retained while the other fish are removed, this is primarily because the fish remains stationary across different views (as evident in the optical flow map of Fig. 7, where the top-left fish exhibits low flow values at its center). Consequently, it has a lower value in the visibility uncertainty map (see the corresponding map in Fig. 7). Without using a mask to label this area for repair manually, the fish's geometric characteristics resemble those of a stationary object, such as a rock, making it indistinguishable from our uncertainty detection system.

In contrast, moving fish create significant geometric inconsistencies across viewpoints, enabling our uncertainty detection to flag them as anomalies. This leads to their removal through the inpainting process. To address these challenging scenarios, we introduced mask annotations for fish detection, providing semantic guidance for our inpainting method. As shown in the last column of Fig. 7, incorporating the mask ensures the successful removal of the top-left fish.

To address the concern, we have revised the discussion in A.1 and Figure 7 by adding the optical flow results and uncertainty map.

To Reviewer M4f5:

1. Impact of Hyperparameters

Response:

Thank you for the suggestions and comments. We have added the discussion about the influence of the hyperparameter $\vartheta$ in Eq. (3) and noise reduction ratios in the Appendix section Sec A.3 with the revised Figure 12.

Impacts of $\vartheta$ in Eq. (3). We investigate how different $\vartheta$ increasing strategies affect reconstruction quality. The hyperparameter $\vartheta$ is initialized by 0 and increases by 0.1 with each iteration in our paper. We evaluate five different increase ratios {0.1, 0.2, 0.3, 0.4, 0.5} and analyze their impact on reconstruction quality across iterations using our dataset. As shown in Figure 12 (b), all ratios lead to improved PSNR values over iterations. In the first two iterations, a higher increase ratio improves the reconstruction performance. However, an increase ratio above 0.1 indicates that the algorithm becomes overly confident in the inpainting areas too early, resulting in insufficient interaction of geometric and semantic information between the VISTA-GI and VISTA-CL modules, which subsequently leads to a decline in reconstruction performance in later iterations.

Impact of noise reduction ratios in diffusion inference. During diffusion model inference, we investigate how different noise reduction strategies affect reconstruction quality. Starting from an initial noise strength of 1.0, we systematically decrease the noise at each iteration by a fixed ratio. We evaluate four different reduction ratios {0.1, 0.2, 0.3, 0.4} and analyze their impact on reconstruction quality across iterations using our dataset. As shown in Figure 12 (a), while all ratios lead to improved PSNR values over iterations, the reduction ratio of 0.2 achieves optimal convergence in the fewest iterations. Based on this empirical analysis, we adopt 0.2 as the noise reduction ratio in our method.

2. Computing cost comparison and analysis

Response:

Thank you for the comments. To quantitatively evaluate performance and computational efficiency, we compare our method against baseline approaches (InFusion, SPIn-NeRF, and SpotLess) on the synthetic scene shown in Fig.8. This scene provides ground truth data, enabling evaluation through reference-based metrics for both rendering quality and computational efficiency during optimization. As shown in Table 1, while our method incurs additional computational overhead compared to vanilla 3DGS due to integrating iterations and diffusion models, it achieves superior rendering quality while maintaining comparable efficiency to state-of-the-art 3DGS methods (e.g., SpotLess). Furthermore, our approach demonstrates significantly better reconstruction quality while being approximately 10× faster than leading NeRF-based methods such as SPIn-NeRF.

Table 1: Quantitative results and time costs on the synthesis data in Fig.8.

3. Normalization on $\mathbf{U}_i$

Response:

Thank you for the comments. To standardize the visibility uncertainty values, we apply normalization to the pixel-wise uncertainty map computed using Eq (2). Specifically, each pixel's uncertainty value is divided by the standard deviation of all uncertainty values in the map. The resulting normalized visibility uncertainty map is denoted as $\mathbf{U}_i$.

To address the concern, we have added the above revision into Sec. 4.1.

Dear Reviewer M4f5,

I hope this message finds you well.

I am one of the authors. First, I would like to express my gratitude for your time and effort in reviewing our paper. We greatly appreciate your valuable feedback and have made the necessary revisions based on your comments.

As the submission deadline approaches, we would like to ask if you could provide further insights or feedback as soon as possible. Your thoughts are very important to us, and we are eager to make any final adjustments before resubmitting.

Thank you for your understanding and support!

To Reviewer eMA4:

1. How visibility uncertainty benefits texture consistency

Response:

Thank you for the comments. Given multi-view images from a static scene, the original 3DGS trains the Gaussian splats through the $L_1$ loss and SSIM loss function (i.e., Eq.(1) in the submission) to make sure the Gaussian splats are able to render detailed-preserved image across all perspectives. Hence, the texture consistency across views is implicitly achieved via the mix of $L_1$ loss and SSIM loss functions that are good at reconstruction of structure and details and are validated in [A].

However, when the scene contains dynamic objects, the object could be saw in one view but cannot be observed at another view. This leads to the conflicts between different view renderings and affects the training of the Gaussian splats through the original objective function.

Our visibility uncertainty map indicates the potential conflicts across different views and we use it to reweight the constributions of different pixels in the Eq. (1) and get Eq. (4) in the submission. Intuitively, we assign low weights to the pixels with high visibility uncertainty. Such a strategy avoids the conflicts during the training of Gaussian splat, and leads to better texture consistency.

We analyzed the quality of reconstruction between the masked and non-masked areas in an underwater scene. The results are as follows:

Table 3: Quantitative results of different areas.

It can be observed that when the mask is not used, our method maintains much better texture consistency in both the masked and non-masked areas. For the restoration of the masked areas, our method also ensures texture consistency by introducing the limitations imposed by uncertainty.

[A] H. Zhao, O. Gallo, I. Frosio and J. Kautz, "Loss Functions for Image Restoration With Neural Networks," in IEEE Transactions on Computational Imaging, vol. 3, no. 1, pp. 47-57, 2017.

2. Quantitative analysis of large viewpoint differences

Response:

To evaluate the impact of variants of viewpoint difference, we first capture 34 images from continuously distributed viewpoints around a scene to create a ground truth (GT) 3DGS model. We then systematically reduce the number of viewpoints by sampling them at different intervals $\{2,3,4,5,6,7\}$, where larger intervals represent larger viewpoint differences. For each sampling interval, we construct a new 3DGS model and assess its quality by comparing its rendered images against those from the GT model using standard metrics: LPIPS, SSIM, and PSNR. This methodology allows us to analyze how viewpoint difference affects reconstruction quality quantitatively.

Table 4: Quantitative results of large viewpoint differences.

Considering that the reduction in available viewpoints for training itself leads to a decrease in 3DGS reconstruction quality, our method still achieves good results even with significant viewpoint variation. This validates that our approach can detect inconsistencies between viewpoints and repair those areas despite the large viewpoint differences. However, in extreme cases, the absence of key viewpoints results in a loss of critical complementary information between viewpoints, leading to a significant decline in the reconstruction metrics of the scene.

To validate our advantages in the extreme case with large viewpoint differences, we conducted a quantitative evaluation of various methods for the extreme case mentioned in Figure 8, and the results are as the following table. It can be seen that our method still outperforms existing methods in removing dynamic distractors under such extreme conditions.

Table 5: Quantitative comparison of different methods in the extreme case.

To address the concerns, we have added the discussion in Section A.5.

Dear Authors,

Thank you for your detailed and comprehensive responses to my previous comments. I truly appreciate the effort and the additional insights you’ve provided. However, I still find that several critical concerns remain unaddressed, leading me to maintain my current score. Below are my key points:

1. Quantitative Comparison with GScream on SPIn-NeRF Dataset: While you noted the difficulty in running GScream on your underwater dataset, I believe a quantitative comparison on the SPIn-NeRF dataset, which includes ground truth, would provide a stronger evaluation of texture consistency. Without this comparison, it is challenging to conclusively validate your method's advantages over GScream in scenarios with both masked and non-masked regions.

2. Blocky Artifacts in Uncertainty Maps (Figure 11): The uncertainty maps for the SPIn-NeRF dataset (Figure 11) exhibit blocky artifacts. It is unclear whether these artifacts are a byproduct of your method or the dataset itself. More importantly, their impact on the reconstruction quality needs further explanation. Without clarity on this issue, the robustness of your approach is difficult to assess.

3. Discrepancies Between Table 4 and Table 5: There is a noticeable mismatch between the quantitative results in Table 4 and Table 5. If these tables represent different datasets or evaluation conditions, this needs to be explicitly clarified. The current presentation raises questions about the consistency and reproducibility of your results.

4. Ablation Studies on SPIn-NeRF Dataset: Your ablation studies were conducted on the Underwater 3D Inpainting Dataset. While insightful, the lack of ablation results on the SPIn-NeRF dataset, which includes ground truth, is a missed opportunity. Such experiments would provide more concrete evidence of the contributions of VISTA-GI and VISTA-CL, especially in controlled settings where objective metrics can be thoroughly assessed.

While I commend the efforts made to improve the clarity and comprehensiveness of your work, these unresolved issues leave significant gaps in validating the proposed approach. Therefore, I will maintain my current score, as I believe further revisions are necessary to address these critical aspects.

Reviewer eMA4

Thank you for your attention to our work.

1. We compared our method with the data from the GScream, and the results are as follows:

It can be seen that our method outperforms all related methods in two metrics, but there is one metric that is lower than GScream. Considering that other methods cannot repair dynamic scenes, while our method applies to dynamic and static scenes, we believe our method has certain advantages.

2. The blocky areas are not artifacts. The explanation is as follows: When calculating uncertainty, our method references the rendered images from adjacent viewpoints, but not every point can be observed from any adjacent viewpoint. Therefore, we need to assess the visibility of spatial points and remove those not within the camera's visible space. This operation reflects on the uncertainty map, resulting in blocky areas (which can be considered as the intersection of the visible regions from adjacent viewpoints projected onto the current viewpoint).

   This phenomenon does not lead to a decrease in performance because such areas are merely due to the inconsistent number of samples from adjacent viewpoints for different points. Essentially, they are effective samples of the true distribution, and the estimated uncertainty/variance is also valid. This phenomenon can be resolved by increasing the number of samples from adjacent viewpoints, but it will introduce more computational costs.

3. Tables 4 and 5 were tested on the same synthetic underwater dataset, and the visual content is shown in Figure 8. Table 4 compares the time required for different methods, while Table 5 compares the results obtained using different resolutions during the reconstruction process. Since it is the same scene, our optimal results are consistent.

4. We conducted ablation experiments on our method using the Spin-NeRF dataset, the results are as follows:

It can be seen that VISTA-GI has a more significant impact on the reconstructed scene than VISTA-GL. We analyze this as being due to the latter's reliance on the uncertainty of the former. When VISTA-GI cannot generate high-quality uncertainty images, it first leads to an inability to utilize uncertainty to help exclude interference during the reconstruction process. Consequently, the subsequent VISTA-GL also struggles to identify the areas that need repair, resulting in further performance degradation. The absence of VISTA-GL has a relatively smaller impact, but due to the loss of semantic constraints, the generated areas are difficult to ensure their validity, leading to a decline in performance.

Thank you once again for your attention and valuable suggestions for our work. We look forward to your recognition of our efforts.