Gaussian Graph Network: Learning Efficient and Generalizable Gaussian Representations from Multi-view Images

Thanks for your valuable comments. Below we address specific questions.

1. The difference between this paper and other works that combine GNN and GS

Thank you for pointing out these relevant works SAGS [1] and Hyper-3DG [2]. We would like to highlight our key differences with these works.

One of the key differences is the construction of Gaussian graphs. SAGS sets each point as a node and constructs graphs for KNN points, while Hyper-3DG first obtains patch features by aggregating gaussian features within each patch, and then sets each patch as a node with its feature. However, SAGS is still an optimization-based method, where the Gaussian-level graph is less efficient due to the large number of Gaussian points, while the patch-level graph in Hyper-3DG fails to support message passing to exploit Gaussian-level relations.

In our work, we propose a hierarchical method to construct Gaussian Graphs - we formulate each view as a node but also fully preserve the Guassian-level structures in each node. We fully exploit the important view-related information, i.e. (1) group of pixel-aligned Gaussians belonging to the same view and (2) projection relations between different views. This information enables us to capture accurate and sparse cross-view Gaussian-level relation in an efficient manner with the edge matrix $E^{i\rightarrow j}$.

Based on our carefully-designed hierarchical structure, we specifically design linear layers for feature learning by extending the scalar weight of an graph edge to a matrix. Furthermore, we propose a new pooling strategy to fuse Gaussians based on their spatial positions to avoid the redundancy of Gaussians in pixelSplat and MVSplat, while SAGS and Hyper-3DG do not introduce pooling layers in their graph networks.

[1] Ververas, E., Potamias, R. A., Song, J., Deng, J., & Zafeiriou, S. (2024). SAGS: Structure-Aware 3D Gaussian Splatting. arXiv preprint arXiv:2404.19149.

[2] Di, D., Yang, J., Luo, C., Xue, Z., Chen, W., Yang, X., & Gao, Y. (2024). Hyper-3DG: Text-to-3D Gaussian Generation via Hypergraph. arXiv preprint arXiv:2403.09236.

2. Comparison with NeRF-based generalizable methods

We reported the comparison with NeRF-based methods on 2-view settings in Table 2 of our paper. We also conduct further experiments with different number of input views, as shown in the following table. Experimental results show that our method have better rendering quality and inference speed, which is benefit from our efficient and generalizable Gaussian representation learning.

Table 1: Comparison of NeRF-based methods on RealEstate10K with different number of input views.

[3] Yu, A., Ye, V., Tancik, M., & Kanazawa, A. (2021). pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4578-4587).

[4] Xu, H., Chen, A., Chen, Y., Sakaridis, C., Zhang, Y., Pollefeys, M., ... & Yu, F. (2024). Murf: Multi-baseline radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20041-20050).

Thanks for your valuable comments. We agree with you that the analysis of training and inference latency of our proposed method is important. We discuss this topic in our general rebuttal. We will add this discussion on training and inference latency in Section 4.2 (line 190) as you suggested.

Thanks for your valuable comments. Below we address specific questions.

1. The selection of input views

As the number of input views increases, they are sampled from larger regions of the scene. Thank you for point out this question. We will clarify this setting in Section 4.1 (line 157) in our final version.

2. Large-scale scenes

We further evaluate our method on longer videos sampled from large-scale scenes in RealEstate10K. The visualization results are shown in Figure 1 in the one-page pdf. Our method can reconstruct large-scale outdoor scenes and render better novel views than previous methods, benefiting from multiple input views. We also validate our method on indoor full room scenes. With 16 images capturing a room, we reconstruct the full room reasonably, while pixelSplat and MVSplat still suffer from the redundancy of Gaussians, due to the fact that the large-scale scenes require more input views. We report the quantitative results of each scene in the following table for detailed comparison.

Table 1: Quantitative comparison of different methods on large-scale scenes.

3. Release of the source code

We will release our paper on arXiv and the source code on github if our paper is accepted.

4. Questions about the different number of input views in Table 1 and Figure 4

For results of different number of input views, they are obtained with a single model. In our opinion, it is convenient and flexible for users to use one model to process different number of input views. The results also demonstrate that our model has the capability to reconstruct scenes with different number of input views.

5. Efficiency analysis

We agree with you that the analysis of training and inference latency of our proposed method is important. We discuss this topic in our general rebuttal. We will add this discussion on training and inference latency in Section 4.2 (line 190) as you suggested.

Thanks for your valuable comments. Below we address specific questions.

1. Training and inference latency

We agree with you that the analysis of training and inference latency of our proposed method is important. We discuss this topic in our general rebuttal. We will add this discussion on training and inference latency in Section 4.2 (line 190) as you suggested.

2. Inspiration from point cloud processing community

Your suggestion does inspire us a lot. We further add a point cloud processing part from Point Transformer V3 to extract Gaussian features (Model A). As shown in Table 1, such design has improvement on performance. Due to the limitation of rebuttal time, we adopt a simple design, but this design still boost the performance. We argue that this is a promising direction, and we will consider this suggestion as one of our future works to further benefit from the progress in point cloud processing area.

Table 1: Results of PSNR on RealEstate10K benchmark.

Thank you, Reviewer sL12, for your insightful comments. I completely agree with your points and would like to echo your thoughts. Indeed, I also feel that combining point cloud processing with Gaussian splatting has strong potential, and I am excited to see what our community will develop in this direction :)

Best Regards,

Reviewer LnYN

Thanks for the rebuttal. My concerns are well addressed. Thus, I keep my original rate to accept this paper.

BTW: I am also considering the possibility of combining point cloud processing with Gaussian splatting. We have spent many years exploring how to handle unstructured data effectively and efficiently, and it's encouraging to see that authors and reviewer LnYN are also interested in this potential.

Thanks the authors for the detailed response. Could the authors share more details about how the image encoder accepts more input views since I guess the multi-view epipolar attention in both pixelSplat and MVSplat would cost a lot of time and memory when the number of views increase? Did the author perform standard multi-view epipolar attention (for example, pair-wise) or is there any modification made here? Thanks.

Thanks for your comments. We are sorry to confuse you due to the unclear markdown table. Below we address specific questions.

Q: Is the U-Net used for depth estimation from the cost volume considered in the inference time?

A: Yes. In the previous table, the time taken by the U-Net used for depth estimation is added to the inference time. Both the cost volume refinement and the depth refinement parts employ 2D U-Net. We report their inference time in our new table separately.

Q: Why does the cost volume take much more inference time than expected?

A: The cost volume construction contains two parts: (a) prepare projected data with respect to Eq. (2) in MVSplat, (b) compute the correlations. The part (a) dominates in the time consumption, since it computes projected features for each image pair and each depth candidates, leading to high inference time. The inference time cost by computing the correlations is not the same as the inference time of cost volume as we reported in the previous table. Actually, the inference time of computing correlations is much less than the inference time of part (a).

Q: Does the image encoder refers to everything before GGN?

A: It seems that the markdown table is not clear enough. $\Phi_{image}$ in Eq. (1) of GGN refers to multi-view feature extraction, cost volume construction and cost volume refinement. $\Phi_{depth}$ in Eq. (2) of GGN refers to depth estimation and depth refinement. We report the inference time of all these components in the following table. We will further clarify this part in our final version (line 113).