Thank you very much for your thoughtful comments on our work. We greatly appreciate your feedback. We will address each of your concerns individually as outlined below. For those that require additional experiments, we will ensure to upload the results as soon as possible. Should you have any further questions or concerns, please do not hesitate to reach out to us.

For Weaknesses1:

Thanks a lot for your so detailed writting comment. Sorry for causing the bad reading experience, we explained the confusing sentences in the following and revised our paper to reduce unclear expressions.

L040 "view inconsistency":

View inconsistency means that 3D reconstructions from various input views are misaligned because of the inaccurate depth prediction from single view separately, which can be clearly illustrated by the attached video (inconsistentpc.mp4). Another example is that, in Figure 1, the handle of the pot generated from input image 1 appears at a different position compared to the handle from input image 2. This difference arises also due to inaccurate depth predictions of each view, leading to spatial variations and resulting in multiple handles being rendered in the views. We have added this explanation in our paper.

L044 view-specific camera space:

In the corresponding sentence, we want to point out that, for methods like Splatter Image [1] and LGM [3], the 3D Gaussians for each view are first reconstructed in the camera space of the corresponding input view, then they are merged in the world space after the camera-to-world space transformation. Therefore, in this context, "view-specific" is to stress that they predict 3D Gaussians in a view-independent manner. We have modified this statement in the revised version.

L045 "These Gaussians are then converted to world space":

For methods Splatter Image [1] and LGM [2], the 3D Gaussians are first reconstructed in the camera space of each input view, then they are merged in the world space after the camera-to-world space transformation. Therefore, the step of transforming 3D Gaussians from the camera space to the world space cannot be ignored, and it is the main cause of the view inconsistent problem.

L049 The authors should make sure that the general concepts and ideas are understandable just from reading the text they provide:

Thank you for your suggestions, we will modify these in the revised version. DETR-like models link object bounding box as queries and treat image tokens as keys and values in Transformer, which have made a great success. We borrow the similar philosophy that links each 3D Gaussian with queries and also treat image tokens as keys and values, then refine 3D Gaussians iteratively. % We will use more widely accepted paper and explain DETR briefly.

L070 camera modulation:

Sorry for the bad reading experience again, we have modified the paper following your suggestions. To be clear, camera modulation means we linearly transform image features of each view, say $F' = WF + b$, where $F$ is the original image feature, $W$ and $b$ are weights and bias regressed by an MLP with camera parameters as input. Such operation gives each view its corresponding camera pose information. We have added a brief explanation in the introduction.

L074 "multi-view distinctions:

The "multi-view distinction" means that we use camera modulation to distinct queries before projecting them to each view to retrieve image features, so as that queries can be aware of different camera poses.

L140 "In total" This is not needed:

Thanks for pointing it out, we will delete it.

L151 unitary 3D Gaussians representation: Also, typo: "3D Gaussians representation" -> "3D Gaussian representation".

Unitary 3D Gaussian representation means we define a unique set of 3D Gaussians in the world space no matter how many input views are given. By contrast, previous methods predict one set of 3D Gaussians in camera space for each input view, so there will be multiple sets of 3D Gaussians given multiple input views, and then merge them together in the world space to get the final output 3D Gaussians. We will fix the typo.

L158 "Spatially efficient self-attention":

Thanks for pointing this out, we miss the reference to Section 3.2.3 and will add it in the revised version. Spatially efficient self-attention means to do self-attention in a memory efficient way by sampling part of 3D Gaussians as keys and values, and we detailed this part in the Section 3.2.3.

[1] Stanislaw Szymanowicz, et al., Splatter Image: Ultra-Fast Single-View 3D Reconstruction.CVPR2024.

[2] Zi-Xin Zou, et al., Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers. CVPR2024.

[3] Jiaxiang Tang, et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV2024.

For Weaknesses2:

We agree that "3D recontruction" in earlier literature usually include estimating camera parameters, but here we use "3D reconstruction" to denote the reconstruction of 3D Gaussians following recent 3D GS papers Splatter Image[1], Triplane-Gaussian[2]. We also discussed the camera pose problem in the limitation part in our paper.

For Weaknesses3:

Thank you for pointing this out. We will add the comparison as soon as possible.

[1] Stanislaw Szymanowicz, et al., Splatter Image: Ultra-Fast Single-View 3D Reconstruction.CVPR2024.

[2] Zi-Xin Zou, et al., Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers. CVPR2024.

[3] Jiaxiang Tang, et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV2024.

Thank you very much for your thoughtful comments on our work. We greatly appreciate your feedback. We summaries the main concerns in weaknesses1 and 2 and response them in the following. We agree that these related methods should be mentioned in our paper to position our paper in the field of 3D reconstruction. We discuss each method below and have added them in the related work part in the revised PDF. We will add the quantitative comparison with those that open-sourced codes are available as soon as possible. Should you have any further questions or concerns, please do not hesitate to reach out to us.

Weakness

We summaries the main concerns in weaknesses1 and 2 and response them in the following. For GS-LRM and GRM, by merely reading their papers, both of them claim that they use a scalable large transformer with the similar pixel-aligned structure and are trained with many A100 GPUs but it is not clear what are the key components that has made their model work well. Moreover, the view inconsistent problem theoretically exists in all those methods that first predict 3D Gaussians in each camera space and then naively merge them in world space as the depth of the predicted 3D Gaussians in each view would always have errors, which will inevitably lead to the misaligned 3D Gaussians merged in the world space. The problem is clearly illustrated in the attached video inconsistentpc.mp4.

However, we are not able to fairly compare our method with them as they have not released the codes. And their quantitative results are computed not following the same dataset or settings with previous methods, making them hard to be fairly compared quantitatively. For MVSplat and pixelSplat, we are now working on comparing them and will release the comparison results as soon as possible. For One-2-3-45++, it is a generative model, suffering longer inference time that take 20 seconds to 1 minute for one generation while fast-forward methods only takes around 1 second. As for Mesh-LRM and MeshFormer, they use representations like NeRF and voxel, which have disadvantages in the aspect of computational efficiency, especially for rendering. Moreover, MeshFormer requires 3D supervision while Mesh-LRM utilize triplane, who compresses 3D space, leading to a lack of detailed information in the 3D structure and imposing a rigid grid alignment that limits flexibility ([1][2]). TripoSR is a single image 3D reconstruction model, which uses an encoder-decoder structure with triplane decoder and triplane-based NeRF, and it also has the limitations of NeRF and Triplane as discussed earlier. Its inference time is 3.29s for a single forward process (without mesh extracting) and rendering time for 22.73s, which is much slower than 3D GS-based models. We added the discussion of the mentioned methods in the related work part of the revised version and will add the comparison with MVSplat and pixelSplat as soon as possible.

Questions1:

The main contribution of our method is proposing a new unitary 3D Gaussian modeling approach for multi-view 3D reconstruction that can avoid the view inconsistent problem. This new design can also be adopted by other mentioned 3D GS-based methods. We will discuss more about the mentioned prior works in the revised PDF.

[1] Jiaxiang Tang, et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV2024.

[2] Charles R Qi, et al., PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In CVPR 2017a

Results of TrioSR on GSO dataset for single view situation

We test TripoSR and Triplane-Gaussian on the single image reconstruction setting with the checkpoint they provide on github. As shown in the following table, our model surpass the previous methods on both the performance and the inference speed. In terms of rendering speed, both Triplane-Gaussian and our model employ Gaussian Splatting, known for its fast rendering speed. Conversely, TripoSR utilizes NeRF, a method slower in comparison to Gaussian Splatting.

Table 1: Quantitative results trained on Objaverse LVIS and tested on GSO. 3D sup. means need 3D supervision.

We have updated this table as Table 5 in the revised version of the PDF.

I appreciate the authors' response and the additional experiments provided.

However, I'm not convinced that "the view inconsistency problem theoretically exists" in all per-pixel Gaussian prediction methods. While it is true that "the depth of the predicted 3D Gaussians in each view would always have errors," the authors’ method will also introduce errors, albeit not view-aligned. In general, those per-view Gaussian methods are designed to learn to aggregate multi-view Gaussians in an end-to-end manner, which learns to fuse the depth and achieve consistency. Theoretically, as these methods minimize errors during training, their consistency and quality improve. In essence, both per-view Gaussian methods and the proposed unitary method rely on similar processes of network optimization and gradient descent to achieve consistency and rendering quality. Of course, the proposed method doesn't have a concept of "view-"consistent depth, but it has 3D errors of point locations, which might also lead to inconsistent renderings across novel viewpoints. So I cannot agree that this is a theoretical limitation.

Regarding the inconsistencies shown in inconsistentpc.mp4 and other supplementary examples, I revisited the GRM and GS-LRM websites to verify their results. GS-LRM, for instance, provides a lot of ply files of their GS reconstruction shown in an interactive 3DGS viewer. I reviewed several examples in their viewer and also downloaded multiple ply files to inspect in Meshlab, finding their results to be of high quality and very consistent. I did not observe a similar level of inconsistencies shown in the LGS examples in the supp.

Overall, I do not believe the inconsistency issue is a theoretical problem inherent to all per-view Gaussian methods. The evidence suggests that this issue may be unique to LGM or other baselines. To clarify, I do not hold a strong bias toward per-view Gaussian methods, and I am also disappointed about papers like GS-LRM not releasing their code to facilitate comparisons. I also really appreciate the authors' efforts to explore unitary GS prediction, but my main concern is the potential to mislead the community here. The paper currently is written like per-view Gaussian methods are inherently worse, which I do not see sufficient evidence to support. If the inconsistency problem is specific to LGM or other baselines, this should be clearly stated, and the issue should not be generalized to all per-view Gaussian methods.

On the other hand, the new single-image reconstruction results are very impressive. However, I am surprised by the reported PSNR values, as they seem unusually high. To be more specific, I checked the numbers in the recent NeurIPS paper MeshFormer; the SOTA single-view reconstruction methods achieve PSNRs only around 21 but their quality already looks very good. Typically, the PSNR for single-image reconstruction is low because the unseen back side of an object introduces significant uncertainty, making it theoretically impossible to recover accurately. Deterministic models often produce blurry back-side renderings, while probabilistic models, such as diffusion models, tend to generate sharp but potentially mismatched back-side renderings. I personally even feel PSNR and other rendering metrics are not the best choice to evaluate single image reconstruction techniques because this is more of a generative task. But why can the proposed method lead to such a high PSNR over 23? Could the authors clarify how this experiment was conducted? Specifically, details on the resolution, training/testing view settings, and other relevant parameters would be helpful. Additionally, providing more visual examples, including input views and 360-degree renderings, would help illustrate what is happening in the reconstructions.

Similar to LGM, both pixelSplat [1] and MVSplat [2] follow a workflow that regress Gaussians from each view within the respective camera spaces and subsequently merge them in the world space. In pixelSplat, the integration of cross-view-aware features is through an epipolar Transformer, and it still suffers from inaccurate depth estimation. MVSplat adopts a design that incorporates a cost volume storing cross-view feature similarities for all possible depth and makes a more accurate depth prediction. However, they assign each pixel with a 3D Gaussian and thereby generates a planar representation rather than the object itself. In addition, MVSplat tends to obscure object details due to the occlusion by 3D Gaussians from other viewpoints, resulting in suboptimal outcomes. To address this issue, we mask the 3D Gaussians on background pixels to help it focus on rendering 3D Gaussians contributing to the object itself, noted as 'MVSplat (masked)' in the results.

We present the comparison to pixelSplat [1] and MVSplat [2] in Appendix A.5 and the quantitative results on the GSO-random dataset is shown in Table 12. From the table, we can see that their results is significantly worse than ours. It is probably due to the fact that they have only been trained on the scene reconstruction dataset RealEstate10 [3], which only contains small camera difference among views. The cameras of object reconstruction dataset GSO-random has larger variations, so we observe more severe misaligned 3D Gaussians (view inconsistency) from different input views for MVSplat, as shown in the visualized results in Figure 21 (we also add the corresponding videos and ply files in the MVSplat_results folder of supplementary materials). And we find that MVSplat cannot correctly predict the back side of the object. It is not a big issue for scene reconstruction as their camera only moves a little, but would lead to incomplete reconstruction of objects. In the figure, we present the centers of 3D Gaussians generated from different views with different colors and the novel views are rendered from the 3D Gaussians from all views. As for pixelSplat, it almost cannot output reasonable results when use GSO-random dataset for testing, so we have not presented their visualized results. We provide the content of Table 12 as following:

Table: Comparison with MVSplat and pixelSplat on the GSO-random dataset

[1] David Charatan, et al., pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. CVPR2024.

[2] Chen, MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images. ECCV2025

[3] Zhou, et al., Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPH), 2018

For Questions1:

Thank you for pointing it out. Actually, the feature extractor for both stages will output feature maps first, then for the coarse stage, a convolution layer (omitted in the figure) is used to regress pixel-aligned 3D Gaussians as coarse initialization. We have modified the figure to make it clear in the revised PDF.

For Questions2:

The function of coarse initialization is mainly to avoid out-of-boundary projected points, thus minor variations will not make a big difference for the refinement stage. In the coarse stage, no matter which view is selected, the 3D Gaussians are first reconstructed in its camera space, and then transformed to the world space (the camera space of the first view) using camera pose parameters. Therefore, the concept of "front view" does not exist in this context. No matter which view is selected to be the input of the coarse stage, it will not make a big difference. We also add the ablation study on the number of images used during the coarse stage. As shown in Table 9 of the revised PDF, the number of images used during the coarse stage does not influence the final result. We also show the content of Table 9 here.

Table: Ablation study results of different views and different numbers of views for the coarse stage (with 4 views in the refinement stage)

For Questions3:

The input images (1 or 2 views) for the coarse stage are sampled from the all multi-view input images, and the 3D Gaussians are first reconstructed in the camera space and then transformed to the world space (the camera space of the first input view). The 3D Gaussians in the same world space will be the initialization of the refinement stage, which are already coarsely aligned with the other input views. Their final positions and other parameters will be iteratively refined to align all input views in the refinement stage.

[1] Jiaxiang Tang, et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV2024.

[2] Charles R Qi, et al., PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. CVPR2017a.

[3] Stanislaw Szymanowicz, et al., Splatter Image: Ultra-Fast Single-View 3D Reconstruction. CVPR2024.

Thank you very much for your thoughtful comments on our work. We greatly appreciate your feedback. We will address each of your concerns individually as outlined below. For those that require additional experiments, we will ensure to upload the results as soon as possible. Should you have any further questions or concerns, please do not hesitate to reach out to us.

For Weaknesses1:

Our method differs from Triplane-Gaussian in the following aspects. First, Triplane-Gaussian is a single-view reconstruction method so it does not need to consider the multi-view information fusion problem while our method target at reconstructing a unitary set of 3D Gaussians with arbitrary number of input views, resolving the view inconsistent problem that only happens in the multi-view input setting. Second, Triplane-Gaussian adopt the triplane representation, which would lead to a lack of detailed information in the 3D structure and imposing a rigid grid alignment that limits flexibility (LGM[1], PointNet[2]). Third, Triplane-Gaussian requires 3D supervision to achieve a good performance while our method only requires multi-view 2D images. Even so, our method still outperforms the Triplane-Gaussian given the single-view image as input, as shown in Table 5 in the revised version. We also provide the table here.

Table: Quantitative results trained on Objaverse LVIS and tested on GSO. 3D sup. means need 3D supervision.

As for Instant3D, it also utilize the triplane and apply the NeRF to represent the reconstructed object. The NeRF representation is fundamentally different from 3D Gaussians and cannot be naively replaced. Therefore, our method is essentially different from the mentioned methods. MVDFA is designed to to save memory occupation and training and inference time so that we can set a larger number of 3D Gaussians to represent an object. The two-stage framework is also important because we empirically found that a good initialization for 3D Gaussians would make a big difference, as shown in Table 4 and Table 10 in our paper.

For Weaknesses2:

As shown in the Figure 7 in the revised PDF, we visualize the predicted 3D Gaussians centers of LGM and Splatter Image and paint them of different views with different colors. From the figure, the misalignment of 3D Gaussians from different views can be obviously seen, which is what we call "view inconsistency", while our method share a unitary set of 3D Gaussians for each view so there is no such severe view inconsistent problem. Moreover, in Figure 4, we also shows the view inconsistency example of LGM that predict not aligned objects from different views.

When masks are applied to remove backgrounds, as shown in Figure 16 and the attached vedio (masked_point_cloud.mp4), the misaligned 3D Gaussians can be alleviated to some extent but the problem still exists. The corresponding quantitative results are presented in Table 6. Removing background points lead to less outliers and better rendering results but there still are obvious artifacts that can be observed. The Table is also given here:

Table: Comparison between masked and original pixel-aligned methods

For Weaknesses3:

The coarse stage share a similar structure with the previous method (Splatter Image [3]) whose function is to provide a coarse initialization for the refinement stage to avoid hard convergence problem. The results in Table 2 and Table 4 use different validation dataset, so their numerical values cannot be directly compared. The visualized comparison of Splatter Image and the coarse stage of our model are shown in Figure 15. From the figure, we can see that the output of the coarse stage is not as good as Splatter Image, but with the refinement stage, the final output outperforms the Splatter Image.

[1] Jiaxiang Tang, et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV2024.

[2] Charles R Qi, et al., PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. CVPR2017a.

[3] Stanislaw Szymanowicz, et al., Splatter Image: Ultra-Fast Single-View 3D Reconstruction. CVPR2024.

Dear Reviewer 9662,

We want to express our sincere gratitude for your insightful suggestions which are instrumental in enhancing the quality of our work. We hope that our proposed modifications would have addressed your concerns about the clarity of our presentation. We would really appreciate it if you could let us know if there are any further questions or aspects of the paper that require additional clarification. Thank you once again for your time and consideration.

Thank you very much for your thoughtful comments on our work. We greatly appreciate your feedback. We will address each of your concerns individually as outlined below. For those that require additional experiments, we will ensure to upload the results as soon as possible. Should you have any further questions or concerns, please do not hesitate to reach out to us.

For Weaknesses1:

Thanks for pointing this out. The blurry are mainly caused by the automatic zoom-in of the PDF editing software as our model is trained with the resolution of 128, so as the rendered output image. Now, we are working on the result with the resolution of 512 to solve this problem.

For Weaknesses2:

Apart from PSNR, other metrics SSIM $\uparrow$ (increased by 0.3) and LPIPS $\downarrow$ (reduced by 0.1739) are also significantly improved. We provide the visualization on Splatter Image and our method in Figure 15 in the revised version. Although the improvement for PSNR is not significant, the visualization of our model is much better than Splatter Image. (The visualizations for other methods are in Figure 10). Moreover, previous novel view synthesis papers like MVGamba[1], Splatter Image[2], Instantmesh[3], pixelsplat[4] also provide PSNR $\uparrow$ improvement around 0.5dB, so it is not a marginal increase.

For Questions1:

We give more view result in Appendix Figure 17 in the revised PDF. Our model is positioned on the 'sparse view' setting, which indicates the number of views less then 10, so we only reports the performance of views from 2 to 8. With the increase of input views, information from similar views becomes redundant, so the gain for our model has become plateaued while other methods suffer from performance drop as they cannot handle too many input views due to the view inconsistent problem. As we keep increasing the number of input views larger than 8, our method can still benefit from more input views (as shown in Appendix Figure 17) while others meet the CUDA-out-of-memory problem.

For Question2:

Thanks a lot for pointing this out. The setting of random input view is obvious a more challenging task than the setting of fixed input view, thus our method also inevitably suffers from a performance drop but still performs better than other state-of-the-art methods. As for Splatter Image [2], it also meets a significant performance drop when random input views are used as its SSIM $\uparrow$ decreased from 0.9151 to 0.8932 and LPIPS $\downarrow$ increased from 0.1517 to 0.2575 despite its PSNR $\uparrow$ has a slight increase. We visualize the results of the two settings to show the difference in Figure 14 in the revised PDF. Therefore, it does not mean Splatter Image [2] demonstrates superior generalization regarding input view pose distribution, but it appears that the PSNR $\uparrow$ of Splatter Image [2] does not increase when the setting is switched from random input view to fixed input view, which might be caused by some its inherent problems. We add the analysis in our Appendix in the revised version.

[1] Xuanyu Yi, et al., MVGamba: Unify 3D Content Generation as State Space Sequence Modeling. arXiv2024. [2] Stanislaw Szymanowicz, et al., Splatter Image: Ultra-Fast Single-View 3D Reconstruction. CVPR2024. [3] Jiale Xu, et al., InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Mod- els. arXiv2404. [4] David Charatan, et al., pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. CVPR2024. [5] Jiaxiang Tang, et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation. ECCV2024.

Dear Reviewer 4FKX,

We want to express our sincere gratitude for your insightful suggestions which are instrumental in enhancing the quality of our work. We hope that our proposed modifications would have addressed your concerns about the clarity of our presentation. We would really appreciate it if you could let us know if there are any further questions or aspects of the paper that require additional clarification. Thank you once again for your time and consideration.

Dear Reviewer 4FKX,

For weakness1, we now provide the updated visualization with resolution 512 in the revised PDF Appendix Figure 20. Should you have any further questions or concerns, please do not hesitate to reach out to us.

Dear Reviewers,

Many thanks for your reviews on submissions of ICLR 2025. Could you please read the authors' rebuttals and give your replies?

Best wishes,

AC