Thank you for your valuable comments. Below we address specific questions.

1. Training Setting and Result on 2 Views

Thank you for your valuable question. You are correct that there are differences in the training configurations between pixelSplat, MVSplat, and PixelGaussian in the current results. Therefore, we retrain pixelSplat, MVSplat and PixelGaussian on RealEstate10K dataset using the same setting to ensure a fair comparison. The results of common cases and challenging cases on RealEstate10K dataset are listed in the Table 1 below and in our the general rebuttal, respectively. Additionally, we offer the detailed training cofiguration for all the models in Table 2 for your reference.

Table 1: PSNR comparison of multiple input view settings on RealEstate10K validation dataset

Table 2: Detailed training configuration of pixelSplat, MVSplat and PixelGaussian

From the results, we do observe that MVSplat slightly outperforms PixelGaussian in the 2-view input setting. We have corrected and clarified this in our revised version. However, as the number of input views increases, PixelGaussian still demonstrates a significant advantage due to the Gaussian overlap and redundancy can be effectively mitigated through our adaptive refinement process. We appreciate it that you point out this issue.

2. Inference Efficiency

Thank you for your comment. As illustrated in Table 3 in our origin paper, our PixelGaussian requires higher inference latency and memory due to the extra CGA and IGR blocks, while can achieve a higher rendering FPS since the quantity of Gaussians is smaller. We further conduct experiments to analysis the training efficiency, inference efficiency and rendering FPS of our network with multiple input view and resolution settings. We use a 48G A6000 and set the batch size as 1 uniformly.

Table 3: Training FPS

Table 4: Inference atency (ms) on multiple view settings

Table 5: Inference latency (ms) and on multiple resolution settings with 2 input views

Table 6: Rendering FPS on multiple view settings

Table 7: Rendering FPS and on multiple resolution settings with 4 input views

As illustrated in Table 4 and 5, our proposed PixelGaussian does require higher overall latency for training and inference compared to MVSplat due to the extra blocks and we also acknowledge that efficiency is not a primary contribution of this paper when considering both 3D reconstruction and NVS. However, Tables 6 and 7 demonstrate the significant advantage of PixelGaussian on rendering at higher resolutions and with increasing input views. This is particularly important when rendering a large number of novel views, and it even helps mitigate the weakness of inference efficiency. Additionally, the adaptation and refinement in PixelGaussian can effectively address the issues of Gaussian overlap and redundancy in both pixelSplat and MVSplat. Thank you for your insightful suggestion and we have added this explanation in our revised paper.

3. Model Performance as Input View Increases

The performance saturation you mentioned is quite insightful. In our general rebuttal, we have conducted experiments with additional view settings across multiple datasets. Due to the memory limitation, we are only able to inference our model with a maximum of 16 views as input. From the results, we do observe that as the number of input views increases, the performance gains gradually diminish, which suggests the presence of performance saturation. Therefore, it is crucial to find a balance between the increasing computational overhead and the marginal performance improvements in practical applications.

Reviewer HfoT

Thank you for your valuable comments. Below we address specific questions.

1. View Setting for Training

We agree with you that training baseline models on different view settings could potentially lead to better results. As you suggested, we are still training the model using 4 and 8 views as input. However, we argue that it is more user-friendly to use a single model regardless the number of input views. This is why our goal is to achieve generalizable reconstruction from arbitrary views.

2. Results on challenging scenarios

We agree that most scenes have substantial FOV overlap. To demonstrate the performance of our model on NVS across a wider range of scenes, we conduct experiments on challenging cases in both the RealEstate10K and DTU datasets in our general rebuttal. As you suggested, we have included these experiments in our revised paper.

3. View Sampling

All the novel views are sampled and interpolated within the range of reference views. As the number of input view increases, the reference views cover a wider range of FoV. For the hard cases illustrated in the general rebuttal, we sample the reference views to cover as wide a view range as possible.

4. Generalization on Large-scale and Wide-range scenes

Your suggestion is quite insightful. As we point out in our Discussion section, the introduction of generative model can address unseen areas of the reference views, enabling NVS for extrapolated perspectives. Additionally, diffusion-based methods can leverage the strong diffusion priors and achieve a boost in NVS for interpolated views (such as ReConX can achieve a PSNR of 28.31 on the RealEstate10K validation set). Since we are not able to access the code of ReConX and ReconFusion, we only compare our PixelGaussian with the NVS results from the video diffusion model in ViewCrafter on 5 selected scenes in the RealEstate10K validation set.

We see that with the strong diffusion priors, ViewCrafter achieves better performance on extrapolated view settings. Your valuable insight inspires us to combine our adaptive Gaussian pipeline with video diffusion models, which may potentially lead to remarkable results on reconstruction and NVS for large-scale and wide-range scenes.

Thank you for your insightful comments. Below we address specific questions.

Comparison with NeRF-based Models

Thank you for your comment. We provide a more details comparsions between NeRF-based methods and our PixelGaussian in the following table. All experiments are conducted on the challenging cases on RealEstate10K as described in our general rebuttal with the batch size of 1 on a single A6000 GPU.

Table 1: Detailed comparisons on PSNR and inference time with NeRF-based methods.

As shown in the results, NeRF-based methods do not experience the Gaussian overlap and redundancy issues as it is in pixelSplat and MVSplat, which explains why they can slightly benefit from an increase in input views similar to PixelGaussian. However, thanks to the efficient rasterization-based rendering of Gaussians, PixelGaussian demonstrates superior computational efficiency compared to NeRF-based methods.

More comparisons on Multiple Datasets and Settings

Thanks for the suggestions. We have conducted experiments on challenging cases with multiple view settings across a wider range of scenarios in our general rebuttal, which have demonstrated that our PixelGaussian can consistently benefit from the increase of input views across all experimental settings. We have included these results in our revised paper.

Training and Inference Efficiency

We agree with you that additional comparisons on training and inference efficiency are essential. We further conduct experiments to analysis the training and inference efficiency of our network with multiple input view settings. We use a 48G A6000 and set the batch size as 1 uniformly.

Table 1: Training FPS

Table 2: Inference Latency (ms) on multiple view settings

Table 3: Rendering FPS on multiple view settings

Our proposed PixelGaussian does require higher overall latency for training and inference compared to MVSplat due to the extra blocks. However, inspired by reviewer X5K2, we have found that PixelGaussian achieves significantly higher rendering speeds as the number and resolution of input views increase. This becomes particularly valuable when rendering a large number of novel views, and it can make up to the limitations of our inference speed.

Figure

Thank you for your advice. We have revised the figure you mentioned and added both depth and point cloud visualization to our revised paper.

From my perspective,

(1) The generalizable 3D-GS setting is more plausible for few views (e.g. 2 views in PixelSplat/MVSplat). I'm not fully convinced by the point of "main focus is on scenarios when more views are provided" in the feed-forward setting. If you are feeding in more views as input, why not directly compare with per-scene optimized 3D-GS methods? These methods could likely achieve better PSNR, albeit at the cost of longer training times. The paper primarily proposes the CGA and IGR modules to reduce GS redundancy, but they do not provide sufficient evidence of a performance gain ("not significantly slower than MVSplat") in rendering FPS for the crucial settings with fewer views (<4, such as 2 or 3).

(2) I acknowledge that "DTU is known for its artificial setup", so why not consider adopting my suggestion to evaluate more real-world-like datasets, such as the "Mip-NeRF 360 dataset," or "even using self-captured real-world inputs"? I suggest the Mip-NeRF 360 dataset because I’d like to see how the model performs on less forward-facing inputs when generalizing. As for the self-captured real-world inputs, I’m interested in seeing how the model applies to real-world scenarios (please refer to the DepthSplat project page for some examples).

(3) I understand that rendering FPS is much more important than "inference latency", but since you put the latency comparison in the main paper, it naturally raises my concern regarding the effectiveness of architecture design. For rendering FPS, please see (1).

Dear Reviewer z7ms,

Thank you for the detailed reviews and comments. While I agree with some of your points, I'm wondering why would it be so critical not to introduce additional latency compared to MVSplat? As far as I understood, the inference speed should not be of much importance in this work, given that under 4 view settings, it is not significantly slower than MVSplat, and the main focus is on scenarios when more views are provided. Also, DTU is known for its artificial setup and far from real world scenarios, which makes it hard to conclude that the method does not generalize well. It would be better to decide whether the method generalized well or not by looking at datasets like DL3DV.

The performance gain when N>5 shouldn't be a bonus since it is shown by PixelGaussian that other methods clearly struggle with more views.

I have reviewed all the comments and replies from the other reviewers. For me, the main issue is as follows:

The authors propose the CGA and IGR modules to reduce 3D-GS redundancy, but these additions introduce extra overhead, leading to higher inference latency compared to MVSplat. Furthermore, in the test setting with fewer than 4 views, the rendering FPS of PixelGaussian is also similar or perhaps even worse than MVSplat, which is a more critical scenario than dense-view NVS. These points suggest that the design of the CGA and IGR modules is not sufficiently effective and requires further elaboration.

Considering these concerns and the rebuttal's inability to fully address them, I am finalizing my score at 5, marginally below the acceptance threshold, with confidence 5.

(Extra message)

1. The term 'PixelGaussian' and 'AdaptiveGaussian' mix together (e.g., Table 3 and Table 4), which is a cheap mistake.
2. Do not leave the reply on the last minutes/days even if you are occupied with CVPR submission in that period.

Thanks for the authors' response, however

1. It seems my question regarding W1 hasn’t been fully addressed, as the comparison of training efficiency and memory usage has not been presented quantitatively. I would like a more comprehensive comparison, as the performance improvement is not particularly evident.

2. Regarding the additional experiments on DTU, why do all the methods perform poorly? I believe that a visual quality below 15 PSNR indicates significant issues. This suggests that the method does not generalize well to different data settings. Additionally, why was evaluation not performed on MipNeRF-360? I think this wouldn’t require extra training, just an adjustment of the camera intrinsics/extrinsic if the aspect ratio is incorrect. Was the performance of this dataset not satisfactory?

3. Regarding the comparison of Training FPS/Inference Latency, I understand that the additional CGA and IGR modules introduce extra overhead. However, I’m still disappointed that PixelGaussian performs similarly or perhaps even worse than MVSplat in rendering FPS when using fewer than 4 views. As we know, sparse views are typically analyzed with 5 or fewer views, so the performance difference in denser-view settings should be considered a bonus rather than a main focus. Moreover, the fact that PixelGaussian lags behind MVSplat in latency speed is concerning.

4. Figure 5 still doesn’t convincingly demonstrate the performance gain and is not beautiful in terms of ICLR-level presentation.

(1) I’m not demanding that you compare with per-scene optimized 3D-GS methods; I simply use this as an illustration to show that emphasizing more dense-view input in feed-forward 3D-GS is not inappropriate.

(2) I still believe that '12-view' is too large for the input in a sparse-view setting. While I’m confident that dense-view inputs can support the generalizable properties of the proposed method, they should not be the main selling point, as sparse-view inputs are more critical in this context.

(3) I disagree with the idea that “2-view settings are less practical”. In fact, exploring a one-shot setting could be both more interesting and more challenging. Practically speaking, if you were to take pictures with your phone for 3D scene reconstruction, it would be more convenient to take just 1-3 shots from different viewpoints. Dynamic factors like people moving or changing environmental lighting could introduce interference, and feeding in additional views might worsen inconsistencies.

Since you have your own perspective on this, I’ll leave it to the AC to make the final judgment. Thanks for the discussion.

Thank you for the quick reply! I agree with your points 2 and 3. However, just one last comment I would like to make. "These methods could likely achieve better PSNR, albeit at the cost of longer training times", the cost of longer training times is a significant disadvantage in many scenarios and I think it would not be ideal to suggest to compare with per-scene optimization approaches. N<12 views are still considered sparse views and typically, the 2 view settings of Pixelsplat or MVSplat would be considered less practical, as 2 view settings is the most basic multi view geometry setting.

I believe that since the paper attempts to address the apparent limitations of existing pixel-aligned Gaussians (redundancy), we should focus on this aspect rather than putting much focus on the performance directly compared to per scene optimizations or SOTA.

Nevertheless, I agree with your other points, which I appreciate your time for the discussion with me.

Thank you for your valuable comments and suggestions. Below we address specific questions.

1. Visualization of Depthmap and Point Cloud

We agree that additional visualizations are essential. We have included more comparisons of depth maps and point clouds, specifically using 4, 8, and 16 views as inputs in our revised paper. Thank you for your valuable suggestion.

2. More Input Views, Datasets and Cross-dataset Generalization

Thank you for the suggestion. We agree it is important to access more challenging view settings and perform cross-dataset generalization experiments. The results are listed in Table 1 and 2 in our general rebuttal. We have included these experiments in our revised paper.

3. Motivation of Deformable Attention

Thank you for pointing this out. Since the Gaussian predictions in our method are not strictly pixel-aligned, the projection of the Gaussian center may not precisely correspond to the specific location on the feature map. Therefore, we incorporate deformable attention into our CGA and IGR blocks, which enables more flexible and adaptive interactions to ensure each Gaussian can share information with the exact location in the score maps and feature maps. To further demonstrate our design, we conduct a simply ablation with 4 view inputs on the RealEstate10K dataset. When a rigid perception field is used for each Gaussian query in the CGA and IGR blocks, we observe an average performance drop of 1.58 in PSNR. A brief explanation of this issue have been added in our revised paper as you suggested.

Table 1: PSNR comparison for ablations on deformable attention

4. Paper Title

Thank you for the advice and we agree that PixelGaussian may lead to misunderstanding that our model predict pixel-aligned Gaussian representations. Therefore, we have changed the title to AdaptiveGaussian as you suggested.