Geometric Neural Process Fields

Q4. Compared with SOTA methods.

We compare GeomNP with GeFu (2024 SOTA) in the 2-view setting in answer 1.

---

Q5. Performance of more views.

We provide more comparisons with recent SOTA, GNT [1], on the Drums class of the NeRF synthetic dataset. GNT is a recent transformer-based generalizable method. Our method can seamlessly extend into it by utilizing its backbone and NeRF architectures. This extension not only provides a consistent baseline but also demonstrates the flexibility and compatibility of our method with existing architectures. As shown in the following table, with more views used (from 1 to 10), our method is able to achieve better performance. Also, we outperform GeFu with 2 views and GNT with 1, 2, and 10 views. This indicates the effectiveness of our method.

To demonstrate the high-quality rendering results, we also provide the video on the Drums class of the NeRF synthetic dataset. Additionally, we provide a qualitative visualization of GeomNP and GNT in the 1-view setting, available here. The visualization demonstrates that GeomNP is able to generate higher-quality results than GNT using only one context view.

[1]. Wang P, Chen X, Chen T, et al. Is Attention All That NeRF Needs?[J]. ICLR, 2023.

---

Thank you for your time and efforts. Please don’t hesitate to reach out if you have further questions or need more information.

We sincerely thank # Reviewer SbEu for their insightful comments. The following addresses their concerns and provides answers to their questions.

Q1. Comparisons with GPF and GeFu

We thank reviewer SbEu for bringing these two interesting works.

Comparison with GPF.

Similarities: Both GPF and GeomNP construct 3D point representations (in either implicit or explicit bases manner) based on features extracted from input views.

Differences: GPF uses PatchmatchMVS to initialize point cloud scaffolds, which are discrete and may cause explicit visibility problems, while GeomNP directly predicts Gaussian bases with centers, anisotropic covariance matrices, and semantic features. Hence, GeomNP only needs 2500 Gaussian bases on the NeRF Synthetic dataset, while GPF uses 50K or more points, while we

Comparison with GeFu.

Similarities: Both GeFu and GeomNP involve accumulating point-wise descriptors into features, which are then decoded into pixel colors.

Difference: GeFu uses cost volumes of multi-view features combined with resampling techniques, while GeomNP constructs continuous Gaussian bases with semantic features and aggregates them using radial basis functions.

We conduct the comparison with GeFu on the Drums class of the NeRF synthetic dataset. We use the score of the 2-view setting from the Gefu paper. The experimental results in the Table below demonstrate that our method outperforms the GeFu.

---

Q2. Discuss which types of geometric descriptors (implicit or explicit) are superior.

The choice between implicit and explicit geometric descriptors largely depends on the specific goals and requirements of the method. Explicit Geometric Descriptors enable explicit incorporation of geometric prior, interpretability, and direct manipulation. However, this may involve multi-stage processing and training (e.g. point cloud scaffolds initialization, explicit visibility computations, and point refinement in GeFu. ). In contrast, our implicit neural geometric bases use a continuous way to represent scenes and are advantageous for capturing fine details as well as representing complex structures without discretization artifacts. Our implicit geometric bases can be trained in an end-to-end manner.

In fact, the two can also be complementary to each other. Our implicit geometric bases adopt Gaussian bases with spatial location. The centers of Gaussian bases can be seen as a set of point masses. In this sense, the explicit geometric descriptors techniques can be used to improve the spatial quality of our bases. However, this is beyond the scope of this paper.

---

Q3. Videos to prove the effectiveness

To demonstrate the high-quality rendering results, we also provide videos on the Drums class of the NeRF synthetic dataset, available here.

---

Algorithm: Inference Procedure

Input: Context set $({\bf{X}}_C, {\bf{Y}}_C)$, target input ${\bf{X}}_T$
Output: Prediction ${\bf{Y}}'_T$

1. Estimate the context bases ${\bf{B}}_C$ (Eq. 12).

2. Estimate the object-specific latent variable ${\bf{z}}_o$ based on the context set:

   $${\bf{z}}_o \sim p({\bf{z}}_o \mid {\bf{X}}_C, {\bf{B}}_C) \quad \text{(Eq. 7)}$$

3. Estimate the ray-specific latent variables ${\bf{z}}_r^{T}$:

   $${\bf{z}}_r^{T} \sim p({\bf{z}}_r^n \mid {\bf{z}}_o, {\bf{x}}_T^{n}, {\bf{B}}_C) \quad \text{(Eq. 8)}$$

4. Modulate the MLP $f$ using the latent variables $\{{\bf{z}}_o, {\bf{z}}_r^{T}\}$ (Eqs. 16 & 17).

5. Render novel views $\hat{\bf{Y}}_T$ using the modulated MLP $f$.

Q3. Is the 2D part pre-trained on a variety of different objects?

No, the 2D part is trained from scratch. We will highlight this detail in our main paper.

---

Q4. Comparison with IBRNet and MVSNeRF on the NeRF synthetic dataset.

Thanks for the suggestion. We have conducted a comprehensive comparison of our method with IBRNet and MVSNeRF on the NeRF synthetic dataset. As shown in the following table, GeomNP outperforms baselines, which indicates the effectiveness of our method.

---

Q5. More multi-view results for qualitative comparisons.

More qualitative comparisons of multi-view results are presented in Fig 15 in the revised paper.

---

Q6. Present individual results for each DTU object.

We present the individual results for each DTU object in the supplementary materials. We also present a video regarding the experiments on the NeRF synthetic dataset, available here.

---

Q7. In Table 4, which subset of Lamps is used?

In order to fast evaluation for the ablation study, we randomly select 50% of data from the Lamps dataset.

---

Thank you for your time and efforts. Please don’t hesitate to reach out if you have further questions or need more information.

We sincerely thank # Reviewer 5vP9 for their insightful comments. The following addresses their concerns and provides answers to their questions.

Q1. Clear clarification about technical terms.

Thank you for your suggestions. The terms “context set” and “target set” originate in neural processing (NP) research.

"modulating a neural network”: Modulating a neural network means dynamically adjusting its parameters or intermediate computations based on the condition.

As our method is based on neural processing, we use this term to keep consistent with NP concepts.

We will add this clarification in our final version of the paper.

Q2. The modulation layer needs a clearer presentation. Pseudocode block to explain the training and inference processes of GeomNP.

Thanks for the suggestion. We have included a pseudocode block to explain the modulation layer in Algorithm 1, section B3 of the revised paper.

Here, we present the pseudocode block to explain the training and inference processes of GeomNP.

Algorithm: Training Procedure

Input: Context set $({\bf{X}}_{C}, {\bf{Y}}_C)$,

and Target set $({\bf{X}}_{T}, {\bf{Y}}_T)$

Output: Prediction ${\bf{Y}}'_T$

1. Estimate the context bases ${\bf{B}}_C$ and the target bases ${\bf{B}}_T$ (Eq. 12).

2. Estimate the object-specific latent variables:

   • For the context set ${\bf{z}}_o^C$: $${\bf{z}}_o^C \sim p({\bf{z}}_o \mid {\bf{X}}_C, {\bf{B}}_C)$$
   • For the target set ${\bf{z}}_o^T$: $${\bf{z}}_o^T \sim q({\bf{z}}_o \mid {\bf{X}}_T, {\bf{B}}_T) \quad \text{(Eq. 7)}$$

3. Estimate the ray-specific latent variables:

   • For the context set ${\bf{z}}_r^{C}$: $${\bf{z}}_r^{C} \sim p({\bf{z}}_r^n \mid {\bf{z}}_o^C, {\bf{x}}_C^{n}, {\bf{B}}_C) \quad \text{(Eq. 8)}$$
   • For the target set ${\bf{z}}_r^{T}$: $${\bf{z}}_r^{T} \sim q({\bf{z}}_r^n \mid {\bf{z}}_o^T, {\bf{x}}_T^{n}, {\bf{B}}_T) \quad \text{(Eq. 8)}$$

4. Modulate MLP $f$ using the target latent variables $\{{\bf{z}}_o^T, {\bf{z}}_r^{T}\}$ (Eqs. 16 & 17).

5. Render novel views ${\bf{Y}}'_T$ using the modulated MLP $f$.

6. Compute losses:

   • Reconstruction loss between predictions and ground truth: $$\mathcal{L}_{\text{recon}} = \text{MSE}({\bf{Y}}'_T, {\bf{Y}}_T)$$
   • Latent variable alignment losses (KL divergence) using context and target latent variables (Eq. 10).

I appreciate the authors’ efforts to address my concerns, particularly regarding the pseudocode block and the comparisons on the NeRF synthetic dataset. However, my primary concerns remain insufficiently addressed.

$**Regarding Q1**$, I acknowledge the authors’ clarification that the terms "context set" and "target set" originate from neural processing (NP) research. However, within the literature on radiance reconstruction/generation from sparse or single views, these terms appear uncommon. In my view, their usage adds unnecessary difficulty to understanding the paper.

$**Regarding the NeRF synthetic dataset comparisons**$, based on my observations, the qualitative results seem inconsistent with the quantitative results. The authors provide qualitative results only for the "Drums" and "Lego" cases. For many views, the qualitative results appear suboptimal. However, it is quite confused that the proposed GeomNP achieves a PSNR of 27.92 on the NeRF synthetic dataset, which is very competitive (compared to NeRF's 31.01). It is worth noting that NeRF's qualitative results are really good as it utilizes 100 views for case training. I believe that case-by-case qualitative and quantitative results are necessary for a more comprehensive evaluation.

$**Regarding the DTU comparisons**$, while the supplementary qualitative results are not competitive, I could not find the corresponding quantitative results.

Overall, I feel that the issue of clarity persists. And it remains challenging for me to draw definitive conclusions about the effectiveness of the proposed method based on the current experimental evidence.

Thank you for your time and feedback.

Regarding Q1. Thanks for the suggestion. We will carefully use the terms "context set" and "target set" in the revised paper to reduce confusion. We would like to also point out that the terms have also been used in the recent NeRF work[1] mentioned by reviewer MgaS. Also, as our method is a generic method for both 3D and 2D signals. In 2D implicit neural fields research (SIREN [2]), the term “context” is also used for partial observations.

Regarding the NeRF synthetic dataset comparisons. Sorry for any confusion regarding the qualitative results. The qualitative results for "Drums" are provided in the following video: https://anonymous.4open.science/api/repo/GeomNP-6D83/file/drums-video.mp4?v=e63d78fa. The Lego results are intended to demonstrate better cross-category performance than the baseline (trained on Drums and tested on the Lego class) as requested by Reviewer 5cgt.

To clarify, we provide a qualitative comparison (view by view) with 1-view and 10-view contexts in the following image: https://anonymous.4open.science/api/repo/GeomNP-6D83/file/nerf-syn-drums-comparsion.pdf?v=725141d8 (PSNR values are also included). The visualization quality is competitive, exceeds the baseline (GNT), and aligns with the PSNR values we reported.

Regarding the DTU comparisons, the visualizations provided for different scenes are based on the integration of our method with pixelNeRF (16.99 for ours vs. 15.80 for pixelNeRF, as reported in Table 2 of the main paper). These results are consistent with the quantitative results we presented. To achieve better performance on the DTU dataset, we plan to apply GNT+GeomNP to the DTU dataset and include the results in the revised paper. Once the new results are available, we will provide an update.

[1] Tewari, Ayush, et al. "Diffusion with forward models: Solving stochastic inverse problems without direct supervision." Advances in Neural Information Processing Systems 36 (2023): 12349-12362.

[2]. Sitzmann, Vincent, et al. "Implicit neural representations with periodic activation functions." Advances in neural information processing systems 33 (2020): 7462-7473.

---

Thank you for your time and efforts. Please don’t hesitate to reach out if you have further questions or need more information.

We sincerely thank # Reviewer MgaS for their insightful comments. The following addresses their concerns and provides answers to their questions.

Q1.1. Cross-category evaluation.

We thank the reviewer for highlighting this new setting. To clarify, our approach follows the standard practice in previous works, which primarily focus on single-category evaluation on ShapeNet, without considering cross-category settings. However, in response to the reviewer’s request, we conducted additional experiments by training our model on the drums class of the NeRF synthetic dataset and evaluating it on the Lego class. In these cross-category evaluations, our method outperforms the baseline GNT [1], achieving superior quantitative results (1.1 PSNR higher) and qualitative improvements (see the visualization in this link).

---

Q1.2. Qualitative results for the hierarchical ablation

Qualitative results for the hierarchical ablation are provided in Fig. 14 in the revised paper. As illustrated in Fig.14, the absence of the global variable prevents the model from accurately predicting the object's outline, whereas the local variable captures fine-grained details. When both global and local variables are incorporated, GeomNP successfully estimates the novel view with high accuracy.

---

Q2. "Key-related work is missing and should be discussed."

Thanks for bringing these key-related works. We have included and discussed them in the revised paper.

DiffRF 2023 integrates the forward model that maps unobserved signals to observations into the denoising step of a diffusion model. The is probabilistic, and it aims to sample from a distribution of signals that are consistent with a set of partial observations. In contrast, our method aims to model the distribution of functions to ensure the rendering function is sample-specific and consistent between rich observation and limited observation.

Tewari et al. 2023 use a diffusion model to generate radiance fields in the format of explicit voxel grids. It requires an initial fit for radiance fields and then performing diffusion. In contrast, our method works on function distribution instead of voxel grids and can be trained end-to-end.

PointNeRF 2022 uses neural 3D point clouds, with associated neural features, while our method constructs continuous 3D Gaussian bases with semantic features.

---

Q3. "The main goal of the paper is not straight."

Thank you for your suggestion. Our method is a generic generalizable INR method and can be applicable to both 3D scene reconstruction and 2D image regression tasks. This is demonstrated by our experiments on two tasks. We will clarify this based on your suggestion.

Q4. The ablation of the geometric bases on higher resolution.

To show the effectiveness of the geometric bases on higher image resolutions, we present the ablation of geometric bases on 178x178 images. The experimental results in the following table demonstrate that using more bases leads to better performance.

---

Q5. The experimental setup for the uncertainty estimation.

To estimate the uncertainty map, we sample ten times from the predicted prior distribution to obtain 10 latent vectors, resulting in ten NeRF functions. By doing so, we obtain ten predictions for this image. Then calculate the variance on each location to estimate the uncertainty map.

---

Q6. Erkoc et al. 2023 is incorrectly classified as a deterministic model.

Thanks for the correction. We have corrected this in the revised paper.

---

Q7. best PSNR in Tab. 4.

Thanks for the suggestion. We have updated Table 4 in the revised paper.

---

Q8 - In L.420: How exactly is the method incorporated into PixelNeRF?

To integrate our method into PixelNeRF, we leverage the same feature extractor and NeRF architecture. Specifically, we utilize a pre-trained ResNet34 to extract features from the observed images. From the latent space of the feature encoder, we predict geometric bases, which are then used to re-represent each 3D point in a higher-dimensional space. These point representations are aggregated into latent variables, which are subsequently used to modulate the first two input MLP layers of PixelNeRF's NeRF network. During training, we conduct alignment between the latent variables derived from the context images and the target views. We have included the details in the revised paper.

[1]. Wang P, Chen X, Chen T, et al. “Is Attention All That NeRF Needs?”[J]. ICLR, 2023.

---

Thank you for your time and efforts. Please don’t hesitate to reach out if you have further questions.

We sincerely thank # Reviewer g7b2 for their insightful comments. The following addresses their concerns and provides answers to their questions.

1.Comparison with other baselines on popular 3D datasets.

Thanks for pointing out these baselines. We have cited them and added related experimental results and discussions in our updated paper.

We provide two more comparisons with recent SOTA (GNT and splatter image) on the Drums class of the NeRF synthetic dataset and the hydrants class of the CO3D dataset. Specifically, as our method is a generic method, we integrate our method (Neural Processing and Geometirc bases) with the two baselines for fair comparison purposes.

We conducted experiments on the Drums class of the NeRF synthetic dataset, evaluating both 1-view and 10-view scenarios. The results, presented in the table below, show that our method is advantageous in settings with extremely limited context views (1-view), improving GNT by about 2.7 PSNR. The qualitative comparison (1-view) is given in this link.

For Splatter-image, we perform neural processing (predicting latent variables) in the latent space of the U-Net used in splatter-image. We perform the experiments on the hydrants class of the CO3D dataset, which is shown in the table below. The results demonstrate the proposed NP method is effective, leading to a 0.15 improvement in terms of PSNR.

For LRM, due to limited time and resources, we are not able to conduct the comparison with LRM at this stage. However, we use splatter-image as a proxy. Splatter-image achieves better performance than OpenLRM, and our method can be used to improve splatter-image.

---

2.Training time comparison with baselines and the number of parameters.

Our method does not increase the overall training time compared to baseline models, especially converging faster than the primary baseline VNP. As shown in Fig. 13 of the revised paper, our method achieves comparable or better performance with less training time on the ShapeNet Car dataset. Specifically, GeomNP consistently outperforms VNP in terms of PSNR throughout the training process, highlighting its efficiency and effectiveness.

The comparison of the number of parameters is shown in the following table. Our method maintains a smaller than VNP but performs better on the ShapeNet Car dataset.

---

3. Suggestion. Comparative Table highlighting differences with previous methods.

Thanks for the suggestion. The following table highlights the differences across key aspects: whether the method requires meta-gradient steps during testing (“Feedforward”), whether it incorporates probabilistic modeling, how it uses structural information, its hierarchical design, and whether it serves as a general INR that can handle different types of signals.

---

Thank you for your time and efforts. Please don’t hesitate to reach out if you have further questions or need more information.

Thanks for replying to my queries. Splatter Image + GeomNP (proposed) has a higher LPIPS score for the hydrants class. Is this consistent across different categories? My concern is whether the proposed method is similar in performance to the Gaussian Splatting-based methods. If yes, then you should discuss this in the Limitations section.

However, the proposed method clearly outperforms NeRF-based methods. Training time is better than the VNP and converges faster. I also appreciate the table provided by the authors highlighting the differences between the methods. Hence, I raise my score to 6.

Q5. How does the diversity of training scenes affect generalization?

As it is not straightforward to measure the diversity of the training set, we alternatively use a small partition of the training scenes for training the model. By doing so, we implicitly reduce the diversity of training scenes. The experimental results in the following table show that with only 30% of training data (less diversity), our method achieves comparable performance with the deterministic baseline method, TransINR, learned on the full training set. This demonstrates the effectiveness of the probabilistic framework.

Additionally, we train our model on the drums class of the NeRF synthetic dataset and evaluate it on the Lego class. We compare our method with the baseline, GNT [1], in terms of cross-category evaluation. Our method outperforms GNT, both quantitively (1.1 PSNR) and qualitatively (see the visualization in this link).

---

Q6. What is the effect of partial occlusions or noise on GeomNP?

We would like to refer to Fig 7 to illustrate how partial occlusions affect GeomNP. In this figure, we randomly occlude 80% and 90% of the observed image, and GeomNP is able to reconsruct the corresponding images. Moreover, we provide a quantitative comparison for occlusion in the following table. The performance degradation (about 5 PSNR) is still acceptable with such severe occlusion (80%).

[1]. Wang P, Chen X, Chen T, et al. “Is Attention All That NeRF Needs?”[J]. ICLR, 2023.

[2] Szymanowicz, Stanislaw, Chrisitian Rupprecht, and Andrea Vedaldi. "Splatter image: Ultra-fast single-view 3d reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

Thank you for your time and efforts. Please don’t hesitate to reach out if you have further questions or need more information.

We sincerely thank # Reviewer 5cgt for their insightful comments. The following addresses their concerns and provides answers to their questions.

Q1. Comparison with other baselines.

We provide more comparisons with recent SOTA (GNT [1] and splatter image [2]) on the Drums class of the NeRF synthetic dataset and the hydrants class of the CO3D dataset, respectively. GNT is a recent transformer-based generalizable method, and splatter-image is a recent Gaussian Splatting-based generalizable 3D reconstruction method. Our method can seamlessly extend into these related works by utilizing their backbone and NeRF architectures. As shown in the following tables, our work consistently boosts GNT and splatter-image, especially with the improvement of 2.7 PSNR when context view (1-view) is extremely limited.

---

Q2. Computational cost. We compare our method with representative deterministic (TranINR) and probabilistic (VNP) methods. As shown in the table, our method requires fewer parameters than the baseline, indicating lower computational costs and memory usage.

---

Q3. Concern about complexity because of the probabilistic framework.

In Table 6 (a) (also shown below), we compare our method with the deterministic method TransINR on the 2D tasks. Both our method and TransINR have similar transformer encoders. However, our method outperforms TransINR by 1 - 1.4 PSNR on the two image datasets. This indicates that the probabilistic framework introduces very few extra complexity and is able to achieve better performance than the deterministic counterpart, TransINR.

---

Q4. Apply the method on Gaussian splatting.

Yes, our method, specifically the probabilistic design of using neural processing can be used in Gaussian splatting-based methods. We choose splatter-image, a Gaussian splatting-based generalizable 3D reconstruction method, as our baseline for demonstration. The results are provided in the second table of answer 1, which improves by 0.15 in terms of PSNR, demonstrating the proposed method can be effectively applied to splatting methods.

We sincerely thank # Reviewer xqWC for their insightful comments. The following addresses their concerns and provides answers to their questions.

1. Extra comparisons with SOTA generalizable approaches.

Thanks for bringing these references. To ensure a fair comparison, we have integrated the proposed method, GeomNP, into the GNT [2] framework, using the same feature encoder and NeRF architecture. This further demonstrates the flexibility of our method. We conducted experiments on the Drums class of the NeRF synthetic dataset, evaluating both 1-view and 10-view scenarios. The following table shows that our method improves GNT. As one would expect from a (Bayesian) probabilistic method, we see significant benefits with few data (1 context view) and bigger uncertainties. With more data, we also maintain strong performance.

---

2. Generalizable INR setting in the proposed method

We clarify that our method is an any-view reconstruction method. Compared with the mentioned reconstruction works [1,2], our method can also handle generalizable INR under enough observations (10 views).

For the setting details, such as benchmarks from 3D and 2D image regression, we follow representative generalizable INR works ([Learn Init (Tancik et al., 2021), Trans-INR (Chen & Wang, 2022), VNP (Guo et al., 2023) ]), where both Learn Init and Trans-INR are deterministic methods.

We would like to clarify that our probabilistic paradigm is designed for INR functions by modeling distributions over functions, aiming to utilize observations efficiently and effectively.

---

3. Clarification of the statement on “structural information and the information misalignment”.

We clarify that previous methods (Kosiorek et al., 2021; Hoffman et al., 2023; Dupont et al., 2021; Moreno et al., 2023) directly use the feature from the 2D observations and do not consider 3D structure; this implies that they miss the structural information in the 3D space and information alignment between 2D and 3D.

In contrast, our work introduces the geometric bases to model the 3D structure. To investigate the effects of structural information, we compare the variant with perturbation, which disturbs the structure information of basis by adding random noise. As shown in this table, the variant with perturbation underperforms our method w/o perturbation by a large margin, demonstrating the benefits of modeling structural information.

---

Thank you for your time and efforts. We hope the provided experiments and clarifications have addressed your concerns. Please don’t hesitate to reach out if you have further questions or need more information.

Thanks for the update from the author. After reading the rebuttal comment, I still have few concerns that are not addressed.

Q1: When compared with other baseline methods (like GNT and Splatter-image), the improvement is very limited. E.g., although adding proposed method to GNT improved PSNR, both LPIPS and SSIM are not improved. When talking about Splatter-image, the LPIPS score is getting worse. As we have learned from previous literature, PSNR is not convincing compared to LPIPS (which is why LPIPS is invented). And this result raise concern about the effectiveness of the proposed method. Also, the GNT method is a reproduced-version, we do not have information about whether it is still effective when applied to the official one.

Q2: As mentioned in the rebuttal comment to reviewer g7b2, the authors state "probabilistic" as one important contribution. Given this statement, other diffusion-based model (which is also probabilistic model) should be also included into discussion. Those methods (zero123++ [4], MVDreamer [5]) produce pleasing results on multi-view reconstruction. It will be more convincing if the author can make more detailed clarification of comparing these two.