Retri3D: 3D Neural Graphics Representation Retrieval

Q1 [VLMs and Noise Features]

We observed that noise features consistently impact modern VLMs trained to generalize across a broad range of image data. This was demonstrated using both the convolution-based vision encoder from XDecoder and the transformer-based vision encoder from OpenCLIP. Models that are less well-trained or have been exposed to a narrower variety of image data during training are likely to perform worse.

Q2 [Complex Outdoor Environments]

Retri3D does not require special handling for complex outdoor environments. However, if the environment is very large or contains more content, a greater number of renderings would be needed to capture different aspects of the scene. One of the scenes, Sunnyside from LERF, is an outdoor setting that captures a garden with many different objects. Capturing this scene involved a person walking along a path in the clustered garden, making it more complex and information-dense. We present retrieval examples from this scene in Figure 31 (a, b, c).

Q3 [Initial Camera Setup and Poorly Sampled Initial Pose]

We initialize the smart camera movement module by randomly sampling the location from a Normal distribution and the viewing direction from a uniform distribution. The Normal distribution sufficiently distributes the initial cameras, with a bias for more center views which is more likely to have partially noise-free views. SCMM is capable of rotating to a clean view, even when starting from a viewpoint where only a small section is noise-free (as demonstrated in Figures 23 and 24). Combined with uniform sampling of the viewing direction, this approach is more likely to provide different angles of the objects in the center while also capturing objects in the periphery. This approach provides good coverage of the scenes, while enabling the cameras to navigate to more noise-free views. Additional discussion and details on this are provided in Appendix A.4.1.

If the initial pose is poorly sampled, our SCMM (with rotation) can adjust the viewpoint by rotating to recover a good view, provided that a clean view is accessible from the same camera origin with a new viewing direction. However, if the camera origin is surrounded by noise and no view direction offers a good view, SCMM (with rotation and translation) will be required to recover a suitable viewpoint.

Q4 [Comprehensive Coverage and Corners]

We enhance the visual semantic diversity of the coverage by selecting the top k dissimilar clean renderings (discussed in Appendix A.4). Specifically, low-resolution renderings chosen for further high-resolution rendering and analysis must have cosine similarities below 0.9. This approach prevents the selection of overly similar renderings. Since the smart camera occasionally converges to similar viewpoints despite starting from different initializations, this process helps improve view variability.

With SCMM, we aim to maximize coverage while still ensuring less noisy rendering. This is because even if we used a very noisy rendering of a corner, it would not be useful for retrieval.

Q5 [FOV]

We use the FOV consistent with the training images to ensure a consistent FOV is used for comparison of retrieval accuracy among training viewpoint renderings, SCMM, and random sampling. The rendering quality should remain similar if a different FOV is used, thus the retrieval performance should be minimally affected by the specific FOV chosen.

Using FishEye or Panorama projections may introduce unwanted distortions, as existing NGRs typically assume a pinhole camera model. Increasing the FOV while maintaining the pinhole camera model could be a more practical approach, offering the advantage of greater coverage but potentially sacrificing detail. Whether increasing the FOV is overall beneficial likely depends on the specific scene and the focus of the user’s text queries.

W1 [Complex Queries]

In Appendix C.7, Figures 31–35 of the updated PDF, we have included additional visualizations of the retrieval results for queries focusing on different aspects. We observed that when provided clean views with sufficient coverage by SCMM, the VLM generally performs well in retrieving content relevant to the descriptions.

W2 [Noise Pattern and Noise Analysis in Few-shot Setting]

We agree that noise patterns may vary across different NGR architectures. Our findings show that the noise patterns in Splatfacto and Nerfacto are sufficiently similar to enable the use of a shared noise model for effective smart camera movement. Additional visualizations have been added in Figure 30, demonstrating that the noise analysis and camera movement behavior remain very similar when the same noise distribution is applied to handle noise from both Splatfacto and Nerfacto. If other NGRs exhibit distinct noise patterns, a new noise Gaussian model can be fitted and used with Retri3D without requiring further modifications.

For extreme noise in few-shot reconstruction settings, we expect the noise analysis module to remain effective, as renderings with almost complete noise are abundant in the current setting. However, identifying areas in the scene that provide reasonable-quality renderings becomes more challenging. In such cases, an increased number of camera samples may be necessary. For a visualization of noise analysis behavior under conditions of nearly complete noise, please refer to Figure 11(c).

W3 [3D-specific Information]

We agree with the reviewer that in cases where 3D-specific information is part of the model but not observable in 2D, such as with 3D CAD model designs, our approach can lead to information loss and should be complemented by other 3D-specific models. In some cases, however, 3D-specific information can also be represented in 2D if the chosen viewpoint captures it effectively. For instance, in Figure 31 (b, c, d), we illustrate several complex object relationships, such as “a group of,” “near,” “attached to,” and “on.” While these terms describe object relationships in the 3D world, the same relationships can still be understood in 2D when the objects are viewed from an appropriate direction.

W3 [Small Areas and Small Objects]

The primary VLM we used, XDecoder, is capable of segmenting objects within an image and, theoretically, can identify small objects in renderings. For example, the glass mug in Figure 32(d) is a very small object located in a highly clustered area. If retrieving small areas is the primary concern, the SCMM algorithm could potentially be modified to move the camera and zoom in on individual small objects as they are detected. This approach would provide a more detailed, close-up view of these small objects.

I appreciate the authors' response and the additional experiments that address parts of my concerns. I maintain the acceptance rating.

Q1 [Retrieval Application]

The retrieval solution can benefit existing platforms hosting NGRs such as PolyCam and Luma AI, improving user experience for content search. Searching for relevant NGR content also have important downstream applications. Leading softwares in game and art design has started incorporating NGRs [3, 4, 5], necessitating data stores for 3D assets in NGR formats. Additionally, since online NGR stores do not ship with training images, we believe our efficient retrieval solution can be highly beneficial for designers who wish to incorporate pre-existing NGRs into their work.

[3] Luma AI Unreal Engine Plugin: https://www.fab.com/listings/b52460e0-3ace-465e-a378-495a5531e318 [4] Unity Gaussian Splatting: https://radiancefields.com/gaussian-splatting-unity-plugin-updated [5] Blender Gaussian Splatting: https://radiancefields.com/editable-gaussian-splatting-in-blender

Q2 [Comparison with Traditional 3D Representation Retrieval]

We could not compare with the retrieval accuracy by converting to the traditional 3D representations: First, traditional representation 3D retrieval focuses on single 3D objects as opposed to complex scenes. To our knowledge, existing retrieval works for 3D objects cannot be directly applied to point cloud or other traditional representations converted from NGR scenes. Second, as we demonstrate in this work, addressing noise in NGRs is a critical challenge in retrieval that we address with SCMM. This challenge would be exacerbated if the noise in the NGRs is converted into artifacts in point clouds or other representations, where it may not even be identifiable as noise. We attempted to convert NGRs to point clouds and found the quality to be too poor for effective analysis.

W1 [Using Training Views]

We agree with the reviewer that when training views are available, incorporating a small number of training views may not incur significant storage overhead. However, if these training images do not provide comprehensive coverage of the scene, SCMM can still improve variability in visual semantics by increasing overall scene coverage. For larger or more complex scenes, Retri3D offers the opportunity to render and leverage as many views as needed for accurate retrieval, providing a scalable and efficient mechanism to perform this search. In either case, Retri3D’s retrieval pipeline would still offer advantages in terms of speed and memory efficiency compared to other baseline methods.

However, we observe that training images or related information are not always available. Pre-trained NGRs are increasingly treated as standalone data formats for sharing, as seen on platforms like PolyCam [1] and Luma AI [2]. These platforms typically distribute NGR models without accompanying training images, perhaps to simplify sharing or protect intellectual property or user rights. In such scenarios, our method is particularly valuable, as it enables robust and effective retrieval without requiring training views. This flexibility ensures broad applicability across diverse real-world use cases where training data may not be accessible.

[1] PolyCam: https://poly.cam/ [2] Luma AI: https://lumalabs.ai/dashboard/captures

W2 [Viewpoint Selection for Noise Gaussian Training]

We determine if a feature is noise or clean based on its distance to a trained noise Multivariate Gaussian distribution. To train this noise distribution, we generate renderings at randomly sampled viewpoints across multiple scenes. Specifically, the camera origins are sampled uniformly within the 1 × 1 × 1 box of each scene, and the view direction components x, y and z are sampled uniformly between [−1, 1], and the view direction vector is normalized.

The features generated from these renderings are suitable for noise distribution training for the following reasons: a. Noise Dominance in Random Viewpoints: Renderings produced by this random viewpoint sampling process are dominated by noise content, as illustrated in Figure 29. Thus, the features generated by the VLM from these renderings are primarily noise features. b. Consistency of Noise Patterns: Second, the noise patterns are repetitive and consistent across different scenes, leading to a noise feature cluster as seen in Figure 2 (Noisy). c. Minimal Influence from Clean Features: Only few clean features exist and they are not consistent across different scenes.

We use a Multivariate Gaussian distribution that has a single center for noise representation. Since noise features form a single cluster, this distribution is suitable to represent the noise features and provide a strong form of regularization. Note that we do not use GMM (Mixture of Gaussians), as it will introduce multiple centers, not representative of the single cluster the noise features form. Additionally, the few clean features do not consistently shift the Gaussian center in any specific way, and should have minimal impact on the noise distribution training. Therefore, the Multivariate Gaussian distribution effectively represents the noise.

We have updated Appendix A.2 to incorporate details on the camera sampling for the noise Multivariate Gaussian distribution training.

W3 [Camera Initialization and Initialization from Training Viewpoints]

We initialize the smart camera movement module by randomly sampling the location from a Normal distribution and the viewing direction from a uniform distribution. The Normal distribution sufficiently distributes the initial cameras, with a bias for more center views which is more likely to have partially noise-free views. SCMM is capable of rotating to a clean view, even when starting from a viewpoint where only a small section is noise-free (as demonstrated in Figures 23 and 24). Combined with uniform sampling of the viewing direction, this approach is more likely to provide different angles of the objects in the center while also capturing objects in the periphery. This approach provides good coverage of the scenes, while enabling the cameras to navigate to more noise-free views. Additional discussion and details on this are provided in Appendix A.4.1.

If the cameras are initialized from the training views, as the training view is mostly noise free, SCMM will not move the camera.

The response solved most of my concerns. According to the author's response, I agree that the proposed method could benefit direct 3D asset retrieval. I will raise my score to 6.

1. [Noise Across Models]

We thank the reviewer for pointing this out and we clarify that noise features are more consistent for the same model, but can be less similar across different models. However, the noise features are still closer to noise features from other models than to the clean features. We have clarified the statement in the paper (page 6, line 305). To evaluate the impact of these differences on the noise analysis and camera movement, we provide additional visualizations in Figure 30. These visualizations compare noise analysis and camera movement when the same noise Gaussian distribution (trained from GS) is applied to handle noise from both GS and NeRF. A summary is provided here and also in Appendix C.6:

In summary, we observe the following:
i) The noise distribution assigns significantly higher clean scores to actual clean areas compared to noisy areas, whether the noise originates from GS or NeRF.
ii) The noise distribution creates slightly sharper distinctions between GS noise and clean content than between NeRF noise and clean content. However, this difference is small and has minimal effect on the eventual camera movement.

Applying a mixture of Gaussians could enhance robustness in noise analysis. However, as demonstrated in Table 9, the source noise for the Multivariate Gaussian training has minimal effect on overall retrieval accuracy. Consequently, incorporating a mixture of Gaussians is unlikely to significantly impact retrieval accuracy.

2. [SCMM on “ground-truth” Scenes]

If a scene can be rendered from any viewpoint without introducing noise, SCMM's noise navigation module is not strictly necessary to achieve clean renderings. In such cases, the performance of random sampling can approach that of SCMM. However, it is important to emphasize that SCMM's performance does not degrade under these conditions, as the SCMM still has good coverage of various content of the scene by selecting views that are not very similar, as described by the process in Appendix A.4. Thus, SCMM is a robust solution for both noisy and cleaner scenes. Additionally, Retri3D’s retrieval pipeline with embedding generation and matching is designed to be efficient and effective for scenes of varying noise levels.

In practice, we note that even for simple synthetic scenes that are well-covered by training viewpoints, noisy renderings can still occur. To illustrate this, we use the nerf-synthetic LEGO object as an example of a “clean” scene in Figure 36. This LEGO object is well-covered from all training viewpoints, resulting in minimal floater artifacts. Despite this, renderings can appear “noise-like” if the camera is positioned very close to or inside the object. This happens since neural graphics representations with limited model parameters may not perfectly represent the scene or because some viewpoints may have insufficient coverage in the training data.

3. [Camera initialization and Convergence Behavior]

We initialize the smart camera movement module by randomly sampling the location from a Normal distribution and the viewing direction from a uniform distribution. The Normal distribution sufficiently distributes the initial cameras, with a bias for more center views which is more likely to have partially noise-free views. SCMM is capable of rotating to a clean view, even when starting from a viewpoint where only a small section is noise-free (as demonstrated in Figures 23 and 24). Combined with uniform sampling of the viewing direction, this approach is more likely to provide different angles of the objects in the center while also capturing objects in the periphery. This approach provides good coverage of the scenes, while enabling the cameras to navigate to more noise-free views. Additional discussion and details on this are provided in Appendix A.4.1.

To obtain good renderings from SCMM for retrieval, ideally we want good coverage of the scene, while capturing noise-free views. With SCMM, it is possible for some initial camera placements to fail in navigating to a noise-free view. Thus, an initialization strategy that distributes the initial camera points, increases the chances of identifying cleaner views while providing sufficient coverage.

2. [Dependency on VLM Embeddings and Scene Complexity]

We have added a discussion and evaluation of this in Appendix C.7. While Retri3D enables identifying clean renderings with high coverage for better matches, Retri3D is reliant on the efficacy of the embeddings generated by the VLM. We have added more visualizations in Figure 31 of the updated version to showcase the system’s performance given more complex queries in more clustered complex scenes. We observe that the system successfully retrieves scenes when queries involve subtle object relations such as “group of,” “near,” and “attached to.” Additionally, we demonstrate an example where “A duck on a cube” is retrieved even when the object pair is partially occluded.

Beyond the VLM’s capability, several design choices enable Retri3D to handle more complex object relationships in complex scenes:

a. SCMM for High-Quality Viewpoints SCMM allows Retri3D to discover high-quality viewpoints from different angles. In complex scenes, Retri3D can sample diverse, high-quality viewpoints to increase the likelihood of matching complex text queries that may only be evident from specific viewing directions. When VLMs struggle with text-visual associations due to occlusion, Retri3D’s ability to render non-occluded views can lead to more relevant visual embeddings, improving their alignment with text descriptions (as demonstrated in Figure 31(d)).

b. Multiple Queries for Multi-Object Scenes A complex scene can contain a variety of objects. If the user is aware of them, or the user wants to find a scene that matches several descriptions, the user can use multiple queries simultaneously for retrieval. The scene that is the best match (on average) for most of the queries can be identified, and processing multiple queries can be performed very efficiently with Retri3D (detailed in Appendix B.5, Multiple Queries Retrieval).

c. Adaptability with More Powerful VLMs Retri3D can flexibly integrate more advanced VLMs to improve system performance in the future. This would only require simple retraining of the noise gaussian, rather than the visual semantic field or the 3D transformer as in the existing works.

3. [Incorporate other Uncertainty Estimation Methods]

We compare our noise analysis method with the state-of-the-art NeRF uncertainty estimation method (Bayes’ Rays) in Appendix A.5. We demonstrate that our solution achieves similar performance (better in RGB uncertainty estimation, worse in terms of depth uncertainty estimation) compared to Bayes’ Rays. During our testing, Bayes’ Rays is reasonably fast (about a minute to train on top of pre-trained NeRF). During scene noise analysis, our solution uses VLM’s activation maps, where Bayes’ Rays render additional uncertainty values. We expect using Bayes’ Rays for noise analysis to achieve slightly worse accuracy given that we move cameras based on RGB uncertainty. Overhead for computing and memory is minimal for either solution.

Other solutions can incur very high overhead [2] or require a modification to the underlying NGR model [3], which is not very suitable for our setting.

A major advantage of our solution over Bayes’ Rays is generalizability. Since Bayes’ Rays is a dedicated work that tackles NeRF uncertainty estimation (via NeRF MLP Hessians), it is unclear how it can be applied to Gaussian Splatting. In addition, using the VLM’s activation also introduces fewer components over the existing pre-trained NGR since the VLM is required for feature extraction already.

[1] SuperPoint: Self-Supervised Interest Point Detection and Description, DeTone et al., CVPR 2018 [2] Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields, Sunderhauf et al., ICRA 2022 [3] Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representations, Shen et al., 3DV 2021

1. [Additional Comparisons and References]

We thank the reviewer for the references to the recent works. Since none of the mentioned works have been open-sourced, we are unable to compare against them directly quantitatively. We have included a qualitative comparison against these works in Section 2.4. We also discuss here in more detail.

TIGER uses a gaussian-based visual embedding field similar to the one proposed in LangSplat. It further enables editability of the Gaussian field using SDS loss of 2D image editing diffusion model and multi-view diffusion model. N2F2 uses a NeRF-based visual embedding field similar to the one proposed in LERF. Its main contribution is using different subsets of the vector sampled from the visual embedding field to represent the visual semantics at different scales. This differs from LERF in using a scale parameter as input to the LERF MLP to implicitly encode visual semantics at varying scales. Overall, all of TIGER, N2F2, LERF and LangSplat requires training a visual embedding field, and render from it to produce dense visual embeddings for retrieval. ConDense does not train a visual embedding field directly, it instead generates a visual embedding field from a 3D transformer that takes in the NeRF feature grid as input. This visual embedding field is further processed through volumetric rendering and compared with 2D visual features for regularization. The dense visual features can be made sparse by training the 2D and 3D backbones to decide the key sparse features, regularized by SuperPoint [1] during training. However, since they have only tested with viewpoints on the original camera trajectory (ConDense Appendix A.3), whether such key feature detection process is still valid from random viewpoints would require further experiments.

When used for efficient retrieval of scenes, TIGER and N2F2 have the same shortcomings as LERF and LangSplat (qualitatively compared against in the paper): i) expensive training of the visual embedding field alongside the RGB NGR, and retraining is needed if a new VLM is used ; ii) expensive volumetric rendering or rasterization through the visual embedding field before the visual embeddings can be compared with the text embeddings for retrieval; iii) the need for the original training images to create the visual semantic field; and iv) the need for storing dense visual features.

ConDense does not incur the computation cost associated with training the visual embedding field per scene. However, it requires 35000 V100 GPU hours and 4000 A100 GPU hours to prepare the NeRF scenes and train the 3D transformer. Moreover, a 3D transformer trained for NeRF may not work for different types of NGRs (e.g, Gaussian Splats) and this is not demonstrated in the paper. ConDense also requires the original camera trajectories for sampling.

I appreciate the authors' response and the additional discussions added into the manuscript. I will maintain my original score and recommend acceptance of the paper.