INRet: A General Framework for Accurate Retrieval of INRs for Shapes

The need for encoder/decoder after the training process. The need for multiple representations

One notable feature of INRet is that the user models inserted into the database do not need to be trained along with the decoders that are used to help train the INR encoders. The user INRs can be trained from scratch by the user without having access to the encoder or decoder. To be more precise, the common decoder in Figure 2(c) is not needed for the embedding creation as new INR models are added to the database. The encoder in Figure 2(b) is frozen after the initial database setup, and is required when new INR models are added to the database. In summary, once the encoders are trained, given an INR pretrained by the user, the INR embedding can be created with a single forward pass through the trained encoders, which is much faster than converting the trained INR to other representations such as point cloud.

We leave the option of creating an embedding using different INR representations because different INRs have different strengths. For example, as reviewer “U5A9” has pointed out, a multi-layer car might only be properly represented with a UDF INR. In the meanwhile, one may want to represent a watertight surface and ensure it is watertight so he/she may choose an SDF INR. We design INRet so that the users can feel free to use the best INR implicit function for the most suitable representation without worrying about the incompatibility between them for the retrieval process.

Figure 2 mesh specific and global

INRet is not generally tied to mesh. We use mesh in this work only as a starting point to perform training of INRs. Figure 2(a) involves training of the INRs from mesh, however, specific INRs can also be trained from other 3D representations (UDF from point cloud, Occ from voxel grid). The encoders in Figure 2(b) are used for encoding new INRs put into the database, they are therefore global throughout the training and deployment phases. The decoder in Figure 2(c) is only used for the encoder training, and is not useful once the encoders have been successfully trained.

Summation and concatenation of the features

We choose summation or concatenation of the features based on how the features were originally used in the grid-based INR. NGLOD used summation of the features and iNGP used concatenation of the features. One of the reasons for their design choice might be iNGP requires all the features to mitigate the negative effects of hash collision using the MLP, but NGLOD does not have the issue. INRet follows their design choice and uses the same method to combine features as input to the Conv3D encoder.

We ran an additional experiment to test out what would happen if the features from NGLOD are concatenated (instead of summed), and the features from iNGP are summed (instead of concatenated). In summary, the retrieval accuracy significantly decreases, dropping 14.8% for NGLOD and 53.8% for iNGP. Please see further details in Appendix B.7.

Reconstruction quality and its impact on retrieval performance

We provide details on the reconstruction quality achieved by various architectures in this work in Appendix C.1 Table 15. We also provide more visualization in Appendix C.2 Figure 4 to demonstrate the reconstruction quality.

To understand how the retrieval accuracy is impacted by the reconstruction quality, we perform another experiment in Appendix C.1 (Table 16) where we train an iNGP INR with a small number of epochs (2) so that the iNGP INR has a lower reconstruction quality than an MLP-only INR. In summary, we found that despite having lower reconstruction quality than the MLP-only INR, the features in the iNGP hash grid still made retrieval accuracy from a low-reconstruction quality iNGP higher than the MLP-only INR. Therefore, supporting INRs with feature grid in INRet is essential for improving the retrieval accuracy irrespective of the reconstruction quality of the INRs. Please refer to Appendix C.1 Table 16 for more details.

Meshes vs Reconstruction

We would like to note that for INRet, we use the mesh to train INRs and then perform retrieval using the features and weights in these trained INRs. Our goal in this work is to analyze how retrieval can be done effectively if the shapes are implicitly represented with different kinds of INRs.

If we were to support querying the INR database using meshes, there are two possibilities. Since INRet is designed to store INRs, to query INRs with the mesh, we can either convert the query mesh into an INR and use INRet, or reconstruct the INRs in the database and render into multi-view images and use view-based retrieval methods. As we have demonstrated in the experiments, converting the mesh to INR and use INRet is more accurate compared to multi-view-based methods (Section 5.2 Table 1).

Figure 2 Labelling

We thank the reviewer for the suggestion and we have added additional labels/symbols in Figure 2.

Database containing various types of INRs

INRs are becoming a very popular representation for shapes, graphics objects, scenes etc. For example, nerfstudio has recently published a tool to export trained NeRF scenes to unreal engine, which would make production use of these INR models much easier [1]. We also see companies starting to use INR to encode many forms of data, such as encoding fonts [2], materials [3] and various forms of 3D assets [4]. We expect to see more tools like this in the future, which means INRs will have an increasing impact in the real world. This would require a database to store these models for efficient retrieval, similar to relevant head searching in existing mesh-based databases [5].

[1] Nerfstudio: Exporting to Unreal Engine [nerfstudio, Unreal Engine [https://docs.nerf.studio/extensions/unreal_engine.html]

[2] Joint Implicit Neural Representation for High-fidelity and Compact Vector Fonts [ICCV2023, Adobe, https://openaccess.thecvf.com/content/ICCV2023/papers/Chen_Joint_Implicit_Neural_Representation_for_High-fidelity_and_Compact_Vector_Fonts_ICCV_2023_paper.pdf]

[3] Real-Time Neural Materials using Block-Compressed Features [preprint, Ubisoft, https://hal.science/hal-04255874v1]

[4] [Luma AI, https://lumalabs.ai/]

[5] [Ubisoft, https://montreal.ubisoft.com/en/using-ml-and-complex-math-to-deconstruct-and-reconstruct-human-faces/]

Matching among SDF, UDF, Occ. Matching of multilayer car and open shirt

One of the main design goals of INRet is to support retrieval across INRs with different implicit functions, especially when the shape might be well represented with a specific kind of implicit function but not the others. While it might be difficult to find a “closed-surface” shirt counterpart to a open-surface shirt, we can easily demonstrate INRet’s robustness using multilayer car and single-layer watertight car.

We demonstrate the robustness of INRet’s cross implicit function retrieval with a multilayer car and watertight car. In Appendix C.2 (Figure 4), we show that using a multilayer car as query (represented by a UDF INR) can successfully retrieve watertight cars (represented by SDF INR).

Qualitative Visualizations

We have added more visualizations in Appendix C.2 to show qualitative comparison between our method and the baseline methods.

INR with MLP-only, triplane and points

We note that INRet trivially supports encoding of MLP-only INRs by simply not encoding the feature grid component if it is lacking. We added more experiments in Appendix B.3 (Table 10) showcasing INRet’s performance when applied to MLP-only INRs. We have also included an evaluation for INRs with triplane feature grids which is supported by INRet (by simply encoding features from triplane grid with Conv3D encoder). For MLP-only INR, using INRet improves retrieval accuracy across different implicit functions drastically. For triplane INR, using INRet leads to retrieval accuracy similar to NGLOD and iNGP INR. Please see Appendix B.3 (Table 10) for more details.

We would like to note that since Point-NeRF uses points to perform novel view synthesis. The points alone cannot encode a continuous 3D field of SDF/UDF/Occ values and are thus not suitable for implicit shape representation.

The importance of regularization terms

We show tSNE plots highlighting the difference between INR embeddings when the common decoder is applied and is not applied. In summary, using the common decoder regularization leads to the embedding for the same category being much more tightly clustered. Without the common decoder, the embeddings for shapes belonging to the same category are much more spread out, this is likely a result of different decoders requiring the INR embedding for the same shape to be used for different purposes (decoding UDF, SDF, or Occ). Please refer to Appendix D (Figure 5) for further details.

tSNE plot; conversion and ensuring meaningful INR weight/embedding (also in Q1)

We provide tSNE plots including all of the UDF/SDF/Occ embeddings at the same time and separately in Appendix D (Figure 5 & 6), which helps to visualize the embedding space created by INRet.

If the input is a 3D model (not INR), we apply the INR distillation technique detailed in Section 5.3 and Appendix A.3. It would take around 30 seconds to train a NGLOD or iNGP INR from the test 3D model to enable retrieval (Section 5.3 Table 2).

To ensure the embeddings are meaningful, we require the INR’s MLP to be initialized in the same way as the INRs that were used for training the INR encoders. This is similar to inr2vec. We note that we do not have the same initialization requirement for feature grids as the feature grids are typically initialized with very small random values. This is further explained in Appendix A.1 INR Initialization.

In the case that the INR MLP weights are initialized differently, we need to apply the INR distillation technique to train another INR with a specific MLP initialization. However, we note that using the same weight can be enabled via setting a constant random seed and does not affect the performance of the INR itself (such as the reconstruction quality).

Writing Quality We thank the reviewer for pointing out the mistakes and we have fixed them.

Novelty in proposing new architectures, representations, and metrics

Since the goal of our work is to demonstrate the feasibility of creating a common latent space for retrieval of INRs, we selected popular existing INR architectures and implicit functions without further modifications. This obviates the need for the use of new architectures or representations. Our hope is to keep the framework general such that it can easily support additional existing INR representations (such as triplane pointed out by reviewer U5A9) and future INR representations. Our analysis showed that embeddings to encode INRs can be generated using simple architectures such as Conv3D and MLP.

In the updated version, we propose a new metric that addresses shortcomings of previously proposed metrics. We find that existing metrics evaluate the success of retrieval based on the category of the shape, while largely ignoring the differences between shapes within the category. Therefore, we propose a new metric: Category-Chamfer, where the retrieval is only successful if the retrieved shape is the shape that shares the same category while having the least Chamfer Distance with respect to the query shape. This was initially omitted from the paper since it not only applies to INR but to 3D shapes in general. We provide further details about this new metric as well as a way to accelerate retrieval of the shape required by this metric in Section 3.4 and Appendix E.

Weighting and conversion between UDF/SDF/Occ embedding

In our original implementation, we used equal weighting for the explicit loss between UDF, SDF and Occ. We ran further experiments that used different weights between different explicit loss pairs. In summary, using different weights for the explicit loss has minimal impact on the final retrieval accuracy. Results and further explanations can be found in Appendix B.5.

Regarding the potential conversion between UDF/SDF/Occ representations. We would like to note that the features INRet obtained from the feature grids are not actual UDF/SDF/Occ values. They are latent features used by the MLP of the INR for shape reconstruction, or used by the Conv3D encoder of INRet for INR embedding generation. Therefore, unlike the conversion between actual distance values (UDF/SDF/Occ) which is straightforward, there is no simple way to perform conversion between the latent vectors sampled from the feature grid. The difficulty of conversion in the latent space is one of the primary reasons we designed INRet to handle INRs with different implicit functions.

Does occupancy contain more information than SDF/UDF

It is hard to quantify whether the occupancy grid is more feature-rich than the S/UDFs as they all ultimately serve to implicitly represent the surface of the shape. As shown by the quality of shape reconstruction in Appendix C.1 (Table 15) of the updated draft, Occ/SDF/UDF INRs have slightly different shape reconstruction quality (UDF>SDF>Occ) but they are in general at the same level.

Theoretically, the SDF might contain the most information as it can be easily converted into UDF (by eliminating the sign) and Occ (by eliminating the magnitude). This may be why SDF-SDF retrieval (84.6 for iNGP) has higher accuracy than UDF-UDF retrieval (83.0 for iNGP) as seen in Section 5.4 (Table 3).

Choice of loss for the common decoder; bias towards the loss of the common decoder (UDF); pairwise differences

We performed more experiments where we used a different type of loss for the common decoder (SDF and Occ). We provide experiment details and results in Appendix B.4 (Table 11). In summary, INRet is in general robust to the choice of common decoder loss. The retrieval accuracy is high regardless of the choice of the loss. This could enable applications such as shape generation if the trained decoder is used for this purpose. Please refer to Appendix B.4 (Table 11) for further details.

Q1. L1 vs L2 norm for UDF common decoder in Equation (4)

The first loss is L1 norm. We chose L1 norm to be consistent with existing work that uses L1 norm for UDF loss (we used a UDF common decoder) [1]. We performed further experiments applying L2 norm to the UDF common decoder. In summary, the choice of L1 and L2 norm does not have a significant impact overall. Please refer to Appendix B.6 for further details.

Q2. Potential bias towards the loss used by the common decoder (UDF); Pairwise differences

We performed further experiments to study the bias with respect to the choice of the common decoder loss. The choice of loss indeed makes the embedding slightly more biased towards the specific type of loss. For example, using the SDF common decoder, the SDF-SDF retrieval accuracy improved from 84.6% to 85.8%, while using the UDF common decoder resulted in the highest UDF-UDF retrieval accuracy. Please refer to Appendix B.4 (Table 11) for more details.

We design a common decoder to output the same type of value regardless of the implicit function of the original INR. Therefore, the common decoder is not designed to take pairwise differences between the INRs. This is performed by the explicit L2 regularization.

[1] Neural Unsigned Distance Fields for Implicit Function Learning [Neurips 2020, https://arxiv.org/abs/2010.13938]