LLaNA: Large Language and NeRF Assistant

W1-Q1

Our choice of using frozen models as baselines has been motivated by LLaNA being the only assistant that works on NeRFs, i.e. whose input modality is a radiance field parametrized as a neural network. Indeed, the papers most closely related to ours -- those proposing object-centric assistants that ingest point clouds (PointLLM, GPT4Point) -- employ frozen VLMs as baselines operating on a different modality (i.e., images).

Nevertheless, we agree with the reviewer on the importance of complementing our evaluation with experiments to assess the performance of the baselines trained on ShapeNerf-Text. Thus, we trained all the baselines with their official code on ShapeNerf-Text following their official protocol, which, for all of them, keeps the modality-specific encoder frozen and trains an adaptor and the LLM in two steps.

We report the results of the new experiments alongside those already included in the paper (Frozen baseline models) in Table 1 and 2 (rebuttal PDF; see global comment). We notice that the trained baselines exhibit different behaviors wrt their frozen counterparts, with LLaVA performing significantly worse and PointLLM showing clear improvements. As for GPT4Point, we observe more variability across metrics, though, overall, we are led to reckon that it does not benefit from training on ShapeNerf-Text. The last row in both Table 1 and 2 points out how LLaNA yields the best performance compared to all baselines, either frozen or trained on ShapeNerf-Text.

Eventually, we highlight that the new experiments fortify the key finding of our paper: directly processing the weights of a NeRF is the most effective way to reason about the underlying radiance field. Moreover, converting a NeRF into a different modality, e.g. a point cloud or images, is inefficient and cumbersome (see A.3, main paper). For instance, extracting a point cloud from a NeRF mandates voxelizing the 3D space to evaluate the density function, the required computation scaling cubically with the desired resolution. Images are also not easy to handle, as NeRFs do not come with an object-centered 3D reference frame, so one would need to guess from which viewpoint(s) to render meaningful images. Obtaining the rendering(s) can be pretty slow, especially if high-resolution images are needed to capture important object details. Yet, rephrasing our key finding, it is not worth undertaking any costly NeRF-to-X conversion, as processing a NeRF as a NeRF is more effective.

Q2

To assess the generalization ability of LLaNA, we trained NeRFs for the subset of 200 Objaverse objects with human-annotated captions used as a test set in the PointLLM paper [77]. This sets forth a challenging out-of-domain and open-set experiment (164 out of the 200 Objaverse objects belong to categories not present in ShapeNet). We could also perform a comparative evaluation as we trained the baselines on ShapeNerf-Text (see W1-Q1). Purposely, we extracted point clouds and rendered front views from the 200 Objaverse NeRFs. Results are reported in Table 4 (Rebuttal PDF). We can observe that the scores of all models are significantly lower compared to Table 1 (Rebuttal PDF), which hints at all models struggling when evaluated on objects very diverse to those belonging to the training domain. LLaNA achieves the second-best generalization performance after PointLLM. Yet, it is worth highlighting that the frozen modality-specific encoder of PointLLM (and GPT4Point) is PointBERT, which has been trained on Objaverse, while the meta-encoder used by LLaNA, i.e nf2vec, has been trained only on ShapeNet and thus has never seen any object outside such categories. We also wondered whether the comparison to PointLLM and GPT4Point suggested by the reviewer concerned frozen versions of these models. So, we conducted this experiment and found out that both frozen models provide better scores than LLaNA on the considered 200 Objaverse objects. For example, PointLLM and GPT4Point yield a Sentence-BERT equal to 38.09 and 33.61, respectively, while the score for LLaNA is 30.07. We consider these results reasonable because the frozen models have been trained on Objaverse, so their performance deals with in-domain data. Yet, the gap between GPT4Point and LLaNA, for which this experiment implies reasoning on out-of-domain data, is relatively limited.

W2 – Q3

Following the reviewer's suggestion, we realized a multi-view baseline by rendering images from N viewpoints randomly chosen between the set of camera poses used to train the given NeRF. Then, we concatenated tokens from the N images and fed them into LLaVA alongside text instructions. We set N=3 because the model cannot process correctly a higher number of images. Results are in Table 4 (Rebuttal PDF). As for all reasoning tasks, we observe a slight improvement of LLaVA in the multi-view set-up. Yet, LLaNA keeps outperforming LLaVA by large margins. Conversely, using multiple images boosts the zero-shot classification performance of LLaVa, which turns out to be the best model for this task.

W3

The dataset does not contain classes not seen during training. Indeed, it features 13 classes, and the train, val, and test splits are obtained by randomly sampling objects within each class, i.e., holding out a fixed percentage of objects per class (80%, 10%, and 10% for train, val, and test, respectively).

W4

We apologize for the typo in Table 5; we should have used “Back View,” which has the same meaning as in Tables 1 to 4.

W5

Regarding NeSF, it proposes a novel semantic field representation of 3D scenes rather than seeing neural fields as an input modality. On the other hand, we agree with the reviewer that NeRF-RPN also considers NeRFs as an input modality. Therefore, we agree that the sentence at L20 needs rewriting, and we will modify it.

W1

The proposed ShapeNeRF-Text is built upon the 3D object dataset ShapeNet. ShapeNeRF-Text has 13 classes of ordinary objects, such as cars, airplanes, and chairs. The shapes are divided into 30939, 3846, and 3859 for the train, val, and test set. A NeRF is trained for each of these objects. Regarding the text, we have a brief and detailed description, three single-rounds and a three multi-round QA for each object.

The average lengths in words of the instructions/responses are 8.81/14.25 for single-round QAs, 8.80/14.14 (per round) for multi-round QAs, 8.51/22.76 for brief descriptions, and 7.82/77.90 for detailed descriptions. We report instruction/response length histograms in Figure 1 - bottom (rebuttal PDF; see global response).

Figure 1 - top (rebuttal PDF) shows word clouds obtained after removing generic words like “model,” “object,” and “NeRF”, emphasizing frequent words in the detailed description instruction and response texts.

Finally, we would like to point the reviewer to Appendix C, where we included several details on ShapeNeRF-Text.

W2 - Q1 - Q2

Akin PointLLM and GPT4Point, our paper focuses on an object-centric setup rather than scenes. The proposed ShapeNeRF-Text dataset contains many questions that require a holistic understanding of 3D objects. We prove that with the following two experiments.

First, we evaluate our dataset questions with an LLM (LLaMA3). For each question $Q$, we ask LLaMA3:

Is a random viewpoint of the object enough to answer this question?

$Q$

If so, reply "YES"; if a specific viewpoint is needed, answer "NO"

By doing so, we obtained 5163 “YES” and 5847 “NO”, highlighting that most questions require multi-view information to be answered correctly.

Second, we run a Vision-Language model, LLaVA13b, on each question of the single round QA dataset on the front and back views of objects. Then, we select only the LLaVA responses, where the answer for the front or back view achieves a SimCSE score higher than 80%, i.e., likely correct answers, which selects approximately 45% of the answers. Among these correct responses, we calculate the percentage of those where the front and back answers are extremely different (i.e., a difference in SimCSE scores > 10). Remarkably, 26 % of answers are correct from one point of view but wrong from the other: these questions would have required multi-view information to be answered correctly. We report two qualitative examples in Figure 2. In the first row, the Mercedez-Benz logo cannot be recognized from the back view. In the second row, the monitor seems turned off, and thus it is not possible to identify correctly the helicopter displayed on the screen. Similarly, Figure 11 of the Appendix shows other examples of this kind of QA.

W3 - W4

LeRF [2] ([32] in our paper) and LangSplat [3] are innovative representations of 3D objects and scenes. They extend the radiance field formulation, considering functions that model density, color, and language features at each spatial coordinate. These language fields are parameterized by either a neural network (LeRF) or a set of 3D Gaussians (LangSplat). We believe that rather than competing with LLaNA -- a multimodal LLM that considers NeRFs as an input modality -- [2] and [3] propose field-based representations semantically richer than standard NeRFs.

3D-LLM [1] ([24] in our paper) is a multimodal LLM that processes a colored mesh or a set of posed images. Thus, like PointLLM and GPT4Point, it can be applied to data extracted from NeRFs. In our experiments, we did not include it among the baselines because 3D-LLM addresses scenes and scene-specific tasks, such as 3D grounding and navigation. On the contrary, LLaNA focuses on object-centric scenarios, like PointLLM and GPT4Point.

Yet, as requested by the reviewer, we have evaluated 3D-LLM on our ShapeNeRF-Text dataset to compare its performance to LLaNA. Purposely, we have extracted colored 3D meshes from the NeRFs belonging to the test set of ShapeNeRF-Text and processed these data by the official 3D-LLM code to render the images and compute both the 2D and 3D features required by the model at inference time. The results are reported in Table 4 (Rebuttal PDF). We notice that 3D-LLM provides performance somehow comparable to PointLLM (see Tables 1, 3, 4, and 5 of the paper). Yet, as highlighted by the last row of Table 4 (Rebuttal PDF), LLaNA performs much better on all tasks.

Moreover, we highlight that 3D-LLM has several disadvantages compared to LLaNA.

1. 3D-LLM requires several components to extract 3D language features which are the input of their LLM. Indeed, it requires a pre-trained vision-language feature extractor (CLIP), a general-purpose semantic segmentation network (SAM), and a way to handle noisy 3D data (gradSLAM) to project language features in the 3D space. Conversely, LLaNA extracts NeRF features by a single forward pass of the meta-encoder, which takes just a few ms on a GPU.
2. As it operates on meshes and images, 3D-LLM does not scale well with the resolution of the input signal. Conversely, the number of NeRF weights is decoupled from the spatial resolution of the underlying signal, making LLaNA resolution-agnostic.
3. In our scenario where NeRF is the input modality, we must extract a 3D mesh and render N views to apply 3D-LLM. However, as already stated in the paper (see L53-57), this process is not trivial and time-consuming. Moreover, important details might be lost when the extracted geometry from NeRF is too noisy or low resolution.

W5

To assess potential spurious patterns in the question-answer relationships, we evaluated the performance of LLaMA 2 fine-tuned on ShapeNeRF-Text (Table 5 rebuttal PDF). There is a significant performance gap between LLaMA2 and LLaNA, highlighting that our dataset consists of questions that can only be answered with access to information about objects.

W1

ShapeNeRF-Text is built upon ShapeNet. 3D shapes are divided into 30939, 3846, and 3859 for the train, val, and test set, respectively. A NeRF is trained for each of these objects. As for text, the dataset features a brief and detailed description and 3 single-rounds and one three-rounds QA for each object. We computed word clouds and the instruction/response length statistics, partially shown in Figure 1 (rebuttal PDF; see global response). We will add basic details on the dataset size in the intro and all the details in the Appendix.

W2-W3

To address W8 (see below), we have trained all the baselines considered in our paper on ShapeNeRF-Text (Table 1, Rebuttal PDF). To train the baselines, we have used the official training code, which, for all of them, keeps the modality-specific encoder frozen and trains in two steps both an adaptor and the LLM. Thus, to probe generalization, we have evaluated LLaNA and the trained baselines on the subset of 200 Objaverse objects with human-annotated captions used as a test set in the PointLLM paper [77]. This sets forth a pretty challenging out-of-domain and open-set experiment (164 out of the 200 Objaverse objects belong to categories not present in ShapeNet). To conduct our experiments, we created NeRFs for all 200 objects. Then, we extracted the point clouds and rendered front views from NeRFs to test the baselines. The results are in Table 4 (Rebuttal PDF). We can observe that the scores of all models are significantly lower compared to Table 1 (Rebuttal PDF), which hints at all models struggling when evaluated on objects very diverse to those belonging to their training domain. LLaNA achieves the second-best generalization performance after PointLLM. Yet, it is worth highlighting that the frozen modality-specific encoder of PointLLM (and GPT4Point) is PointBERT, which has been trained on Objaverse. In contrast, LLaNA meta-encoder, i.e, nf2vec, has been trained only on ShapeNet and thus has never seen any object outside the ShapeNet categories.

W4

The addition shows how to calculate the number of rows of the input matrix $M$ obtained by stacking the NeRF’s weights. The number of rows of $M$ depends on the number of hidden layers, $L$, the number of units per hidden layer, $H$, and the dimension of the input, which is a $144$-dimensional array obtained by frequency encoding of 3D coordinates. We will move the detailed calculation in the Appendix.

W5

We apologize for any confusion. We meant that, as our ShapeNeRF training dataset was generated using LLaMA's automated text creation, which includes built-in safeguards, and as ShapeNet is a manually curated dataset with only common items such as chairs and cars, our model trained on these data should also be safe. However, we acknowledge that we have not conducted any experiments to validate this claim and will remove that statement.

W6

The image represents the automatic pipeline used to create the ShapeNeRF dataset. Given a 3D model corresponding to a NeRF, we render $N$ views by a computer graphic engine. Each view is then processed by a VLM, i.e. LLaVA, obtaining $N$ view-specific captions. These captions are finally aggregated by an LLM, i.e., LLaMA 3, to obtain brief and detailed descriptions and single-round and multi-round Q&As. We will add this information to the caption.

W7

The proposed ShapeNeRF-Text dataset features 13 classes. Following [61], the train, val and test splits are obtained by randomly sampling objects within each class, i.e., holding out a fixed percentage of objects per class. So, there are no held-out classes.

W8

Our choice of using frozen models as baselines has been motivated by LLaNA being the only assistant that works on NeRFs, i.e. whose input modality is a radiance field parametrized as a neural network. Indeed, the papers most closely related to ours -- those proposing object-centric assistants that ingest point clouds (PointLLM, GPT4Point) -- use frozen VLMs as baselines operating on a different modality (i.e. images).

Nevertheless, we agree with the reviewer on the importance of adding experiments with baselines trained on ShapeNeRF-Text. Thus, we trained all the baselines on ShapeNeRF-Text following their official protocol and using the official training code (see also answer to W3). We report the results of the new experiments alongside those already included in the paper (Frozen baseline models) in Table 1 and 2 (Rebuttal PDF). We notice that the trained baselines exhibit different behaviors than their frozen counterparts, with LLaVA performing significantly worse and PointLLM showing clear improvements. As for GPT4Point, we observe more variability across metrics, though, overall, we are led to reckon that it does not benefit from training on ShapeNerf-Text. The last row in Tables 1 and 2 shows how LLaNA performs best compared to all baselines, frozen or trained on ShapeNerf-Text.

Eventually, we highlight that the new experiments fortify the key finding of our paper: directly processing the weights of a NeRF is the most effective way to reason about the underlying radiance field. Moreover, converting a NeRF into a different modality, e.g. a point cloud or an image or set of images, is inefficient and cumbersome (see A.3, main paper). For instance, extracting a point cloud from a NeRF mandates voxelizing the 3D space to evaluate the density function, the required computation scaling cubically with the desired resolution. Images are also not easy to handle, as NeRFs do not come with an object-centered 3D reference frame, so one would need to guess from which viewpoint(s) to render meaningful images. Obtaining the rendering(s) can be pretty slow, especially if high-resolution images are needed to capture important object details. Yet, rephrasing our key finding, it is not worth undertaking any costly NeRF-to-X conversion, as processing a NeRF as a NeRF is more effective.