TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields

We thank the reviewer for the constructive comments and the opportunity to clarify our approach.

Q1. More Baselines

The reason why not consider more baselines is that few available 3D-supervised methods are designed towards open vocabulary 3D generation, except Point-E and Shap-E. We explained in Q4 of Reviewer oUD4 that available TAPS3D requires a pre-trained GET3D checkpoint, which is supervised in a single-category manner. Similarly, SDFusion has to pre-train a VQVAE, while limited categories are included in its pre-training dataset.

Nonetheless, we still pre-trained a GET3D in our ShapeNet55 dataset (already done in Table 2 of the main text) and attempted to fine-tune it with TAPS3D. As shown in Table R2 of Reviewer oUD4, the generation quality is restricted by the pre-trained GET3D, falling below our method. Considering GET3D can hardly generate high-quality 3D results in a multi-category manner, the fine-tuning strategy in TAPS3D is not an appropriate approach to open-vocabulary generation. SDFusion may meet a similar situation to TAPS3D.

One can only compare our method with the original SDFusion/TAPS3D in a single-category generation task. SDFusion has released its text-to-shape checkpoint, which is trained on a table-chair dataset. Therefore, we update Figure 10 in Appendix, further evaluating the original SDFusion with recursive text prompts. As shown in Figure 10, SDFusion is a shape-only generator with low generation resolution. Besides, it is not sensitive to long sentences, failing to generate curved legs in the last prompt.

Q2. DreamFusion Test List

We reviewed the original paper of DreamFusion and didn't find the testing list with over 400 prompts. The paper mentions that "we use the 153 prompts from the object-centric COCO validation subset of Dream Fields", which is exactly the set we used in our paper. However, we find a prompt list in a public repo threestudio, which includes 415 prompts. We evaluate the CLIP R-Precision in this test set. Since we haven't surveyed any method that is evaluated in this set, we only compare our method with Shap-E. As shown in Table R3, although it is a harder test set, our method still significantly outperforms Shap-E.

Table R3. CLIP R-Precision results on a test list with over 400 prompts.

Q3. More Image-to-3D Results

We provide more visualization examples in Figure 16 of the updated Appendix. Compared with Shap-E, our generation results present more detailed textures and more reasonable geometries. As for the quantitative evaluation, we have provided the ablation study of NTFBind in Table 5 of Appendix, which verifies our improvement in FID. We further evaluate Shap-E in this test set, and its FID is 51.32. The quantitative result is consistent with the visualization comparison. We will update the result in our revision.

Q4. More Complex Prompts

We select some complicated prompts from DreamField/DreamFusion evaluation sets and compare our results with Shap-E in Figure 15 (in the updated Appendix). Our results basically contain more detailed textures and more reasonable geometries when compared with Shap-E, exhibiting stronger open-vocabulary capability. As the reviewer mentioned in Weakness 4, concept mixing is super hard given such limited training data. Our method can understand most of the descriptions in the given prompts. While we acknowledge that the quality of our generation results is still not satisfactory enough. The generation quality compromises as the descriptions are increased. For example, the giraffe in Figure 15 is not as complete as the one generated in Figure 11. And our method is limited in imagination. As for the chair that looks like an avocado, our generated result simply combines the shape of a chair with the color of an avocado. Nonetheless, it is crucial to acknowledge the substantial contributions our work has made in advancing open-vocabulary text-to-3D with limited data. We will add an additional discussion in the limitation part of our revised manuscript.

Dear reviewer,

Thank you again for your insightful comments on our paper, and we genuinely hope that our response could address your concerns. As the discussion is about to end, we are sincerely looking forward to your feedback. Please feel free to contact us if you have any further inquiries.

We thank the reviewer for the constructive comments.

Q1. Diversity Claim

We acknowledge the reviewer's point regarding the challenge of achieving shape diversity with a smaller scale 3D shape dataset in comparison to 2D diffusion models. The term "equivalent" was perhaps optimistic, and we have amended our language to more accurately reflect our findings.In Table 1 of the main text, our CLIP R-Precision results significantly outperform previous 3D supervised methods and even get close to V-L pre-trained methods. The results indicate that our method exhibits its potential towards open-vocabulary generation. Accordingly, we revise the expression here as: With limited data, can we train a real-time 3D generator with the potential towards open-vocabulary content creation?

Q2. Open-Vocabulary Capability

We select some complicated prompts from DreamField/DreamFusion evaluation sets and compare our results with Shap-E in Figure 15 (in the updated Appendix). Our results basically contain more detailed textures and more reasonable geometries when compared with Shap-E, exhibiting stronger open-vocabulary capability. As Reviewer 9TYY mentioned in Weakness 4, concept mixing is super hard given such limited training data. Our method can understand most of the descriptions in the given prompts. While we acknowledge that the quality of our generation results is still not satisfactory enough. The generation quality compromises as the descriptions are increased. For example, the giraffe in Figure 15 is not as complete as the one generated in Figure 11. And our method is limited in imagination. As for the chair that looks like an avocado, our generated result simply combines the shape of a chair with the color of an avocado. Nonetheless, it is crucial to acknowledge the substantial contributions our work has made in advancing open-vocabulary text-to-3D with limited data. We will add an additional discussion in the limitation part of our revised manuscript.

Q3. View-Invariant Experiments

It is reasonable to compare features in the same category. We randomly select 50 instances in each category and calculate the average retrieval precision of all 55 categories in Table R1. Our view-invariant features still present an obvious improvement against vanilla CLIP features. Note that comparing different instances in the same categories is a more challenging task, as different instances in the same view sometimes share more similar content than the same instance in different views. We will update the results in our ablation study. Thanks for the suggestion.

Table R1. Comparison of view-invariant and vanilla CLIP features across instances in the same category. $r$ is the retrieval precision with CLIP features and $\widetilde{r}$ is the retrieval precision with our view-invariant features.

Q4. Comparison with TAPS3D

The reason why not compare TAPS3D is that it is a fine-tuning method based on single-category GET3D pre-training. In the implementation of the original TAPS3D, a pre-trained category-wise GET3D checkpoint (e.g., car checkpoint, table checkpoint, and so on) is first loaded to the generator. The generator is then fine-tuned by the supervision of a fixed discriminator. Considering that GET3D hasn't been pre-trained in an open-vocabulary setting, it is hard to evaluate the original TAPS3D in our benchmark.

Nevertheless, we still provide a comparison based on their official code. We load the checkpoint of our pre-trained GET3D (Table 2 in the main text) and further finetune the generator with TAPS3D. As shown in Table R2, its FID and CLIP-Score (ViT-B/32) results are uncompetitive to our method and even below the version we reproduced (named as (b) in the main text). We think the results are reasonable, as the performance of fine-tuned TAPS3D strongly depends on the pre-trained GET3D. We have demonstrated in Table 2 of the main text that GET3D can hardly generate high-quality 3D results in a multi-category dataset. As a result, TAPS3D is not an appropriate approach to open-vocabulary generation.

Furthermore, since TAPS3D hasn't released the training captions and finetuned checkpoints, we cannot even make a comparison on single-category generation with them. We notice that SDFusion has released its text-to-shape checkpoint, which is trained on a table-chair dataset. Therefore, we update Figure 10 in Appendix, further evaluating the original SDFusion with recursive text prompts. As shown in Figure 10, SDFusion is a shape-only generator with low generation resolution. Besides, it is not sensitive to long sentences, failing to generate curved legs in the last prompt.

Table R2. Performance of the original TAPS3D. The results of GET3D and (b) are from Table 2 in the main text.

Q5. Dynamic Noise

We have tried this non-dynamic noise before. We set $\sigma$ to a static number 0.016, which is exactly the setting in [A]. We trained the non-dynamic noise in ShapeNet55 dataset with BLIP-2 captions, which is evaluated with 27.64 in FID and 30.06 in CLIP-Score (ViT-B/32). The results are not competitive to our dynamic noise (26.94 in FID and 30.35 in CLIP-Score). We will add the results to Appendix.

Q6. Naming of $L_{img}$ and $L_{txt}$

Thanks for the suggestion. We rename $L_{img}$ and $L_{txt}$ as $\mathcal{L}\_{nce}^{I}$ and $\mathcal{L}\_{nce}^{T}$, respectively. The revised names can be more consistent in meaning.

Q7. Textured Mesh Generator

We train the generator from scratch. We aim to train a unified generator for multiple categories, while GET3D aims to train category-wise generators. Loading its pre-trained model is not beneficial for our training.

Q8. Silhouette Loss

Thanks for the suggestion. We will provide an additional explanation in our revision that the silhouette loss is followed from GET3D.

[A] Text-Only Training for Image Captioning using Noise-Injected CLIP.

Dear reviewer,

Thank you again for your insightful comments on our paper, and we genuinely hope that our response could address your concerns. As the discussion is about to end, we are sincerely looking forward to your feedback. Please feel free to contact us if you have any further inquiries.

I appreciate the author's effort to include more results. The new multi-view experiment, explanation with TAPS3D, and the ablation on dynamic noise prove the technical solidity of the paper.

For the open-vocabulary results, from visual perception results, I think the generated shape indeed has higher texture quality, but the shape does not semantically match with the textual input as Shape E. I acknowledge this is a difficult problem using any CLIP-like loss as supervision. I suggest the author include it in the limitation.

Overall, I think the paper is technically solid. The NTFGen and NTFBind are effectively proved by the added experiments. I recommend acceptance of this paper.

Thank you very much for the encouraging and constructive comments.

Q1. Potential Overfitting

Overfitting in generative models is a complicated issue that is worth investigating. In [A], overfitting is defined as "containing an object (either in the foreground or background) that appears identically in a training image, neglecting minor variations in appearance that could result from data augmentation". They detect and analyze content replication in the diffusion synthesis results, concluding that most of the generated images do not contain significant copied content, but a non-trivial amount of copying does occur.

Following [A], we detect the most similar object to our generated beer can among the Objaverse dataset, which is shown on the left of Figure 14 in the updated Appendix. Our generated beer can only shares a similar rectangular text box with it. Similar to the conclusion in [A], our text box is a regional copy, rather than a significant content replication. The same phenomenon can also be observed in the graffiti of the bus in Figure 15, which is similar to the graffiti on the cardboard you mentioned. Therefore, those unusually detailed outputs may be a possible regional copy but are not exactly the overfitting problem.

To some extent, our NTFGen module is beneficial for alleviating overfitting, as the injected noise enlarges the text input to a range of objects. As shown in Figure 14 (in the updated Appendix), we provide 9-shot examples of our generated beer cans and vases. Our results present various textures. Few of them share a similar color or texture. We will update the discussion of overfitting in the Appendix accordingly.

Q2. FID Score

For 3D objects in the evaluation set, we have corresponding captions and ground-truth rendered images $\hat{I}$. To evaluate the FID metric in the text-to-3D task, we generate 3D content according to captions and then render generated images $I$. We compute the FID between the ground-truth set {$\hat{I}$} and the generated set {$I$}.

Q3. 9-Shot Experiments

Our 9-shot experiment is to generate an object nine times and choose the best generation based on CLIP similarity. We include this experimental setting because our generation process is significantly faster than other methods. As an example of visualization results, we update nine beer cans and vases in Figure 14. Our generation results vary greatly in geometry and texture, presenting our diversity in a given text prompt. In the 9-shot experiment, we chose the most similar rendered image (the first one, based on CLIP similarity) for calculating R-Precision.

[A] Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models.

Dear reviewer,

Thank you again for your insightful comments on our paper, and we genuinely hope that our response could address your concerns. As the discussion is about to end, we are sincerely looking forward to your feedback. Please feel free to contact us if you have any further inquiries.