SeMv-3D: Towards Semantic and Mutil-view Consistency simultaneously for General Text-to-3D Generation with Triplane Priors

Thank you for your recognition of our method and paper. Below, we address the questions and weaknesses you identified:

1. Weaknesses

   1. Comparison with Existing Methods We sincerely appreciate your suggestions for additional comparison methods. In response, we have supplemented the relevant comparative results on page 8 and in Appendix A.2.1. Based on both quantitative and qualitative evaluations, we observe that while DreamView achieves higher semantic consistency compared to MVDream, its multi-view consistency deteriorates. Additionally, DreamView does not address the limitation of generating a restricted number of views, which SPAD partially resolves. Consequently, in the user study, DreamView's preferred rate was lower than that of MVDream.

      In contrast, our semv3d method surpasses DreamView in multi-view consistency while achieving comparable levels of semantic consistency and generation quality. Furthermore, our approach enables any-view multi-view generation, providing a distinct advantage. Thus, we believe that semv3d demonstrates overall superior performance.

      Additionally, it is worth mentioning that the second method, DiverseDream, being an SDS optimization-based method, is not a feed-forward approach and was therefore excluded from our comparison.

   2. Specific Weakness Clarification Please refer to the response to your question below for further elaboration on the identified issue.

   ---

   Questions

   Regarding the appearance of minor scatter points in the generated outputs, we identify two potential causes:

   1. Dataset Quality: The dataset contains broken objects that were not fully filtered out during preprocessing. As shown in Figure 10 of Appendix A.5, the model learns the characteristics of these fragmented objects, which manifest as discrete noise artifacts during generation.
   2. Batch Size Limitation: Due to computational resource constraints, the model was trained with a relatively small batch size. This limitation may have impacted the model's robustness, particularly in handling thin or delicate object structures.

   We acknowledge these issues and aim to address them in future work by improving dataset preprocessing and exploring methods to enhance model robustness under resource constraints.

Thank you for your valuable suggestions. We have updated the manuscript to address several details that were previously underexplained. Below, we respond to the weaknesses you raised:

Weakness：

1. Quantitative Results: We acknowledge your point regarding quantitative results. To address this, we have supplemented the corresponding quantitative analyses in the supplementary materials, specifically in Appendix A.2.3.

2. Explanation of TPL Stages: We acknowledge the previous lack of clarity regarding the two-stage structure of TPL. The manuscript has been revised, and a detailed explanation will be provided in response to your first question below.

3. We apologize for the omission in the definition of the Transformer in Equation 6. To facilitate understanding, we have reorganized the logic and updated the manuscript 3.3.1 in page 5. An explanation is also provided in response to question 2. The OA module does indeed have a single input, which we will clarify in the 3.3.1.

4. FID and similar metrics require ground truth data for feature similarity calculations, which is not applicable to text-to-3D tasks lacking ground truth. We use alternative metrics, as explained in response to question 3, and have elaborated on the rationale in Appendix A.4.2.

5. SVS Module Explanation: We regret the lack of clarity in the initial explanation of the SVS module. The manuscript has been updated to better describe its motivation and implementation. Please refer to our response to your question 2 for further details.

6. Thank you for your suggestion. We have updated the manuscript to clarify references to the OA module where appropriate.

7. Inference Time Comparison:

While our method does not achieve the fastest inference time, it demonstrates strong overall performance. Compared to the state-of-the-art prior-based method Shap-e, our method achieves a 29% speed improvement. Notably, methods like MVDream and DreamView, which are the fastest, are limited to generating only four views, whereas our approach supports arbitrary view generation. Additionally, compared to SPAD, an any-view method with an inference time of 66 seconds, our method achieves a ~30% speed improvement while exhibiting superior multi-view consistency.

8. Explanation of Figures 4 and 5: The results presented in Figure 4 are merely visualizations of the Triplane priors and do not represent the final outputs. Nevertheless, at this stage, the visual quality of the Triplane results is already comparable to those of methods like MVDream, even if the visual appeal may seem less striking, likely due to differences in style.

   Further analysis of Figure 5 reveals that, due to the inherent randomness in Triplane generation, we adopted orthogonal views (frontal, top-down, and side) from real data as the Triplane inputs to ensure consistency across different SVS configurations. Since the data itself leans more toward a realistic style, our results may appear less visually striking compared to the more artistic style of images generated by methods like MVDream.

9. We regret that the generated results do not include the corresponding Triplane priors. Due to the inherent randomness in the model's generation process, each Triplane is different for every run. To address this, we have supplemented additional results incorporating Triplane priors in Appendix A.2.2 and provided detailed comparative results in Figure 9 of Appendix A.2.1. These supplementary materials aim to offer a more comprehensive understanding and validation of our method's performance.

Thank you for your valuable suggestions. We have updated the manuscript to polish some of the details not yet fully explained previously. Here, I will first address the questions you raised:

Weakness:

1. Dataset Quality and Scale: Large-scale, high-quality datasets can significantly enhance model performance. To ensure the acquisition of a sufficiently high-quality, large-scale dataset, we have implemented the following measures:

- We selected Objaverse, the largest open-source 3D dataset available at the time.

- Given the varying quality of the dataset, we conducted necessary data preprocessing before training to filter out low-quality data.

2. Limitations of Computational Resources: Limited computational resources can indeed affect model convergence. To maximize model performance, we adopted the following strategies during training:

- Increased batch size and utilized gradient accumulation to further enhance robustness.

- Randomly sampled generated views during training to improve generalization.

- Applied a dynamic cosine learning rate adjustment strategy based on dataset scale to facilitate better model convergence.

3. Innovation in Semantic Matching Strategy: The core innovations and implementation of our semantic matching strategy are detailed in the response to the subsequent questions.

Questions:

1. The core idea behind semantic matching in SVS is to align semantic features with orthogonalized 3D visual regions. Unlike prior-based methods, which do not incorporate semantic matching during the formation of implicit fields, our approach introduces semantic matching during the construction of the triplane implicit field. During the implicit triplane reconstruction, which leverages the spatial orthogonal relationships of the triplane, attention mechanisms guide the adaptive alignment between semantic and visual features. This process allows semantic features to naturally align with the most likely corresponding visual feature regions, ultimately ensuring precise semantic alignment across various 3D visual regions. More illustration Information is provided in Appendix A.3.

2. The implementation details of this semantic matching mechanism are illustrated in Equation 6 in page 6.

First, we extract the triplane visual features, denoted as $token_{tri}$. These features are then enriched with semantics $T$ through CA, represented as $CA(token_{tri}, T)$. As shown in the figure 5, using $token_{tri}$ directly for reconstruction results in surface blurriness, lack of detailed textures, and significant edge artifacts, such as at the foot of the bed on the left side of the figure. Introducing CA mitigates these edge artifacts due to semantic constraints, although texture details are still lacking.

Then, we apply OA to achieve self-spatial orthogonal alignment, further matching semantic information with orthogonal features. As illustrated in the leftmost column of the figure, after incorporating OA on top of CA, details such as the pillows on the bed are fully generated.

3. Our TPL aims to generate high-quality triplane priors by learning the spatial correspondence among the three planes. This primarily ensures view consistency while also enhancing the semantic consistency of each plane to some extent. As illustrated in the figure 4 of page 10, the introduction of OA not only significantly improves view consistency but also maintains semantic consistency. For instance, the red and gray shoes demonstrate both view and semantic consistency after incorporating OA. In contrast, the method on the left, which does not employ OA, maintains view consistency but loses substantial semantic information.

4. Our method is a feed-forward approach, without any iterative optimization of features throughout the process. Therefore, the output of TPL is directly fed into SVS.

5. Our SVS generates an arbitrary number of multi-view images through ray sampling and rendering from the triplane. Essentially, we enhance the semantic consistency of the multi-view images by improving the semantic consistency of the triplane implicit field, achieved through the functionality outlined in response 1/2.

6. In our main experiment, we have already demonstrated the comparative results of semantic consistency. For instance, in Figure 3 of page 8, the green shoes meet both the high-top design and single shoe constraint, which only our SeMv-3D can achieve. However, as you pointed out, the experimental results are not yet sufficient. Therefore, to more comprehensively illustrate our semantic consistency, we have included additional experimental results in the figure 7 of Appendix A.1 and figure 8/9 of Appendix A.2.1.

Thank you very much for your valuable and constructive feedback. Below, we address each of your points in detail:

1. Benchmark for General Text-to-3D: Due to the lack of a standard benchmark for general text-to-3D tasks, we employed GPT to automatically generate prompts, ensuring a fair and consistent comparison across different methods. In the future, we will publicly release our test prompt set to facilitate result reproduction and further validation by the community.

2. Comparative Results: As you rightly pointed out, it was an oversight to include only our method’s results without comparative analysis in the appendix. We have now addressed this by adding more comparative results in Appendix A.2.1 for a comprehensive evaluation.

3. Quantitative Ablation Studies: While the visual results provide strong evidence for our method's effectiveness, we agree that adding quantitative ablation studies would strengthen our argument. Following your suggestion, we have included the corresponding tables in Appendix A.2.3 to provide a more thorough analysis.

4. Our SeMv-3D method achieves an average inference time of approximately 46 seconds per prompt, representing a ~29% speedup compared to the current state-of-the-art prior-based model, Shap-E, which requires 65 seconds. Furthermore, our method demonstrates superior semantic consistency and enables any-view generation. In comparison to the previous any-view method, SPAD, which has an average inference time of 66 seconds, our method achieves a ~30% speedup while also exhibiting better multi-view consistency. These results highlight that SeMv-3D not only offers significant advantages in semantic and multi-view consistency but also achieves substantial improvements in efficiency.

I appreciate the author's clarification in their rebuttal. I tend to retain my original rating, as the evaluation set used in this paper is not convincing for me. The authors need to evaluate their methods on a standard benchmark, such as T3Bench.

Thank you for recognizing our paper and method, as well as for providing valuable suggestions. In this response, we will first address the weaknesses you highlighted and then provide detailed answers to your questions.

Weakness

1. Regarding the motivation, existing general text-to-3D tasks that support direct inference in a feed-forward manner can be broadly categorized into two types. Prior-based methods, such as Shap-E, excel in view consistency but suffer from poor semantic alignment and subpar generation quality. Fine-tuning-based methods, like MVDream, achieve high semantic consistency and superior quality but are limited by view inconsistencies and the ability to generate only a few views. Our motivation is to propose a method that combines the strengths of both approaches, achieving arbitrary view generation with high multi-view consistency and semantic consistency, while also delivering refined and high-quality results. This forms the foundation of our approach.

   • MVDream primarily relies on fine-tuning diffusion-based models, such as text-to-image or multi-view generation models, without incorporating 3D constraints like NeRF used in Shap-E. Instead, it leverages the diffusion model's capability to directly generate multi-view images, which we classify as fine-tuning-based methods.

     In contrast, ET3D belongs to the category of optimization-based methods, which involve iterative optimization of 3D representations like NeRF. For example, ET3D employs a distillation-based strategy to iteratively refine 3D structures. This fundamental difference from feed-forward general text-to-3D tasks is why ET3D and similar approaches are excluded from our comparative analysis.

   • In our approach, DINO is used solely as a tool for extracting visual features. We chose DINO features due to their ability to preserve rich visual characteristics, which is likely why methods such as Instant3D and GRM also employ them. However, as illustrated in Figure 5, solely relying on features extracted by DINO is insufficient to achieve optimal results.

     Our primary focus is on designing a novel semantic alignment mechanism based on the Triplane features themselves. The detailed principles and implementation of this mechanism can be found in the Sec.3.3.1 and Appendix A.3 of the paper. By integrating visual features with this mechanism, we further enhance the precision and effectiveness of semantic alignment, leading to superior overall performance.

2. Multi-view-based reconstruction tasks essentially initialize with a multi-view setup based on general text-to-3D models like MVDream but focus on image-based 3D reconstruction. They don't align with our general text-to-3D baseline model, although future work on extending SeMv-3D to 3D reconstruction could consider comparisons with these methods.

   • while we have not reasonably compared the efficiency of triplane with new representations like 3DGS, further exploration of 3DGS's efficiency in generalization tasks could be valuable. Nonetheless, the triplane representation significantly reduces computational complexity compared to traditional 3D representations, meeting the needs of our generalization tasks while retaining excellent expressive capability. This efficiency justifies our claim.

Questions:

1. Although our method does not hold an absolute advantage in terms of speed, it still performs well overall. Our SeMv-3D method achieves an average inference time of approximately 46 seconds per prompt, representing a ~29% speedup compared to the current state-of-the-art prior-based model, Shap-E, which requires 65 seconds. Furthermore, our method demonstrates superior semantic consistency and enables any-view generation.

   In comparison to the previous any-view method, SPAD, which has an average inference time of 66 seconds, our method achieves a ~30% speedup while also exhibiting better multi-view consistency. These results highlight that SeMv-3D not only offers significant advantages in semantic and multi-view consistency but also achieves substantial improvements in efficiency.

2. Thank you for pointing this out. We have added the prompts in figure 5. Regarding the effectiveness of CA, as shown in the figure 5, SVS with no CA&OA results in surface blurriness, lack of detailed textures, and significant edge artifacts, such as at the foot of the bed on the right side of the figure. As illustrated in the middle column of the figure, introducing CA mitigates these edge artifacts due to some semantic constraints. It demonstrates that CA has some effect, but its impact is limited when used alone. Our proposed semantic alignment aims to align the 3D visual features with semantic features through CA and OA.