Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation

W1: Lack of Cohesion in Core Contributions, the approach appears fragmented and difficult to generalize.

We generally agree that Flex3D requires multiple components and can be considered complex. The Flex3D pipeline is designed to mitigate suboptimal outputs from the first-stage multi-view diffusion model. This requires considerable effort and component design to achieve high-quality text-to-3D and single-image-to-3D generation.

Although the entire pipeline might be difficult to generalize, each individual component within Flex3D can be easily generalized. For example, the minimalist design of the FlexRM architecture allows for easy implementation based on Instant-3D, and it can directly replace existing feed-forward reconstruction models.

W2 and Q1: Inconsistency Concerns regarding two different models for elevation and azimuth.

The use of two different models for elevation and azimuth leads to minimal conflict. Because there is minimal corresponding pixel overlap between the generated views, multi-view inconsistencies are unlikely, even if the two models generate different content.

Furthermore, since the data used for image pre-training and fine-tuning are identical, the models are likely to generate similar content. For visual confirmation, please refer to Figure 5 (View Selection Visualizations), which presents two examples showing all 20 generated images. The results from the azimuth model are located in the top-left part (views 1-4).

W3: Inadequate Simulation of Multi-View Inconsistencies, no cross-view inconsistencies.

This is an insightful point; thank you for your careful review! Some basic multi-view inconsistencies can be simulated by replacing some clean input images with noisy ones as final inputs. This is the strategy used in Flex3D. To simulate more substantial cross-view inconsistencies, we could inject noise multiple times, each time selecting some noisy views to include in the final input set. This approach would be fast and memory-efficient, as noise injection operates on generated Gaussian points. However, it would also add complexity to the pipeline. Therefore, we leave this as a trade-off option.

W4 and Q3: Lack of Flexibility Analysis.

Please see Table 2 for GSO reconstruction results with different numbers of input views. Overall, increasing the number of input views for FlexRM generally leads to improved reconstruction quality. We found the improvements for object reconstruction to be less significant after 16 input views.

Q2: How does FlexRM handle scenarios where significant view inconsistencies occur?

The view selection pipeline is the primary defense against significant view inconsistencies. After this filtering stage, the input views provided to FlexRM typically contain only minor inconsistencies, which are then handled by imperfect data simulation training used in FlexRM.

Thank you for your thorough review and for taking the time to consider our rebuttal, especially for a second time! We appreciate your acknowledgment of the potential contributions our work offers to other areas, and we're glad you find that aspect meaningful.

We understand your remaining concerns regarding the necessity and impact of the "Input View Curation" mechanism. We are offering this further response not to change your score, but rather to further engate in the discussion and perhaps offer insights for future work in this research area. Please feel free to discuss any topics in this research area further.

We agree that if multi-view diffusion models could synthesize perfectly 3D-consistent views, the view selection pipeline would become unnecessary. However, we believe that until such models are widely available and robust—and considering the limitations we've observed in current multi-view diffusion methods (even across various 4-view diffusion models)—our view curation pipeline remains valuable. We envision it as a practical and beneficial tool for at least the next 3 years, particularly because it can also serve 4D generation frameworks that require consistency in both 3D and temporal dimensions.

Regarding performance gains, while intuitively the improvement from view selection might diminish with future, stronger multi-view diffusion models (although the threshold could be adapted to retain more high-quality views), we have found it to be meaningful across different reconstruction models and video diffusion models. This broader applicability suggests a benefit beyond the specific models developed in this paper. It could also be very beneficial for generative reconstruction frameworks such as Im-3D, Cat3D, and Cat4D, where inconsistent views should be filtered out before fitting a 3D/4D representation. We would like to explore this further in future work.

Thank you again for your time and insightful feedback. We greatly appreciate your review and engagement, and we truly enjoyed discussing this work and border the 3D generation research area with you!

W: What is the inference time required to produce the multi-view image proposals? Does it possible to apply video diffusion mode to generate a trajectory where elevation varies according to the sine function, and azimuth is selected at equal intervals instead of two multi-view diffusion model?

While view selection and FlexRM reconstruction require less than a second on a single A100 GPU, generating 20 views with two diffusion models takes approximately one minute on a single H100 GPU. This speed is similar to video-based multi-view diffusion models like SV3D. We have added a note regarding this in the revised paper.

We agree with reviewer 9shi that applying video diffusion models to generate a trajectory where elevation varies according to the sine function, and azimuth is selected at equal intervals, is both sound and feasible. Thank you for this suggestion! We intend to leave this promising idea in future work.

Q1: Please show me some failure cases, especially for your view-selection method that failed.

We have added failure cases in Figure 8 and Section F in the appendix (page 20) of the revised manuscript. A notable failure occurs when the input image contains floaters or small transparent objects. This leads to incorrect generation results, especially at different elevation angles. The view selection pipeline only partially removes these incorrect results. Additionally, our view selection pipeline can sometimes produce incorrect results, particularly with objects containing thin geometries. These thin components can contribute less to feature matching, making them more susceptible to incorrect selection.

Q2: Missing some Reference.

Thank you for your suggestions! We have added all of them in the revised manuscript.

Thank you for your time and consideration in reviewing our manuscript and rebuttal. We appreciate your feedback and are very pleased to hear that the rebuttal adequately addressed your comments and questions!

W1: There is a general lack of technical insights.

We acknowledge similarities with previous works. This is hard to avoid, especially given the rapid progress in 3D feed-forward models since LRM (ICLR 2024), which already has over 214 citations according to Semantic Scholar.

We'd like to address the comment regarding a lack of insights. The starting point of our work stems from a key limitation not yet addressed in existing research. Two-stage 3D generation pipelines, like Instant3D and many others [1-10], are currently the most popular framework and generally achieve the best performance. However, a significant limitation of all these approaches is that while their reconstructors perform well with sparse-view reconstruction, the final 3D quality remains constrained by the quality of the generated multi-views. Our work directly addresses the challenge of mitigating suboptimal outputs from the first-stage multi-view diffusion model. We incorporate view selection, a flexible view reconstruction model, and noise simulation to resolve this issue, crucial for text-to-3D and single-image-to-3D generation. This is the core insight we want to deliver.

[1] Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. ICLR 2024.

[2] GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation. ECCV 2024.

[3] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. ECCV 2024.

[4] LGM: Large Multi-view Gaussian Model for High-Resolution 3D Content Creation. ECCV 2024.

[5] CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction. ECCV 2024.

[6] Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials. NeurIPS 2024.

[7] LRM-Zero: Training Large Reconstruction Models with Synthesized Data. NeurIPS 2024.

[8] GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation. NeurIPS 2024.

[9] InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-View Large Reconstruction Models. arXiv 2024.

[10] Tencent Hunyuan3D-1.0: A Unified Framework for Text-to-3D and Image-to-3D Generation. arXiv 2024.

W2-6: Many components are proposed before or only bring marginal improvements.

Regarding specific technical components, we highlight our key differences and improvements compared to previous work:

FlexRM (Stage 2): FlexRM, unlike previous works [11, 12] that rely on a position prediction branch to determine 3D Gaussian positions, directly determines 3D Gaussian positions from the tri-plane representation using our proposed designs. Furthermore, FlexRM can process up to 32 input views and demonstrates significantly stronger performance than [11, 12].

Multi-view image generation (Stage 1): Our contribution involves generating novel views with separate models for elevation and azimuth angles, minimizing multi-view conflicts and achieving better results compared to single-model approaches like SV3D [13] (We are happy to provide further visual results if necessary). Besides this, our proposed view curation pipeline effectively removes suboptimal or 3D-inconsistent generated views, improving the final 3D asset quality.

Camera Conditioning (Stage 2): This minor design choice improves handling of multiple input views, especially since reconstruction models are trained with a varying number of them. As shown in lines 500-504 in the revised manuscript, the benefits of stronger camera conditioning become more effective with a larger number of input views. This modification is very simple and introduces negligible computational overhead.

Imperfect Data Simulation (Stage 2): This novel contribution enhances FlexRM's robustness to minor imperfections in generated multi-view images, improving performance, particularly in generative tasks. While gains are marginal for reconstruction tasks (Table 5, right side), they are non-marginal for generation tasks (Table 5, left side), where the CLIP text similarity score increases from 27.1% to 27.7%. This metric is typically very close (<2% difference) among different methods.

[11] Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers, CVPR 2024.

[12] AGG: Amortized Generative 3D Gaussians for Single Image to 3D, TMLR 2024

[13] SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion, ECCV 2024.

W7: No computational cost analysis.

We fully concur that reporting computational cost is essential. Thank you for raising this point! Our initial submission included runtime information for some pipeline stages. For instance, we have reported that view selection takes less than a second, and FlexRM generates 1 million Gaussian points in under 0.5 seconds and renders in real-time, comparable to current feed-forward reconstruction models.

Generating 20 views using two diffusion models takes approximately one minute on a single H100 GPU. This speed is comparable to that of video-based multi-view diffusion models like SV3D. We have added a note about this in the revised paper. The revised paper now includes the runtime of the entire pipeline.

Q: Insights for follow-up research.

For follow-up research, the key insight is that we introduced two ways to handle imperfect multi-view synthesis results in a common two-stage 3D generation pipeline: view selection to reduce input errors, and noise simulation to train a robust reconstructor. Each individual component within Flex3D can be easily adopted by future works. For example, the minimalist design of the FlexRM architecture allows for easy implementation in frameworks like Instant-3D, and it can directly replace existing feed-forward reconstruction models.

Q: A much larger improvement in performance expected.

Our evaluation is quite extensive, encompassing both generation and reconstruction tasks. We believe our results on generation represent a significant improvement. We compared our proposed Flex3D with seven recent and strong baselines. In our user study, conducted in a fair environment with randomly selected samples for comparison, Flex3D outperformed all other methods, achieving at least a 92.5% win rate.

Ethics concerns on data.

We thank the reviewer for raising this important point. We understand these concerns and take ethics and IP rights extremely seriously.

The data used in this paper was not obtained by scraping the internet. The data was purchased from a well-respected and widely-known vendor of 3D graphic assets. We acquired a data license that explicitly allows use of the models in machine-learning applications, including all applications in this paper. We follow the terms and conditions of this license scrupulously, also based on internal legal advice. The exact commercial details of the license are, as it may be expected, confidential.

Many thanks for taking the time to review the other reviewers' comments and our rebuttal! We sincerely appreciate you providing further feedback, and we especially value your constructive criticism regarding the insights and broader implications of this paper. We also thank you for summarizing the other reviewers' perspectives.

As no further comments regarding computational cost, performance, and ethical concerns regarding the data have been mentioned, we would like to assume that these concerns have been adequately addressed. Please feel free to raise any concerns in them if you feel our rebuttal did not sufficiently address them.

Regarding your further comments, we would like to offer a response:

Q1: Were these "tricks" applied to a number of existing techniques, and would carry consistent benefits, I would have been more keen to see the paper published.

Thank you for your transparency. We believe that all proposed "tricks" or components in this paper have been fully validated through extensive experiments and ablation studies.

Our experiments cover 3D generation and reconstruction tasks, and for each task, we have included more, or on-par, metrics, baselines, and test set sizes compared to previous research on 3D feed-forward generation models. This provides strong evidence to demonstrate the effectiveness of our proposed Flex3D pipeline and FlexRM model.

Regarding the effectiveness of all components, we have included very detailed ablation studies. Their impact is further validated through rigorous ablations. For instance, we studied and reported the results of four design choices in FlexRM, three design choices in the candidate view curation and generation pipeline, and the noise injection pipeline, with detailed ablation results reported for each. During discussions with other reviewers, we also added many new experiments to further validate and support the effectiveness of our proposed components.

Please feel free to leave any further comments if the effectiveness of any proposed component requires further clarification. We are fully committed to addressing them promptly.

Q2: I don't see any indication of code, data, or trained model being released.

The open-source plan is under discussion, and we are awaiting further guidance on this matter. We would like to clearly outline the potential outcomes to help you assess the impact of open-sourcing.

The raw data will not be released as it was acquired for internal use only. Furthermore, the two fine-tuned multi-view diffusion models are also unlikely to be released. However, we may be able to open-source the weights of the reconstruction model (FlexRM), the code for running FlexRM, and the code for the view selection pipeline.

We would also like to note that many advanced models in recent years, particularly generative models, are not open-sourced due to various complex reasons, and often this decision is not made by the authors. While authors may hope to use public training datasets and public models to build new techniques, this is extremely difficult due to various reasons. Open-sourcing should not be a factor in diminishing a paper's contribution if sufficient re-implementation details are provided. Otherwise, it is very unfair to researchers who cannot freely conduct open-sourcing, and this goes against ICLR's wishes, as it is a premier gathering of professionals dedicated to the advancement of artificial intelligence. Researchers who cannot freely open-source data and models should not be excluded.

Excluding the fine-tuning details of the two multi-view diffusion models, we have included very detailed implementation and training information to facilitate re-implementation. This includes the proposed view selection, FlexRM, and noise injection methods. We provide re-implementation information comprehensively; for example, we have included the 3D Gaussian parameterization details.

I have taken my time to review the other reviewer's opinions, which I summarized at the end of this message. From what I can see, with the exception of Sr7a (and 9shi who I filter out as an outlier given the short review), the other reviewers are telling a similar story.

I agree with their suggestions as well. Were these "tricks" applied to a number of existing techniques, and would carry consistent benefits, I would have been more keen to see the paper published. But as we stand, these are improvements to a closed system that will be complex/impossible to reproduce, especially as I don't see any indication of code, data, or trained model being released.

Which leads me back to my original point... what is the academic going to take away from this paper? Therefore, unless another reviewer champions the paper, and convinces me that it will be a mistake to not have this paper published in its current form, I am very likely to lower my score to reject.

pAxN (marginally below, confident)
- core contribution is relatively straightforward
- demonstrate the degree to which this idea directly and significantly improves the two-stage pipeline

f8fj (marginally above, confident)
- optimizing existing methods rather than introducing fundamentally novel concepts
- questions remain regarding the scalability and broader applicability of the complex multi-stage pipeline

zKWm (marginally below, certain)
- you may apply the view selection to baseline methods to check whether there are consistent improvements
- use the same selected multi-view images to evaluate different reconstruction model

Sr7a (marginally above, certain)
- (offers to raise conditional on 3D metrics and datasets)

oAc3 (marginally below, confident)
- myself

9shi (marginally above, certain)
- review contains little insights

Thank you again for your willingness to engage in further discussion promptly.

We have been fully transparent about our open-sourcing plan, outlining what may be open-sourced, our confidence levels, what cannot be open-sourced, and our proposed alternative. Given that this is an open review process, and even if reviewers have reservations about the authors' claims, we have no reason to make any false statements.

Open-sourcing any material relevant to this work requires approval. We are currently in the process of acquiring approval for open-sourcing, and we are also exploring the possibility of sharing code confidentially in the coming days. If permitted, we will provide an anonymous link to the code for reviewers, ACs, and PCs only. Finally, we wish to clarify that attaching code is not a mandatory requirement for ICLR or CVPR; however, we are making every effort to accommodate your request in this regard.

Many thanks for your willingness to engage in further discussion, and thank you again for your transparency! We also want to express our gratitude for your responsible reviewing and your contributions to fostering open-source development within the academic community.

We do hope to open-source everything, ideally by early next year (Jan or early Feb). Following the double-blind review policy, we cannot disclose any author information at this time. However, we can state that all authors involved in this work are active and well-engaged open-source contributors in the research community, contributing both codes of research papers and open-source public packages/libraries. Furthermore, the first author of this work has open-sourced all previous single-first-authored papers and maintains good GitHub practices, replying to almost all issues. This can be easily verified once the author information is made public after the paper decision, so we are completely truthful here.

Regarding the open-source plan, as stated before, it is still under discussion and we are waiting for further guidance. However, we are fairly confident (with >80% probability) that we will be able to open-source the weights of the reconstruction model (FlexRM), the code for running FlexRM, the code for the noise simulation pipeline, and the code for the view selection pipeline. Therefore, to reproduce all the performance results of Flex3D, the only missing components would be the two multi-view generation diffusion models. We should be able to provide an unofficial alternative using other multi-view diffusion models, such as SV3D, as a replacement. However, we must acknowledge that the performance in generation tasks might slightly decrease as a consequence.

Although we cannot 100% guarantee the open-source plan at this moment, everything stated here is asserted with high confidence. We hope this addresses your concerns regarding open-sourcing!

Although the effectiveness of each trick has been fully validated in the current framework, we are exploring their generalization ability across additional frameworks.

Specifically, we will test (1) a stronger camera condition and (2) view selection by applying them to a variant of the Instant-3D reconstructor. This variant will be trained using between 1 and 16 views as inputs, rather than a fixed four views. Training will begin soon, and we will post the results before the discussion period concludes.

We will also test the proposed view selection pipeline when applied on top of SV3D's generated views, where we will report (3) text/single-image to 3D generation results when applied to SV3D + FlexRM.

W1: The paper does not specify whether the proposed method has been tested across various datasets or object categories.

Flex3D primarily focuses on text- or single-image-based 3D generation, while FlexRM supports various 3D reconstruction tasks. For generation, we used a diverse set of 404 DreamFusion prompts encompassing a wide range of objects, scenes, styles, and abstract descriptions—a relatively large experiment compared to many prior works. For reconstruction, we tested on 947 real-world scanned objects from the GSO dataset, covering common object categories (excluding a few highly similar shoes to avoid redundancy). This test set is also larger than those used in many previous works, which often employ fewer than a few hundred GSO samples.

To further validate FlexRM's robustness across diverse data distributions, we tested it on Blender-rendered views of 500 hand-crafted 3D objects from our internal dataset (similar to Objaverse). This validation set was held out from training. Similar to the GSO procedure, we rendered 64 views per object at four elevation degrees (-30, 6, 30, and 42 degrees), with 16 uniformly distributed azimuth angles per elevation. Results are presented in Table 6 (page 18) of the revised manuscript, showing similar trends to the GSO results.

Finally, we tested FlexRM's robustness to noisy input images as a more diverse case. The results are presented in our response to reviewer f8fj (Q1: Results on robustness across various poses, view counts, and noise levels.).

W2: Adding these 3D metrics would provide a more complete understanding of the method's performance.

We agree that incorporating more 3D metrics leads to a more complete evaluation—thank you for highlighting this! We further report the Chamfer Distance and Normal Correctness for 20,000 points uniformly sampled on both the predicted and ground-truth shapes in the GSO experiment. Results are presented in Table 2 of the revised manuscript. FlexRM continues to outperform other baselines in the 3D evaluation metrics, by clear margins, and we also observe improved results with more input views.

W3: Providing these details would enhance the credibility of the user study findings.

We provide additional details about the user study described in lines 443-450 of the revised manuscript. Our study design generally follows previous work [1-3].

Participant Demographics: Five computer vision or machine learning researchers participated in the evaluation. Two were from the US, two from Europe, and one from Asia. Two participants actively work on 3D content generation.

Methodology: Participants viewed paired 360° rendered videos—one generated by Flex3D and one by a baseline method—presented via a Google Form. Video pairs were presented in random order and randomized left/right positions. Participants selected the video they preferred based on overall visual quality.

Statistical Significance: We collected 1000 valid results (5 participants * 5 baselines * 40 videos). Although the sample size is relatively small, Flex3D was preferred in at least 92.5% of comparisons across all five baselines, strongly suggesting superior visual quality.

Update: During the rebuttal period, we added comparisons with two direct 3D diffusion baselines using the same user study setup, bringing the total number of valid results to 1400.

[1]: IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation, ICML 2024.

[2]: Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning, ECCV 2024.

[3]: VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models, ECCV 2024.

W1: Lacks sufficient detail and analysis regarding the data labeling process, classifier performance, and justification for the chosen selection strategies.

Regarding video quality selection criteria, our Emu-generated videos contain 16 frames each. Our criteria are based on the overall visual quality of the generated multi-view videos, emphasizing multi-view consistency and visual appearance. For labeling, the authors first carefully labeled approximately 100 sample videos. These were then provided to two labelers, and each labeler was asked to label approximately 1,000 videos, resulting in a total of 2,000 labeled videos. A sample size of 2,000 might be considered small. This number follows the setting used in Instant-3D's data curation pipeline. Similar to Instant-3D, we find that using strong pre-trained image feature encoders like DINO allows 2,000 samples to be sufficient for training a robust classifier using SVM. Other pre-trained image feature extractors, such as CLIP, can also be effective, but DINO proved more effective in our experiments. We found that SVM converges more easily than neural net-based linear classifiers. To further verify the success rate, we manually labeled another 100 videos and applied the trained classifier. It achieved aligned results for 93 videos, which we consider a high success rate.

For the consistency selection model, we chose EfficientLoFTR [1] because it represents the state-of-the-art in real-time keypoint matching. In multi-view geometry, 3D consistency primarily relies on 2D correspondences established through keypoint matching. Over the past few decades, keypoint matching methods have evolved from classical approaches like SIFT combined with nearest neighbor search to modern deep learning techniques such as SuperPoint [2] with SuperGlue [3], and EfficientLoFTR. These methods have proven reliable and effective and are extensively used in 3D reconstruction tasks like structure from motion. In our evaluations, EfficientLoFTR performed best, producing all results in under a second. While other viable options exist, such as SuperPoint with SuperGlue, they often do not offer the same optimal balance of speed and accuracy.

[1] Efficient LoFTR: Semi-dense local feature matching with sparse-like speed. CVPR 2024.

[2] Superpoint: Self-supervised interest point detection and description. CVPRW 2018.

[3] Superglue: Learning feature matching with graph neural networks. CVPR 2020.

[4] LoFTR: Detector-free local feature matching with transformers. CVPR 2021.

W2: A comparison with selection via MLLMs like GPT-4V.

This is a great idea; thank you for suggesting it! We tested it using Gemini 1.5 Pro 002, which may be more powerful than GPT4V due to its long context window, allowing us to input all 20 frames directly. Our prompt was: “You are an expert in 3D computer vision. I am providing you with 20 generated multi-view images. Please help me identify the frames that are consistent with the first frame and present high visual quality. Don't be too strict, as minor inconsistencies are acceptable.” This resulted in a total token count of 5,224, much smaller than Gemini 1.5 Pro 002's context window (2M).

We tested 20 samples. Generally, Gemini can understand the task and make reasonable selections, but its performance is not yet as good as our proposed pipeline. This might be because current MLLMs primarily rely on CLIP embeddings to connect visual modalities, making pixel-level perception difficult.

We describe two specific selection results from two sequences used in Figure 5, showing all 20 frames. For the "ramen" example, Gemini selected frames [1, 5, 7, 13, 17], while our pipeline selected [1, 2, 3, 5, 6, 10, 11, 12, 13, 18, 19, 20]. Compared with our pipeline, Gemini rejected frames [2, 3, 6, 10, 11, 12, 18, 19, 20] and selected [7, 17]. Upon careful inspection, frames [2, 3, 6, 10, 11, 12, 19, 20] should be considered high-quality and retained. In Gemini's selected frames [7, 17], the chopsticks are either missing or blurry and should be removed.

For the "robot and dragon playing chess" example, Gemini selected frames [1, 2, 3, 5, 6, 8, 9, 11, 12, 14, 15, 17, 18, 20], while our pipeline selected [1, 2, 3, 4, 5, 6, 12, 13, 14, 20]. Compared with our pipeline, Gemini rejected frames [4, 13] and selected [8, 9, 11, 15, 17, 18]. Frames [4, 13] should be considered high-quality and retained. In Gemini's selected frames [8, 9, 11], the robot's eyes are incorrect, either missing or merged. In frames [15, 17, 18], the dragon's head is incorrect, displaying an unusual shape or blue color.

In conclusion, while MLLMs are not yet as effective or efficient as our proposed pipeline (where our proposed pipeline also runs faster, in under a second), using them for view selection holds strong potential. Future research could address these limitations through fine-tuning to improve pixel-level perception and 3D awareness. We leave this interesting idea for future work.

W3: There is a lack of comparison with diffusion-based baselines that predict 3d via 3d diffusion or 3d dit directly.

We've added results from two recent, open-source, direct 3D diffusion models for comparison: LN3Diff [1] and 3DTopia-XL [2]. Using their official code, we generated 3D assets via their single-image-to-3D pipelines. This allows more controlled generation, and was necessary because the text-to-3D pipeline is not currently available for 3DTopia-XL. We've updated Section 4.1 on 3D generation, including Figure 4 and Table 1. Overall, Flex3D significantly outperforms these direct 3D baselines, achieving considerably higher CLIP scores and demonstrating a 95% and 97.5% win rate in user studies.

[1] LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation, ECCV 2024.

[2] 3DTopia-XL: High-Quality 3D PBR Asset Generation via Primitive Diffusion, arXiv 2024.

W4: Which stage does the improvement mainly come from?

We would state the improvement comes slightly more from the multi-view generation and selection stage. However, to utilize the selected multi-views with a varying number for reconstruction, a reconstruction model capable of handling any number of input views (like FlexRM) is necessary.

W4: In Fig.4, and table 1, do the baselines use the same multi-view images as the proposed method?

No, Figure 4 and Table 1 present the results of text-to-3D generation tasks. The input for each method is a text prompt or a single image generated from a text prompt.

W4: Evaluating two stages separately:

We agree that isolating the contributions of the multi-view generation and selection stage versus the FlexRM reconstruction model would provide valuable insights.

Multi-view generation and selection: Since other reconstruction models are trained with a fixed number of views (e.g., LGM-4, GRM-4, InstantMesh-6, Instant3D-4, LRM-1, VFusion3D-1), directly adapting them to variable view settings leads to performance degradation, hindering a clear comparison of our view selection strategy. Therefore, we included a detailed ablation study (Table 4) to demonstrate the effects of our proposed multi-view generation and selection pipeline.

FlexRM reconstructor: The reconstruction task (where the input views are identical), presented in Section 4.2, shows detailed reconstruction results in different settings with comparisons to representative baselines. FlexRM consistently outperforms other baselines, achieving the best results across different input view settings.

W5: For ablation results like table 3,4,5, do you use Blender rendered images or generated images as the multi-view condition?

We used two distinct settings for ablations: one for reconstruction (Table 3 and the right side of Table 5) and one for generation (Table 4 and the left side of Table 5). For reconstruction, we rendered GSO scanned objects in Blender. For generation, we used generated images and text-prompt as the condition.

W5: How about metrics in table 5 using GT multi-view images rather than generated multi-view images?

Concerning the metrics in Table 5, the right side (reconstruction results) utilizes GT multi-view images for evaluation. The reported results are averaged across experiments using 1, 4, 8, and 16 input views using GSO (scanned objects) as benchmark. Table 5 demonstrates that our data simulation leads to a reasonable performance improvement in generative tasks and a marginal improvement in reconstruction tasks.

W5: Could the data simulation address the domain gap of data?

While the primary purpose of the data simulation is to enhance FlexRM's robustness to minor imperfections in generated input multi-view images, thereby improving its performance in generation tasks, it also indirectly addresses the domain gap. By increasing the diversity of the input data during training, the data simulation can slightly mitigate domain gap issues. However, we have to state that it is not a primary solution for large domain gaps and using it alone won't fully resolve such issues.

Q1: How many views are used for calculation metrics for Flex3d in Table 1? More than baseline methods? If so, is the comparison fair?

Table 1 presents results for text-to-3D generation tasks, where the input for each method is a text prompt or a single image generated from a text prompt. The number of views used for metric calculation (CLIP and Video-Clip) is identical (240) for all methods’ rendered video results. The comparison is consistent with many established practices in this field and is fair.

Thanks for your responses to my questions. My concerns about the implementation details have been addressed. However, I agree with other reviewers that this paper does not bring many new things to the community. So, I would keep my scores for the current version. Thanks for your efforts and active responses. Good luck!

W1: Lack of novel concept or unique contribution.

We acknowledge similarities with previous works, especially given the rapid advancements in 3D feed-forward models since LRM, which, despite being just published at ICLR 2024, has already garnered over 214 citations according to Semantic Scholar. We'd like to address the comment regarding the lack of novel concept or unique contribution. The starting point of our work stems from a key limitation not yet addressed in existing research. Two-stage 3D generation pipelines, like Instant3D and many others [1-10], are currently the most popular framework. However, a significant limitation of all these approaches is that while their reconstructors perform well with sparse-view reconstruction, the final 3D quality remains constrained by the quality of the generated multi-views. Our work directly addresses the challenge of mitigating suboptimal outputs from the first-stage multi-view diffusion model. We incorporate view selection, a flexible view reconstruction model, and noise simulation to resolve this issue, crucial for text-to-3D and single-image-to-3D generation. This is the core idea we want to convey, and we believe it is a novel concept.

[1] Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. ICLR 2024.

[2] GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation. ECCV 2024.

[3] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting. ECCV 2024.

[4] LGM: Large Multi-view Gaussian Model for High-Resolution 3D Content Creation. ECCV 2024.

[5] CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction. ECCV 2024.

[6] Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials. NeurIPS 2024.

[7] LRM-Zero: Training Large Reconstruction Models with Synthesized Data. NeurIPS 2024.

[8] GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation. NeurIPS 2024.

[9] InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-View Large Reconstruction Models. arXiv 2024.

[10] Tencent Hunyuan3D-1.0: A Unified Framework for Text-to-3D and Image-to-3D Generation. arXiv 2024.

W1: The approach appears to rely heavily on integrating and optimizing pre-existing technologies.

Regarding specific technical components, we highlight our key differences and improvements compared to previous work:

FlexRM (Stage 2): Previous works [11,12] rely on a position prediction branch to determine 3D Gaussian positions, but FlexRM can directly determine 3D Gaussian positions from the tri-plane representation using our proposed designs. Furthermore, FlexRM can process up to 32 input views and demonstrates significantly stronger performance than [11,12].

Multi-view image generation (Stage 1): Our contribution involves generating novel views with separate models for elevation and azimuth angles, minimizing multi-view conflicts and achieving better results compared to single-model approaches like SV3D [13] (We are happy to provide further visual results if necessary). Besides this, our proposed view curation pipeline effectively removes suboptimal or 3D-inconsistent generated views, improving the final 3D asset quality.

Camera Conditioning (Stage 2): This minor design choice improves handling of multiple input views, especially since reconstruction models are trained with a varying number of them. As shown in lines 500-504 in the revised manuscript, the benefits of stronger camera conditioning become more effective with a larger number of input views. This modification is very simple and introduces negligible computational overhead.

Imperfect Data Simulation (Stage 2): This novel contribution enhances FlexRM's robustness to minor imperfections in generated multi-view images, improving performance, particularly in generative tasks. While gains are marginal for reconstruction tasks (Table 5, right side), they are non-marginal for generation tasks (Table 5, left side), where the CLIP text similarity score increases from 27.1% to 27.7%. This metric is typically very close (<2% difference) among different methods.

[11] Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers, CVPR 2024.

[12] AGG: Amortized Generative 3D Gaussians for Single Image to 3D, TMLR 2024

[13] SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion, ECCV 2024.

W2: Complex Pipeline Requiring Extensive Fine-Tuning and Training.

We fairly agree the pipeline does require extensive fine-tuning. Other than fine-tuning, the FlexRM architecture is designed with a minimalist philosophy and can be easily reproduced based on Instant-3D. We have included details of FlexRM training at all stages to facilitate full reimplementation. For multi-view generation, SV3D can be used as an alternative, though performance may be slightly reduced.

W3: Performance Concerns Relative to Complexity.

Since CLIP-based metrics can be unstable for 3D evaluation, we place greater emphasis on the user study results and qualitative comparisons. Our method outperformed all baselines with at least a 92.5% win rate in the user study. Furthermore, our anonymous webpage showcases over 60 3D videos and 15 interactive 3D Gaussian Splatting results, demonstrating the qualitative strengths of our approach. We are happy to provide further comparison videos with baselines in the anonymous webpage if needed.

Q1: Results on robustness across various poses, view counts, and noise levels.

Poses: To test this, we ran a GSO single-view reconstruction experiment with different input viewing angles. In addition to the front view reported in the main paper, we tested left, right, and back views. The results below show that FlexRM is robust to different poses and achieves consistently good performance.

View counts: For more results on varying view counts (single-view, 4-view, 8-view, 16-view, 24-view, and 32-view), please refer to Table 2. Overall, FlexRM exhibits robustness with different numbers of input views, and increasing the number of input views generally leads to improved reconstruction quality.

Noise levels: Robustness to noise was evaluated using GSO 4-view reconstruction. Different levels of Gaussian noise (standard deviations of 0, 0.01, 0.05, and 0.1) were added randomly to two of the input images. The results demonstrate that FlexRM is robust to small amounts of noise and maintains good performance even with moderate noise levels:

The presented GSO reconstruction results should be sufficient to demonstrate the robustness of FlexRM. Qualitative results can be provided in the supplementary material if needed.

Q1: How much inconsistency can be tolerated during the multi-view selection process?

We set a hyperparameter (a matching point count threshold) to make our view selection pipeline compatible with different multi-view generative models. The default threshold is the mean minus 0.6 times the standard deviation (as described in lines 250-252 of the main paper). This threshold generally balances removing significantly inconsistent views and retaining those with minor imperfections. It can be adjusted. A lower threshold is more lenient, keeping more views (and thus tolerating more inconsistency). A higher threshold is stricter, discarding more views to ensure greater consistency among the selected subset.

Q2 and Q4: Does the performance continue to improve as the number of views increases? How does the processing time scale with more views? What strategies could be used to efficiently handle a greater number of input views? Where are the results for the 32-view test reported in line 489?

We have added 24-view and 32-view results to Table 2 in the revised manuscript. We found the improvements for object reconstruction to be less significant after 16 input views. Processing time scales only slightly with the number of input views; even with 32 views, FlexRM can still generate 1M 3D Gaussian points in less than a second. To further enhance efficiency, one strategy could be to redesign the network architecture, for instance, by using SSMs (State-space models) to replace the transformer network.

Q3: It could be confusing if the notation for f in line 294 differs from f in line 288.

Thank you for pointing this out! We have added the missing MLP to the equation in the revised manuscript to avoid ambiguity.

Q5: What would the outcome be if the selected views were used for a NeRF-based approach, similar to Instant3D?

Using a tri-plane NeRF, compared to our designed direct tri-plane 3DGS, results in performance degradation. In GSO single-view reconstruction performance degrades by approximately 2 dB PSNR. For 4-view sparse reconstruction, the degradation is more substantial, approximately 3 dB PSNR. This highlights the effectiveness of our direct tri-plane 3DGS design.

Q6: Why are the two-stage training and imperfect input view simulation conducted as separate processes?

This is an interesting point! Two-stage training and imperfect input view simulation can indeed be combined. Separating them allows the simulation to be optional. This is beneficial for models focused solely on reconstruction, where simulating imperfect views isn't necessary.

We sincerely appreciate the time and effort you invested in providing such detailed and thoughtful feedback on our manuscript! We are grateful for your careful reconsideration of our paper, taking into account both our rebuttal and the other reviews. We are pleased that most concerns have been well-addressed by the rebuttal and that valuable insights have also been provided.

We understand and concur with your assessment regarding the complexity of the pipeline. We will keep these points in mind as we move forward with future iterations of this work. Thank you once again for your invaluable feedback!

Dear Reviewer f8fj,

We apologize for the inconvenience. We would like to thank you once again for carefully reviewing our submission, rebuttal, and the other reviews! Your review clearly reflects a very good understanding of this submission and related works, and we find many of your questions insightful. We kindly ask if you would consider reconsidering your confidence score. We feel a confidence score of 3 does not fully reflect your expertise in this area and the effort you put into reviewing this submission.

Warm regards, The Authors

Dear Reviewers:

Once again, we sincerely appreciate the time and effort you have dedicated to reviewing our paper!

As the discussion period concludes in two days, we would be grateful if you could review our rebuttal at your convenience, should your schedule allow. If there are any further points requiring clarification or improvement, please be assured that we are fully committed to addressing them promptly. Thank you once again for your invaluable feedbacks to our research!

Warm regards, The Authors