Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation

We thank you for your thorough review and constructive feedback. We are encouraged by your comments on our pipeline's clarity, the tri-plane + Gaussian Splatting design's performance and potential impact, and the solution's coherence for common two-stage 3D generation pipelines. We provide our responses below, addressing the specific weaknesses and questions raised.

1:Limited novelty.

Two-stage 3D generation pipelines, such as Instant3D and many others, represent a popular and effective class of frameworks. However, a significant limitation of all these approaches is that while their reconstructors perform well with sparse-view reconstruction, the final 3D quality remains constrained by the quality of the generated multi-views. Our work directly tackles the challenge of handling suboptimal outputs from this initial staage. To achieve this, we propose three novel key methods including view selection, a flexible view reconstruction architecture, and noise simulation. We believe both the core concept of mitigating first-stage limitations and the specific proposed methods possess some novelty. For instance, the view selection process utilizes 3D geometric priors, and our reconstruction model, combining tri-planes with 3DGS, offers both speed and the flexibility to handle varying numbers of input views.

2: View selection performance.

Please see our response to Reviewer kNaj on point 1.

3: Compare view selection pipeline to multimodal LLM-based filtering.

Using the IoU metric, we compare our performance with three MLLMs: GPT-4o, GPT-4o mini, and Gemini 1.5 Pro. Their performances are 0.64, 0.49, and 0.57, respectively, all worse than ours (0.72). For the "ramen" example in Figure 5, our pipeline selected [1, 2, 3, 5, 6, 10, 11, 12, 13, 18, 19, 20] . In comparison, GPT-4o selected [1, 2, 5, 6, 10, 12, 13, 17, 18], GPT-4o mini selected [1, 5, 7, 10, 11, 13, 17, 18], and Gemini selected [1, 5, 7, 13, 17]. Compared with our pipeline, Gemini rejected frames [2, 3, 6, 10, 11, 12, 18, 19, 20] and selectedbad frames [7, 17] where chopsticks are missing or blurry. GPT-4o mini also selected these bad frames [7, 17] while missing several high-quality frames like [2, 3, 6, 12, 19, 20]. GPT-4o performed better, selecting mostly high-quality frames, but still missed potentially useful views like [11, 19, 20].

In conclusion, while MLLMs are not yet as effective and efficient as our proposed pipeline for this specific task, using them for view selection holds strong potential.

4: More direct analysis of view selection, how errors in filtering affect final 3D quality, how sensitive?

Although our view selection pipeline is generally strong, achieving 93% accuracy for back view assessment and 0.72 IoU for overall view selection, errors can negatively impact the final 3D quality. Table 4 shows how view selection affects final quality quantitatively. Generally, incorporating bad views degrades the quality of the final 3D output, and the strength of this effect tends to be related to the number of good views. For example, when a larger number of high-quality views are selected as input, the negative impact of incorporating a poor view tends to be less significant. Missing a high-quality view also degrades the final 3D output quality; similarly, this impact is less significant when many other good views are already included.

5: User study details.

Participant: Five computer vision or machine learning researchers participated in the evaluation. Two were from the US, two from Europe, and one from Asia.

Methodology: Participants viewed paired 360° rendered videos—one generated by Flex3D and one by a baseline method—presented via a Google Form. Video pairs were presented in random order and randomized left/right positions. Participants selected the video they preferred based on overall visual quality.

Statistical Significance: We collected 1400 valid results (5 participants * 7 baselines * 40 videos). Flex3D was preferred in at least 92.5% of comparisons across all 7 baselines, strongly suggesting better visual quality.

6: No 3D metrics for evaluation. How does the view selection process improve geometry consistency?

In Table 2, we reported 3D metrics for the reconstruction task, including Chamfer Distance and Normal Correctness, where our FlexRM model clearly outperforms other baselines. Evaluating geometry consistency for generation tasks is challenging due to the absence of GT. We thus employ VGGSfM as a proxy to assess the 3D quality of the generated models. Specifically, we render 16 views covering a 360° azimuth from the generated 3D Gaussians and measure the success rate of VGGSfM in estimating poses (correct poses for at least 8 views). Among 404 results, our full pipeline with view selection achieves a 65.6% success rate, higher than the 59.6% rate obtained without it. This confirms that our view selection strategy improves geometric consistency.

Thank you for your thorough review and insightful feedback. We are encouraged by your recognition of our core view selection strategy as interesting and novel, and we appreciate you noting its potential to improve reconstruction quality by filtering inconsistent views. We also value your positive comments regarding the paper's clarity, motivation, and its potential to serve as a useful baseline for future 3D generation research alongside related works. We address each specific weakness and question below.

1: Performance of view selection module, report classification accuracy.

To evaluate our view selection module, we first manually labeled 404 videos generated by our multi-view EMU model (from deduplicated prompts from DreamFusion). We manually established a ground truth set (GT_Set) for each video. To mitigate subjective bias in determining the absolute 'best' views among all 20 frames, we focused on selecting approximately 10 clearly high-quality views per video to serve as the GT_Set. The labeling process is similar to that described in our response to Reviewer PAZq (2.2: Can we trust manual labeling?). The authors first carefully labeled 20 sample videos. The remaining videos were then assigned to two labelers for annotation.

We used the Intersection over Union (IoU) metric to evaluate the quality of view selection per video. For each video, we compared the ground truth set (GT_Set) with the set of views selected by our model (Selected_Set). The IoU is calculated as:

IoU = |GT_Set ∩ Selected_Set| / |GT_Set ∪ Selected_Set|

Our model achieves an average IoU of 0.72. In terms of accuracy, treating results with an IoU > 0.6 as accurate, we achieve an accuracy of 88.6%. This result indicates strong performance and demonstrates the effectiveness of our proposed selection module, which also operates in near real-time.

2: The SVM is only trained with 2000 labeled samples, will such a small number of data enough for the SVM to generalize to different generation cases?

Yes, we found 2,000 labeled samples to be sufficient for training an accurate filter. This sample size also aligns with practices in related work, such as the data curation pipeline in Instant-3D. Its effectiveness stems from leveraging powerful pre-trained image encoders (DINO).

To validate accuracy and robustness, we tested on 100 held-out videos from our method and 100 from SV3D (out-of-distribution). The classifier achieved 93% accuracy on our videos and 90% on SV3D's. These high rates, especially on OOD data, show the 2,000 samples were sufficient and the filter is robust.

3: If the back view is not selected, could the model still works well with only side view as input?

Yes, our pipeline functions effectively even without selected back views.

Selection Module: Our filter achieves high accuracy (>90%). If it excludes the back view, this typically indicates poor generation quality for the back side. The module correctly removes it and tends to retain other, higher-quality views (often front/side), which is crucial for the final 3D quality.

Reconstruction Model: Our FlexRM model is trained to handle a variable number and arbitrary combination of input views. While it can generate a full 3D object even with fewer or missing views (e.g., no back view), the final reconstruction quality definitely benefits from having more, high-quality input views. This is precisely why the selection module is valuable.

Therefore, filtering out poor-quality back views and feeding the reconstruction model with the remaining, better views leads to higher final 3D quality compared to forcing the inclusion of the low-quality one. The CLIP text sim on 404 DreamFusion prompts are 0.277 and 0.270 for such two cases (with back view selection vs always select back view), supporting our claims.

4: Although the paper proposes a selection module to filter inconsistent inputs, the performance is still bounded by the quality of novel view synthesis models.

We acknowledge that final performance is influenced by the novel view synthesis (NVS) model's quality, a characteristic common to two-stage pipelines. However, our selection module specifically mitigates this limitation. By filtering NVS outputs and selecting only the highest-quality views available, our approach makes better use of the NVS model's capabilities and reduces the negative impact of inconsistent views, achieving a tighter performance bound. Consequently, even with the same underlying NVS model, our method yields superior final 3D quality. Furthermore, NVS models themselves are expected to improve, potentially benefiting significantly from advancements in large-scale video generation models which learn implicit 3D consistency from vast video data.

Thank you for your thorough review and valuable feedback. We appreciate you acknowledging several strengths, including the paper's clear organization, the effectiveness of our view selection in improving input quality, the practicality of handling varying view numbers in reconstruction, and the potential community value of the proposed modules.We provide point-by-point responses below, addressing each weakness and question raised.

1: Exploring fundamental challenges or valuable insights would further enhance the paper's impact.

We concur with the reviewer's suggestion regarding the value of discussing fundamental challenges and insights. While our primary focus is mitigating suboptimal outputs from the first-stage in common two-stage 3D Gen models, we discussed insights and future directions (e.g., feed-forward 3D/4D generation, generative reconstruction) in the Appendix's section A.

2.1: Is 2000 labeled samples enough for accurate filtering?

Yes, we found 2,000 labeled samples to be sufficient for training an accurate filter. This sample size also aligns with practices in related work, such as the data curation pipeline in Instant-3D. Its effectiveness stems from leveraging powerful pre-trained image encoders (DINO). To validate accuracy and robustness, we tested on 100 held-out videos from our method and 100 from SV3D (out-of-distribution). The classifier achieved 93% accuracy on our videos and 90% on SV3D's. These high rates, especially on OOD data, show the 2,000 samples were sufficient and the filter is robust.

2.2: Can we trust manual labeling?

We conducted a rigorous labeling process, for manual labeling, the authors first carefully labeled 100 sample videos. These were then provided to two labelers, and each labeler was asked to label approximately 1,000 videos, resulting in a total of 2,000 labeled videos. The trustworthiness is corroborated by the strong empirical performance of the classifier trained on these labels (high accuracy detailed in 2.1). Furthermore, results in Table 4, Figures 5-6 show this filter significantly improves our generation pipeline's overall performance, indicating the manual labeling was reliable for its purpose.

2.3: Are there better, more systematic ways to evaluate consistency?

We conducted a sensitivity analysis that indicates that the final generation results are relatively robust to variations in this threshold within a reasonable range (50% to 70%). For instance, varying the threshold between 50% and 70% yielded comparable final generation quality, as measured by CLIP text similarity (ranging from 27.4 to 27.7) and Video CLIP text similarity (ranging from 25.3 to 25.7). We agree that more sophisticated methods like learned metrics or adaptive thresholds are promising future research directions for potentially more optimal filtering.

3: A simpler and more efficient training approach might be better—something that still delivers strong results but is easier for others to adopt and build on.

We agree the full pipeline (NeRF pre-training, 3D Gaussian training, imperfect view simulation training) is resource-intensive. However, efficiencies can facilitate adoption: Stage 1 (NeRF) can be bypassed by initializing Stage 2 (GS) directly from available pre-trained Instant-3D weights. Stage 3, which enhances robustness to imperfect inputs, is optional if the primary application is high-quality reconstruction from clean views. These optimizations reduce the core training requirement primarily to Stage 2, substantially lowering the resource barrier and complexity compared to training everything end-to-end from scratch.

For board adapatation, the FlexRM architecture is designed with a minimalist philosophy and can be easily reproduced based on Instant-3D, which can be easily adopted or being implemented by others. Nevertheless, we recognize that further reducing the computational demands of large-scale 3D generative models remains an active and important research challenge across the field.

4: Maybe trying other ways to handle noise, like adversarial training or more varied noise types.

We tested adding Gaussian noise (σ up to 0.05), salt-and-pepper noise (density up to 0.05), and combining Gaussian noise with our simulation. A comparison using 4-view reconstruction showed these other noise types degraded performance noticeably. In contrast, our proposed simulation pipeline using 3D Gaussians slightly enhanced performance, suggesting it is a more suitable approach in this context.