DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization

We thank the reviewer for the positive reviews. We provide our responses below.

Q1: Quantitative comparisons with DreamReward.

A1: Thanks for your suggestion. We conduct a quantitative comparison on GPTEval3D to evaluate human preference, including the correction of numbers. In specific. we report the number accuracy and CLIPScore. The results below show that DreamDPO demonstrates competitive performance in prompt alignment and significant improvement in number correction.

Table 2-1: Quantitative comparisons of DreamDPO and DreamReward on GPTEval3D. We calculate the CLIPScore for prompt alignment and number accuracy for number correction.

We attribute this to that DreamDPO does not rely on precise reward scores, allowing it to leverage various black-box AI models for scoring, which helps correct numbers and attributes and improves human preference alignment.

Q2: The potential limitations on 2D metrics.

A2: Yes. 2D image reward models are relatively mature, with abundant data and good reward performance. However, they have limitations. While they improve prompt alignment, such as correcting character attributes (e.g., changing "man" to "knight"), they struggle to address inherent Janus issues (e.g., characters with three legs). Our DreamDPO is flexible with both 2D and 3D rewards. Experiments in Figure 3-1 demonstrate that it effectively mitigates Janus issues and some potential artifacts, offering a more reliable evaluation for 3D consistency.

Q3: More exploration of LMM capabilities.

A3: Thanks. We have further analyzed the failure cases of DreamDPO with LMMs (see R2-Q2). Additionally, we demonstrate that increasing the number of comparison candidates can effectively mine positive samples, leading to improved performance with LMMs (see R2-Q3). Lastly, we show that DreamDPO works effectively across various LMMs (see R2-Q4).

Q4: Additional discussion with feed-forward methods.

A4: Thanks. Aligning feed-forward methods with human preferences is meaningful but remains underexplored, primarily due to the significant computational resources required. DreamDPO offers valuable insights for feed-forward methods. Specifically, by constructing candidates with varying noise and using ranking-guided optimization, DreamDPO provides a potential pathway for RL optimization of feed-forward methods. We will polish the paper with better contextualize within the broader landscape of 3D generation.

Q5: Ablation studies on the score gap threshold in the 3D setting.

A5: Thanks. Please kindly check R1-Q1.

Q6: More results with other SDS-based text-to-3D generation methods.

A6: Thanks. We evaluate the performance of DreamDPO with different backbones in Section 4.3. In specific, we evaluate the performance of DreamDPO with Stable Diffusion v2.1 (SD2.1). The results demonstrate that DreamDPO works effectively with SD2.1 and achieves competitive results compared to ProlificDreamer. While SD2.1 shows improvements over the baseline, MVDream outperforms due to its superior 3D consistency. Therefore, we adopt MVDream as the default backbone.

We thank the reviewer for the positive reviews. We provide our responses below.

Q1: The computational efficiency comparison to other baselines.

A1: Thank you for your question. We summarize the computational cost of our method compared with other text-to-3D generation baselines as follows:

Table 2-1: The analysis of the time consuming of text-to-3D generation methods.

While introducing additional computational overhead due to pairwise example construction, our method achieves the best results on the GPTEval3D Benchmark, demonstrating superior alignment with human preferences. Meanwhile, we can mreduce generation time by 25% (to ~1.5 hours) while maintaining performance (see R1-Q4).

Q2: How are $x_t^{win}$ and $x_t^{lose}$ determined?

A2: We utilize a reward model to compute the scores of pairwise examples. The sample with a higher score is regarded as the "win" example ($x_t^{win}$), while the sample with a lower score is regarded as the "lose" example ($x_t^{lose}$).

Q3: Can current text-to-3D generated items be used in real-world manufacturing?

A3: Yes. Text-to-3D generation is increasingly suitable for real-world manufacturing, particularly in customized product design. It allows engineers to quickly create 3D models from textual descriptions. While further refinement may be needed to meet precise industrial standards, the technology is proving valuable in automating design workflows and enabling mass customization.

We thank the reviewer for the positive reviews. We provide our responses below.

Q1: Dependence on External Models.

A1: DreamDPO is compatible with external models and rule-based reward metrics, such as image quality metrics (e.g., BRISQUE [1]) and 3D-consistency evaluation. By leveraging 3D geometry information (depth and camera transformation matrices), we could establish stereo correspondence between views to evaluate multi-view consistency. These approaches do not require additional reward models and are suitable for scenarios with limited computational resources. As shown in Figure 1-1, we include a case study demonstrating DreamDPO with BRISQUE, a no-reference image quality evaluator, to show its effectiveness.

Q2: How to mitigate the potential bias from reward model?

A2: Reward hacking is a known issue in RL-based optimization. One solution is to scale the training data and model size of the reward model. As shown in our experiments in Section 4.3, the reward model HPSv2, which has a stronger generalization ability compared to ImageReward, demonstrates superior generation performance in most cases. Moreover, DreamDPO supports an ensemble of reward models, combining their ranking to reduce reliance on a single model and mitigate unwanted biases (see Figure 1-2).

Q3: How does the LMM in STEP2 work, and where do its questions come from?

A3: We detail the LMM-based pairwise comparison in Section B.2. In STEP2, given pairwise examples, LMM conducts the comparison query sequentially. For each query, LMM performs visual question answering based on the provided image and query, and we extract the number of "yes" responses as the score. The questions for LMM can be customized by the user or generated from a template. For instance, LLM can automatically extract questions from a given prompt, such as generating "Is the leaf shouting?" from "A shouting leaf". Alternatively, users can define custom questions, such as "Does the elephant stay on the ground?" for "A dancing elephant".

Q4: Why is the generation time long, and can it be improved?

Table 1-1: Reducing generation time for DreamDPO while maintaining improved performance.

A4: The vanilla SDS takes around 1 hour for optimization. Our method takes approximately 2 hours due to the pairwise example construction. To speed up the process, we can adopt a simple yet effective strategy: performing SDS for the first 6000 iterations and then switching to DreamDPO. As shown in Figure 1-3, this approach reduces the generation time by 25% (to around 1.5 hours) while maintaining improved performance.

[1] Anish et al. Blind/referenceless image spatial quality evaluator. ASILOMAR 2011.

We thank the reviewer for the positive reviews. We provide our responses below.

Q1: Ablation studies on the score gap threshold in the 3D setting.

A1: We present the ablation study of the score gap threshold $\tau$ in the 3D setting (see Figure 2-1). The results indicate that a large $\tau$ (e.g., $\tau=1$) degrades DreamDPO to the SDS loss, resulting in an over-smoothing issue. Conversely, setting $\tau=0$ results in over-saturation, such as a purplish sunlight appearance in rocks. Therefore, we recommend using a small but non-zero $\tau$.

Q2: Show some failure cases to highlight DreamDPO’s limitations.

A2: While DreamDPO has shown improvements in aligning 3D generation with human preferences, there are still some failure cases. For example, number and attribute correction are crucial for prompt alignment, but DreamDPO sometimes fails to address these issues shown in Figure 2-2. We found that this limitation arises from the generative model's capacity. When the positive sample is hard to generate, DreamDPO struggles to construct effective positive-negative pairs, causing it to degrade into SDS.

Q3: Increasing the number of comparison candidates per iteration.

A3: Thank you for your valuable suggestion. We further extend DreamDPO to multi-sample comparisons. In specific, we expand STEP1 to multi-example construction, enabling the creation of more comparison candidates (e.g., 4 or 6). We then select the candidate with the highest score as the "win" example and the lowest score sample as the "lose" example. Experiments in Figure 2-3 (the first and second columns) demonstrate that this improvement is effective, as it better mines positive samples and enables DreamDPO to construct positive-negative pairs more effectively.

Q4: More evaluation on DreamDPO with large multi-modal models.

A4: Following your recommendation, we evaluated the robustness of DreamDPO with various LMMs. Specifically, we conducted experiments using three large multimodal models: Qwen2.5-VL-3B, Qwen2.5-VL-32B, and Qwen-VL-Plus. Notably, Qwen2.5-VL-3B has only 3B parameters, making it a relatively weak LMM. The results in Figure 2-3 show that when the LMM performance is relatively poor (e.g., Qwen2.5-VL-3B in third column), it fails to correct numbers effectively. In SDS-based optimization, number correction should perform early, which requires strong LMM capabilities. Therefore, high-performing LMMs can improve DreamDPO.