Hallo3D: Multi-Modal Hallucination Detection and Mitigation for Consistent 3D Content Generation

Thank you for your feedback. We appreciate your recognition of our method's innovation and effectiveness. Here are our responses to your comments.

W1. "Fig.2 seems to understate the performance of current baselines."

A1. We appreciate your attention to the details in Fig.2. Actually, this illustration is based on the Score Jacobian Chain[1] (a text-to-3D baseline method). The figure's primary purpose is to visually explain the Janus problem by highlighting an example where the multi-faces phenomenon is particularly pronounced. To better illustrate the Janus Problem, we included the expected result under normal conditions as achieved by our method, focusing on the problem's severity rather than comparing different methods. We understand the importance of precision and will consider adding a note in the figure caption to ensure this context is clearer in future versions.

W2. "Equation 6 in line 183 does not seem to be derivable from Equation 12 in the DDIM paper."

A2. Thank you for your thorough review and for pointing out this issue. After carefully revisiting our derivation, we confirmed that there is indeed a typo. We have corrected the derivation by setting $\sigma =0$ in Eq.12 of the DDIM paper. The correct form is as follows:

$$\hat{\mathbf{x}} _ {t-1} = \sqrt{\frac{\alpha _ {t-1}} {\alpha _ t}} \hat{\mathbf{x}} _ t + (\sqrt{1-\alpha _ {t-1}}-\sqrt{\frac{\alpha _ {t-1} (1-\alpha _ t)} {\alpha _ t}}) \tilde{\epsilon} _ {\phi} (\hat{\mathbf{x}} _ t, t, P^{+}, P^{-} _ E)$$

Your feedback has been instrumental in ensuring the precision of the paper. We have revised the formula accordingly in the updated version to enhance its accuracy.

W3. "Ablation study section lacks clarity. Rows 2-4 do not show a clear visual difference."

A3. To demonstrate the necessity of each module in our method, we conducted quantitative ablation experiments, including one that ablates both C and $P _ E ^ -$, i.e., "w/o C & $P _ E ^ -$" to validate the importance of $\mathcal{L} _ {\rm{CG}}$. The results in Q1 of the Global Review confirm that each module in Hallo3D enhances performance.

In Sec.4, we focus on how module A primarily affects color and texture, while module B and module C enhance cross-view consistency. Specifically:

• Row 2, w/o A: The lion is darker than Row1, and the deer shows a blurry halo and unreasonable color shift.
• Row 3, w/o B: The lion's head is derormed in the first and third columns, and the deer misses its head.
• Row 4, w/o C: The "second face" appears on the lion's back in the third column, and the right deer's image shows a clear Janus Problem, with multiple legs and a distorted body visible in the second and fourth columns.

Q1. "The authors use the “DDIM Inversion” to regenerate an image without hallucination. However, generating such an image with CFG without damaging the original structure is not easy."

A-Q1. That's an excellent question. We agree with your point that CFG-based techniques often struggle to maintain structure in regenerated images in image editing or reconstruction. However, in generative tasks, we use DDIM Inversion to regenerate the image as a guidance for constraining consistency, not as the final output. Our primary focus is on the final 3D assets, with the images obtained using rendering techniques that are independent of the diffusion model.

Specifically, generative tasks involve repeated optimization, where each iteration may cause some structural loss, but these intermediary images are not visible. Therefore, as long as the final result is consistent and high-quality, the loss of information is relatively less important. This contrasts with image editing or reconstruction, where only a single image is produced, making structural details critical.

Our experiments confirm this. As shown in Figures 5 and 6, even with structural information loss during the DDIM Inversion process, we achieved consistent results.

Q2. "It appears that you only feed a single image into the LMM. However, a less serious inconsistency can only be observed from certain view directions. How can the effectiveness of Hallo3D be proven?"

A-Q2. Yes, we only input one image at a time to reduce the inference cost of LMM. However, since the camera pose for each rendered image is random, multiple iterations ensure that all sides of the 3D object are addressed, effectively avoiding inconsistencies in specific views.

Our experiments also show that this method effectively addresses even minor multi-view inconsistencies. For instance, in the second image of the fifth row in Fig.5, the car's right rear wheel appears doubled, but our method corrected this. Similarly, in the second example in Fig.6, our method improves subtle Janus problems on the figure's body.

Q3. "The method appears to require frequent use of the LMM. Given that generating 3D requires 5000-10000 iterations, what‘s the time cost?"

A-Q3. We measured the time overhead for both 3DGS-based and NeRF-based baselines, finding minimal additional costs, as detailed in Q1 of the Global Review. While LMM inference can add time, our model uses single-stage dialogue and queries at intervals during later training to improve efficiency. Additionally, due to the Multi-View Appearance Alignment, our method processes four images at a time with batch=4, reducing the number of SDS iterations.

---

We hope that our response addresses your concerns sincerely. Looking forward to further communication with you!

[1] Wang H,et al. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023.

Thank you for your feedback. We appreciate your recognition of our method's innovation and effectiveness. Here are our responses to your comments.

W1. "In lines 148-150, the paper claims defining a new denoising strategy, but there's no further details. Lines 184-186 provide some description, but lack a detailed explanation of how to utilize appearance attention."

A1. Thank you for pointing this out. We’d like to clarify that the "new denoising strategy" mentioned in lines 148-150 refers to the introduced strategy in Sec.3.2.

In the denoising strategy $\tilde{\epsilon}_\phi(\cdot)$, $\rm{AAttn}(\cdot)$ in Eq.4 functions as cross-attention. The key and value are derived from the focal view, with each of the four views calculating a distinct query. This ensures that each view aligns its features with the focal view, achieving consistent appearance, and we will make it clearer in further versions.

W2. "The ablation study lacks quantitative analysis."

A2. We have added quantitative analysis in Q2 of the Global Review, and the quantitative results further verify the effectiveness of our method.

W3. "Lack of enough quantitative analysis showcasing performance improvements on image-to-3d frameworks, such as SSIM, PSNR, etc. The GSO dataset or Objaverse dataset can be used."

A3. Thank you for the suggestion. We followed the experimental setup in [1], randomly selecting 30 objects from both the GSO and Objaverse datasets, totaling 60 objects. To ensure variety, we replaced objects with overly simple structures. We rendered their frontal views at 256x256 resolution for input into our method. Performance was assessed using Chamfer Distance (CD) and Volume IoU (Vol. IoU) for geometric quality, and PSNR, SSIM, and LPIPS for visual quality. The results are shown in the table below.

The experimental results show that our method outperforms the baseline across all metrics, improving both geometry and textures. This further confirms the broad applicability of our approach, enhancing both text-to-3D and image-to-3D tasks.

W4. "The impact of methods on training time is not introduced."

A4. We measured the time overhead for both 3DGS-based and NeRF-based baselines, and the results show minimal additional time costs, as detailed in Q1 of the Global Review.

Q1. "What does consistency refer to in the paper? The SDS-based method is optimized to generate a complete 3D content, not like directly using 2D diffusion models to generate multi-views. Does inconsistency refer to Janus problem?"

A-Q1. Yes, inconsistency refers to the Janus problem. The term consistency refers to a 3D object maintaining its normal visual structure and appearance across multiple views.

Q2. "Why need an additional $\mathcal{L} _ {\rm{CG}}$ loss? Why not add negative prompts directly to the 2D diffusion models in the top left of Figure 3 for $\mathcal{L} _ {\rm{SDS}}$ loss?"

A-Q2. Thanks for your suggestion. Indeed, adding negative prompts directly to the 2D diffusion models is what we do in the w/o C configuration. We also introduced an ablation version, excluding the $P _ E ^ -$ output by LMM from the $\mathcal{L} _ {\rm{CG}}$ calculation, to isolate $P _ E ^ -$ impact. The outcomes from both ablation versions thoroughly evaluate the role of $\mathcal{L} _ {\rm{CG}}$, confirming its necessity.

Q3. "Is it necessary to first use the baseline methods to obtain a raw 3D content before cooperating with Multi-modal LLM for subsequent operations?"

A-Q3. Yes. We initially employ baseline methods to generate a raw 3D content. The method we proposed is designed to perform hallucination detection and mitigation on the results produced by the 3D baseline. Therefore, it's necessary to have 3D content prior to the detection and correction process.

---

We hope our response is helpful in addressing your concerns. We look forward to continuing our communication with you.

[1] Long X, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In CVPR, 2024.

Thank you for your feedback. We appreciate your recognition of our method's innovation and applicability. Here are our responses to your comments.

W1. "I doubt the reproduction of this work."

Thanks for pointing this out. We'll share our code upon publication to help with reproducing our work.

W1.1-1. "Is P_I always the same as the prompts in Figure 4?"

A1.1-1. We used different P_I than in Fig.4. The primary purpose of Fig. 4 is to use a case study to demonstrate how LMMs can infer structural consistency and respond in specific formats. To highlight this capability, we employed two dialogues. In practice, we used a single interaction to query the LMM to achieve faster runtime. Please refer to Q4.1 in the Global Review for the specific setting of P_I.

W1.1-2. "All rendered images are inputted to LMM?"

A1.1-2. For efficiency, we input only one of the four rendered images into the LMM, selecting it randomly to avoid over-intervening in a specific view.

W1.1-3. "Which LMM is used?"

A1.1-3. For LMM, we chose the locally deployed LLaVA, using the version llava-v1.6-34b.

W1.2-1. "How to balance loss_SDS and loss_CG? "

A1.2-1. Thanks for your pointing this issue. There's actually a typo in Eq.8. To balance L_SDS and L_CG at the same order of magnitude, we set w=0.1 for L_SDS. We'll correct this in the further version. Here's the correct formula:

$$\mathcal{L}(\theta) = \mathcal{L} _ {SDS} + w\mathcal{L} _ {CG}, \text{\quad if } D _ \psi(x, P_I) \text{ is not None}$$

And the other case is:

$$\mathcal{L}(\theta) = \mathcal{L} _ {SDS}, \text{\quad if } D _ \psi(x, P_I) \text{ is None}$$

W1.2-2. "Can you provide the loss curve of them?"

A1.2-2. Please refer to Q3 of the Global Review for the detail.

W1.2-3. "What is the proportion does L_CG work?"

A1.2-3. We provided quantitative ablation results in Q2 of the Global Review. The analysis indicates that adding L_CG increases the average CLIP-Score from 24.32 to 27.03, contributing significantly to the overall improvement. Additionally, the CLIP-Score curve in Fig.4 (PDF) further demonstrates the effectiveness of L_CG.

W1.2-4. "Why Dψ can produce 'None'?"

A1.2-4. When Dψ receives low-quality or poorly angled images, LMMs may fail to generate the expected negative prompt. The output then cannot be recognized by our regex and returns None. This process is visualized in Fig.5 in the PDF.

W1.3. "About general negative prompt in w/o B and w/o C."

A1.3. w/o B means that P_E^- is not involved in L_CG, while w/o C indicates that P_E^- is involved in L_SDS during the ablation of L_CG. Our method acts as a universal enhancement for 3D generation, considering the common use of general negative prompts in baseline methods[1]. The general negative prompt used in w/o B can be found in Q4.2 of the Global Review.

W2.1. "About the training time."

A2.1. We detailed the training time in Q1 of the Global Review. LMM does introduce additional training time, but we believe this is acceptable given the significant improvement in performance.

W2.2-1. "About the ablation."

A2.2-1. More ablation study and analysis are detailed in Q2 of the Global Review.

W2.2-2. "Replacing P_E^− with a general negative prompt?"

A2.2-2. As discussed in A1.3, we adopt a generative negative prompt (which is used by baseline methods) in all the ablation study. Specifically, w/o B removes P_E^− and kept general negative prompt.

W2.2-3. "Introduce L_CG all the time?"

A2.2-3. L_CG is designed to constrain the view consistency, but in the early stage, when image quality is too low and lacks sufficient structural information, constraining with L_CG is less meaningful. Additionally, applying L_CG only in the later stages can also reduce training overhead.

W2.2-4. "What is the impact of P_I for Hallucination Detection?"

A2.2-4. The two dialogues in Fig.4 correspond to the two roles of P_I:

1. Activating the LMM's ability to infer 3D view consistency. By including query necessary information about inconsistencies in P_I, the LMM is prompted to identify specific issues in the image.
2. Standardizing the LMM's output format. By providing a one-shot example, the LMM's output is guided to match the negative prompt format, making it easier for our designed regular expression to recognize.

W2.3. "Replace Magic3D with ProlificDreamer."

A2.3. Thanks for suggestion. We added experiments on ProlificDreamer. The results in Fig.2 (PDF) show our method can improve its consistency. We will include more detailed discussion about ProlificDreamer in the final version.

W2.4. "Comparison with 'perp-neg' and 'debiasing scores and prompts'."

A2.4. Thanks for the suggestion. We added experiments on Perp-Neg and Debiasing Scores and Prompts. Results in Fig. 3 (PDF) show that our method improves consistency better than both approaches, as reflected in the quantitative and qualitative outcomes.

Q1. "How to select a focal view?"

We use Fovy, the camera's vertical view, as our selection standard. The first view with Fovy exceeding 120% of each baseline's default becomes our focal view, enabling a broader shot capturing more object details.

Q2. "Why the shown examples are low quality?"

A-Q2. Our method aims to boost view consistency in 3D generation, with quality hinging on the baseline models. We've tested this on high-quality models like GaussianDreamer, DreamGaussian, and ProlificDreamer. The results confirm our method's versatility, elevating both lower and higher-quality 3D generations.

Q3. "Provide 360-degree example?"

A-Q3. Thanks for suggestion. We have displayed the 360-degree view of the results in Fig.1 (PDF).

---

We hope our response resolves your concerns. Due to character limitations, we've provided a concise answer to your questions. We look forward to further communication with you!

[1] Yuan-Chen Guo, et al. Threestudio. https://github.com/threestudio-project/threestudio, 2023.