3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation

Comments on the questions:

Could the authors elaborate on how 3D-Adapter's 3D feedback augmentation differs fundamentally from existing I/O sync techniques?

As stated in the paper,

We broadly define I/O sync as inserting a 3D representation and a render-ing/projecting operation at the input or output end of the denoising network to synchronize multiple views.

Typical output synchronization models use rendered views as outputs for the current denoising step, fed into the diffusion solver. These models often face error accumulation (unless reconstruction and rendering are perfect) or mode collapse (discussed in Appendix A.1). For example, DMV3D is a native I/O sync model, it does not suffer from mode collapse because it is trained to predict $\hat{\mathbf{x}}_t$ as the mean of the distribution $p(\mathbf{x}_0|\mathbf{x}_t)$, where $p(\mathbf{x}_0|\mathbf{x}_t)$ is a joint probability of multiple views. However, it still suffers from error accumulation in late denoising stages (confirmed with the DMV3D authors). On the other hand, SyncMVD, an adapted I/O sync model using Stable Diffusion for texture generation, lacks domain-specific training and suffers from mode collapse due to score averaging.

What makes 3D-Adapter unique is its ability to introduce 3D-awareness while keeping the base model's topology intact. By leveraging ControlNet’s zero-initialized weights, 3D-Adapter integrates 3D priors with minimal impact on the base model, achieving enhanced performance without quality degradation.

Additionally, it is recommended that the authors consider showing intermediate results during the denoising process, such as outputs from the 3D reconstruction model at various stages. This could help readers better understand the contributions of the proposed method.

Thank you for the suggestion. We have added visualizations of the intermediate results in the revised PDF (Fig. 8 in the Appendix).

Thank you for the constructive comments. We have uploaded a revised PDF and the following are comments in response to the weaknesses.

Claims About Existing Methods: The assertion (L046-L053) that 1/0 sync methods degrade residual connections and cause texture quality issues is confusing and not adequately supported. More specific explanations with these methods are necessary.

(Edited) We acknowledge that our use of “residual connections” may be confusing, so we have revised the text for better clarity:

Diffusion model sampling is sensitive to error accumulations (Li & van der Schaar, 2024). I/O sync methods insert 3D reconstruction and rendering operations into the denoiser in a way that disrupts the original model topology and introduces errors during each denoising step (unless reconstruction and rendering are perfect).

Inadequate Definitions: The paper lacks clarity on the a) components of the 3D-Adapter (I understand that it includes VAE Decoder, ControlNet, and 3D Reconstruction Model) and b) the "I/O sync" baselines compared in Fig.1. Explicitly listing the modules included and the compared methods or baseline settings would aid understanding.

Edit: Your understanding is correct. In the revised PDF, we updated Figure 2 to include VAE encoders and decoders.

Comparison Gaps: The image-to-3D generation experiments do not include comparisons with CRM[1], SV3D[2], InstantMesh[3].

Thank you for the suggestions. We have added CRM and InstantMesh to the comparisons. Notably, the strongest methods (3D-Adapter, GRM, InstantMesh, One2345++) all use Zero123++ as their base model. To the best of our knowledge, SV3D is a comparatively weaker base model.

The text-to-texture experiments do not include metrics like FID, KID

Computing FID and KID metrics requires a reference dataset, but none of the text-to-texture models we evaluated are trained on domain-specific data, as they adapt single-view image diffusion models for texture generation. Therefore, a standardized reference dataset does not exist. Using Objaverse for this purpose is unsuitable because (a) most Objaverse objects have poor textures, and (b) the evaluated methods do not disentangle texture from lighting, requiring non-standardized lighting choices for reference data creation.

Comparisons with some established methods such as Paint3D[4] and FlashTex[5].

Edited: Paint3D and FlashTex aim to disentangle lighting from texture, which is beyond the scope of this work (we acknowledge this as a limitation in the revised paper).

More discussions about training efficiency would strengthen the paper, because it requires fine-tuning reconstruction model while keeping the multiview diffusion model in memory.

Thank you for the suggestions. In the revised PDF (Section 4.1 Training details), we have added the training time and VRAM requirements:

The adapter is trained with a total batch size of 16 objects on 4 A6000 GPUs (VRAM usage peaks at 39GB). In phase 1, GRM is finetuned with a small learning rate of $5\times10^{-6}$ for 2k iterations (for Instant3D, taking 3 hours) or 4k iterations (for Zero123++, taking 9 hours). In phase 2, ControlNet is finetuned with a learning rate of $1\times10^{-5}$ for 5k iterations (taking 8 hours for Instant3D and 5 hours for Zero123++).

We think our training cost isn’t particularly heavy, when compared to training the base diffusion model such as Instant3D and Zero123++.

Insufficient References and Discussion of Prior Work.

Edited: Thank you for providing the suggested references. All the references have been added.

Remarks on some of the related work:

• Ouroboros3D is a concurrent work and the high-level design is very close to our 3D-Adapter. The key difference is that Ouroboros3D feeds the rendering to the next denoising timestep, while ours operates within the current timestep. In fact, we have tested the other design as an early iteration of 3D-Adapter, and the results were slightly weaker on the text-to-texture benchmark (CLIP=26.12, Aesthetic=4.83, vs our CLIP=26.40, Aesthetic=4.85).
• Free3D is a novel view model, its role is the same as Instant3D or Zero123++. EpiDiff, SPAD also focus on improving novel view models through epipolar attention. These work did not evaluate the rendered views from 3D representations. In contrast, our work focuses on imposing explicit 3D reconstruction and rendering. In response to Reviwer dfQt, we have tested the official code of EpiDiff, and the generated views exhibit poor consistency.
• Cycle3D is an I/O sync method for image-to-3D. It also produces slightly blurry appearances on the backside of objects.
• IM3D is two-stage generation but with repeated SDEdit-like refinements to the rendered views, which is an orthogonal contribution to our 3D-Adapter.
• Paint3D and FlashTex focuses on PBR texture generation, which is beyond the scope of this work (we acknowledge this as a limitation in the revised paper).

Thank you for the constructive comments! We have uploaded a revised PDF, which hopefully addresses some of the clarity issues. The following are comments in response to the weaknesses and questions.

1. Clarity of exposition: A more comprehensive explanation of the relationship between GRM and models such as Instant3D and Zero123++ would greatly enhance the reader's understanding. It is particularly important to elucidate whether GRM builds 3D representations from the views generated by Instant3D and Zero123++ and whether fine-tuning of GRM is involved in this process.

To clarify, Instant3D or Zero123++ serves as the base model, while GRM acts as the 3D reconstructor. This relationship is outlined in Section 4 and illustrated in Figure 2(c). GRM constructs 3D representations from the base model's intermediate denoised views during both fine-tuning and inference.

According to Equation 2, it appears that the 3D-Adapter could potentially be applied to other I/O sync-based pretrained models. However, it is necessary to discuss whether incorporating Control Net into I/O sync pretrained models beyond GRM would yield similar results.

In Table 5, we tested 3D-Adapter combined with I/O sync for texture generation (which is not based on GRM but texture backprojection). As expected, this combination underperforms compared to 3D-Adapter alone due to error accumulation and blur introduced by I/O sync (see Figure 5 and the drop in Aesthetic score).

… and why Training Phase 1 is essential. If Training Phase 1 is a crucial step for optimizing the performance of 3D-Adapter, including GRM, it should be examined whether this phase is equally necessary when applied to other models.

Edited: Training Phase 1 involves GRM finetuning. As shown in the I/O sync experiments (A2 vs A1), only the finetuned GRM improves geometry quality (indicated by the MDD metric and visualized in Fig. 8). Additionally, C0 vs A0 also reveals the contribution of the finetuned GRM in the 2-stage setting (C0 removes the feedback and effectively becomes 2-stage). These evidences reveal that phase 1 is also beneficial to other models. Given these evidences, it is undoubted that 3D-Adapter should also use the finetuned GRM, and testing 3D-Adapter without GRM finetuning is unnecessary. However, if you're interested, we can test this setting for the rebuttal.

Despite utilizing various 3D representation techniques and loss functions, such as NeRF and DMTet, ensuring global semantic consistency remains challenging.

This is very true and we have discussed it in the 4-th paragraph of Section 4.2. We stated in the introduction that 3D-Adapter is limited to fixing local geometry inconsistencies, and the global semantic consistency is beyond the scope of this work.

The role of augmentation guidance appears to be significant. Could you provide visual evaluations in addition to Table 1 to better illustrate this effect?

Edited: Thank you for pointing this out. We have added visual comparisons in the revised Appendix (Fig. 10).

Please include a dedicated pipeline figure specifically for the process described in Section 4.2 (line 289) to enhance clarity.

Thank you for the suggestion. We will add it in our next revision.

What are the results of using the optimization-based 3D-Adapter without additional conditioning.

Thank you for the good question. This will lead to the Janus multi-face problem, since 3D-Adapter alone does not guarantee high-level global semantic consistency.

Thank you for the timely response!

Regarding the clarity issues, we wil continue to revise the paper based on the comments from other reviewers. We would appreciate it if you could provide more detailed suggestions on writing improvement.

The claims about the I/O sync methods are analyzed in depth in the Appendix (also revised and updated). Please let us know if you find any part of it unclear.

Edit: We further revised the text to clarify the error accumulation issue, and added a reference to the paper On Error Propagation of Diffusion Models:

Diffusion model sampling is sensitive to error accumulations (Li & van der Schaar, 2024). I/O sync methods insert 3D reconstruction and rendering operations into the denoiser in a way that disrupts the original model topology and introduces errors during each denoising step (unless reconstruction and rendering are perfect).

The ControlNet's effectiveness can be directly reflected by comparing B0 and C0 in table 1 (both use the same finetuned GRM). For texture and avatar generation, disabling ControlNet feedback is equivalent to the two-stage baseline and we have shown the differences. Reviewer kMh5 has asked about the comparison between ControlNet and latent fusion, and we have clarified that latent fusion is equivalent to our newly added dynamic I/O sync baseline. Please let us know if you have questions about these validations.

Some additional comments on the I/O sync baseline:

I/O sync itself is not typically the first choice for 3D generation tasks. For example, in text-to-3D and image-to-3D tasks, most state-of-the-art methods (e.g., GRM, CRM, InstantMesh) employ two-stage approaches. I/O sync is more commonly used in texture generation tasks (e.g., SyncMVD, TexPainter), where our own I/O sync implementation outperforms others. Furthermore, its drawbacks are thoroughly analyzed both theoretically in Appendix A.1 (where the linearity assumption holds due to linear texture blending) and empirically, as demonstrated in the experiments in Table 5.

If you find our evidence insufficient, please be specific on the aspects that need improvement so we can better address them.

Thank you for the comments. We have uploaded a revised PDF and the following are comments on the weaknesses.

The quality improvement in the main text-to-3D task is 1) marginal and 2) visuials are not convincing enough to justify complicating the training pipeline. For instance, in Fig. 1, image-to-3D visuals are comparable b/w 3D-Adapter and "two-stage pipeline".

We revised Fig. 3 to highlight the differences between 3D-Adapter and 2-stage results. Across all 3D generation comparisons, 2-stage consistently shows more floaters and fuzzy geometries, which are fixed by 3D-Adapter. This is also reflected in the MDD metric in Table 1. In Fig. 8, we also show the differences of intermediate results.

The goal of 3D-Adapter is to improve geometry consistency without sacrificing visual quality, effectively removing floaters and texture seams. Ideally, 3D-Adapter should reproduce the base model's appearance unless inconsistencies arise. We do not expect 3D-Adapter to produce completely new results, which should be the job of the base diffusion model and the reconstruction method.

The paper lacks significant novelty, as the use of 3D representations for synchronization has been explored before (e.g. NerfDiff). The main contribution appears to be the placement of the adapter in the parallel branch rather than at the input/output stage of the diffusion UN et. However, this seems more like a technical choice, as is the decision to train Control Net on intermediate outputs.

Edited: We have clarified the differences between previous synchronization methods (I/O sync) and our 3D-Adapter. Our approach places 3D reconstruction in a parallel branch, preserving the base model’s topology. In the revised introduction, we clarified the error accumulation problem of I/O sync:

Diffusion model sampling is sensitive to error accumulations (Li & van der Schaar, 2024). I/O sync methods insert 3D reconstruction and rendering operations into the denoiser in a way that disrupts the original model topology and introduces errors during each denoising step (unless reconstruction and rendering are perfect).

Our approach, while simple, proves effective, as shown in our experiments. I/O sync often produces blurry results, whereas 3D-Adapter consistently demonstrates better visual quality.

ControlNet is an intuitive choice for our method because its zero initialization is designed to minimize the impact to the base model, and the base model’s topology remains intact. Additionally, it inherits the base model’s knowledge, requires minimal fine-tuning. Other technical choices are of course possible (e.g., we have tested T2I Adapter, which performed worse than ControlNet), especially considering the other base models such as DiTs. Nevertheless, we believe our core contribution—the parallel 3D branch—is a critical innovation.

Some visuals (e.g. Fig. 3, column 5) raise concerns about the correctness of the implementation.

We have double checked our implementation of I/O sync and found no problems. In fact, our I/O sync baseline is exceptionally strong on the text-to-texture benchmark; for text-to-3D, its MDD metric is also the best overall. Regarding the bad visual appearance, we added the explanation in the revised Appendix A.2 (edited):

While I/O sync works reasonably on our texture generation benchmark, our text-to-3D model using I/O sync (A2 in Table 1 and Fig. 3) exhibits significant quality degradation due to mode collapse. We believe the main reasons are twofold. First, the base model Instant3D generates a very sparse set of only four views, which are hard to synchronize. Second, our fine-tuned GRM reconstructor is trained using the depth loss to suppress surface fuzziness, which has a negative impact when its sharp renderings $\tilde{\mathbf{x}}_t$ are used as diffusion output. This is because a well-trained diffusion model should actually predict blurry outputs $\hat{\mathbf{x}}_t$ in the early denoising stage as the mean of the distribution $p(\mathbf{x}_0|\mathbf{x}_t)$. Only in the late stage should $\hat{\mathbf{x}}_t$ be sharp and crisp, as shown in Fig. 8.

In light of this, in the revised PDF, we added a stronger I/O sync baseline (A3) with the dynamic blending technique:

As shown in Table 1 and Fig. 3, dynamic I/O sync demonstrates significant improvements in visual quality over vanilla I/O sync.

Edit: Please also note that I/O sync itself is not typically the first choice for 3D generation tasks. For example, in text-to-3D and image-to-3D tasks, most state-of-the-art methods (e.g., GRM, CRM, InstantMesh) employ two-stage approaches. I/O sync is more commonly used in texture generation tasks (e.g., SyncMVD, TexPainter), where our own I/O sync implementation outperforms others. Here, its drawbacks are thoroughly analyzed both theoretically in Appendix A.1 (where the linearity assumption holds due to linear texture blending) and empirically, as demonstrated in the experiments in Table 5.

Comments on the questions (part 1):

Since residual connections shouldn't be disrupted if 3D reconstruction is applied before or after the UN et denoising stage, I'm unclear why this is described as a problem. Could you explain?

(Edited) We acknowledge that our use of “residual connections” may be confusing, so we have revised the text for better clarity:

Diffusion model sampling is sensitive to error accumulations (Li & van der Schaar, 2024). I/O sync methods insert 3D reconstruction and rendering operations into the denoiser in a way that disrupts the original model topology and introduces errors during each denoising step (unless reconstruction and rendering are perfect).

Please provide evidence that two-stage approaches specifically suffer from blurry textures.

Thank you for pointing this out. This is our writing issue. It should be "texture seams" rather than "blurry textures". We have revised the text:

A common issue with two-stage approaches is that existing reconstruction methods, often designed for or trained under conditions of perfect consistency, lack robustness to local geometric inconsistencies. This may result in floaters and texture seams.

Texture seams are very noticeable in the text-to-texture and text-to-avatar results.

Could you clarify what you mean by "linearity" in this context?

We have added a brief explanation in our revised Appendix:

When performing diffusion ODE sampling using the common Euler solver, a linear input sync operation (e.g., linear blending or optimizing using the L2 loss) is equivalent to...

To clarify, this means that the synchronized output is a linear combination of pre-synchronization views. This assumption made to simplify theoretical analysis. While some synchronization methods (e.g., our GRM-based I/O sync) are not strictly linear, the mode collapse issue from score averaging still persists.

Regarding concerns over some visuals: Could you provide a detailed explanation of the I/O-sync implementation in Tables 1 and 5? Table 5 shows a slight improvement when using I/O -sync in the "baseline" model, but in Table 1, there's a significant drop from AO to A1.

Table 5 focuses on texture generation. I/O sync is particularly effective for this task, since the depth ControlNet already provides strong conditioning, making view synchronization relatively easy. Notably, TexPainter and SyncMVD (shown in Table 6) are typical I/O sync methods for texture generation, though implemented differently from ours (still, our own implementation achieves better quality and efficiency).

In Table 1, A1 represents the I/O sync baseline using the original GRM without finetuning. The unfinetuned GRM is incompatible with coarse intermediate outputs, and will lead to severe error accumulation and poor geometry with floaters (reflected in the high MDD metric). After finetuning the GRM (A2), the MDD metric improves substantially, even outperforming the 3D-Adapter in geometry. The drop in appearance metrics following finetuning is explained in the additional analysis provided in Appendix A2 (edited):

While I/O sync works reasonably on our texture generation benchmark, our text-to-3D model using I/O sync (A2 in Table 1 and Fig. 3) exhibits significant quality degradation due to mode collapse. We believe the main reasons are twofold. First, the base model Instant3D generates a very sparse set of only four views, which are hard to synchronize. Second, our fine-tuned GRM reconstructor is trained using the depth loss to suppress surface fuzziness, which has a negative impact when its sharp renderings $\tilde{\mathbf{x}}_t$ are used as diffusion output. This is because a well-trained diffusion model should actually predict blurry outputs $\hat{\mathbf{x}}_t$ in the early denoising stage as the mean of the distribution $p(\mathbf{x}_0|\mathbf{x}_t)$. Only in the late stage should $\hat{\mathbf{x}}_t$ be sharp and crisp, as shown in Fig. 8.

In light of this, in the revised PDF, we added a stronger I/O sync baseline (A3) with the dynamic blending technique.

Edit: Please also note that I/O sync itself is not typically the first choice for 3D generation tasks. For example, in text-to-3D and image-to-3D tasks, most state-of-the-art methods (e.g., GRM, CRM, InstantMesh) employ two-stage approaches. I/O sync is more commonly used in texture generation tasks (e.g., SyncMVD, TexPainter), where our own I/O sync implementation outperforms others. Here, its drawbacks are thoroughly analyzed both theoretically in Appendix A.1 (where the linearity assumption holds due to linear texture blending) and empirically, as demonstrated in the experiments in Table 5.

Comments on the questions (part 2):

The visuals in column 5 appear broken and clearly worse than those in, for example, LGM, yet the CLIP score is significantly higher. Please double-check these evaluations.

Column 5 shows the two-stage results from the original GRM. While some results may appear broken, more outputs are included in the appendix for reference. We observed that the CLIP score is not sensitive to broken geometry, which is why we rely on the MDD score for evaluating geometry quality. Our evaluation results in Table 2 are valid and align well with the original GRM paper.

Could you justify the use of ControlNet? How does it compare to using pure latent fusion?

Thank you for the question. As noted above, we use ControlNet to preserve the original model topology. Since ControlNet uses zero-initialized weights, its impact to the base model is minimal, allowing 3D-awareness to be introduced without degrading quality.

Latent fusion has been tested in our added dynamic I/O sync baseline, which performs dynamic fusion between the UNet outputs and the rendered outputs in the latent space. While this improves upon vanilla I/O sync, the quality remains below the 2-stage baseline and 3D-Adapter. Details are provided in the revised Appendix A.2.

It's quite complex and specific training process, it would be great to hear about the reasoning/ablations behind these numbers.

The reconstruction method is not part of our main contributions, and we do not claim it to be optimal, though we tuned it extensively. More details are in the Appendix.

The purpose of this section is to provide a consistent platform for testing the optimization-based 3D-Adapter and comparing it to 2-stage and I/O sync baselines. The text-to-avatar task effectively demonstrates 3D-Adapter's advantages over these baselines when using the same reconstructor.

Thank you for reconsidering your assessment! While we acknowledge the uncertainty around the adoption of new methodologies, we believe our work provides a strong foundation for further exploration:

• Versatility: Our experiments span text-to-3D, image-to-3D, text-to-texture, and text-to-avatar setups, showcasing broad applicability. Other applications, such as panorama generation and 4D generation, could also benefit from improved synchronization methods like 3D-Adapter.
• Ease of Integration: 3D-Adapter maintains the base model's topology, making it plug-and-play if existing ControlNets and reconstructors are available. For example, we demonstrated its use with pretrained ControlNet and customized Stable Diffusion for SOTA texture generation—no training required, ensuring accessibility for the community.
• Significance of Results: Our experiments clearly demonstrate improvements across tasks, both visually and quantitatively, reinforcing the practical impact of our approach. Please note that 3D generation is a very competitive field, and our improvements in metrics are not trivial.

Thank you for the comments. We have uploaded a revised PDF and the following are comments in response to the weaknesses.

Lack of information on computation and memory overhead for competing methods

Our evaluation focuses on comparing 3D-Adapter with 2-stage and I/O sync methods. Detailed runtime analysis is provided in Table 7 (Appendix) and briefly mentioned in Section 5.2. Table 7 shows that 3D-Adapter's per-step time is 0.707 sec, with 0.531 sec spent on added modules. The remaining 0.176 sec is the time for 2-stage method (excluding the second reconstruction stage). Results for the Zero123++ model are similar.

Edit: In the revised PDF, we added inference time comparisons with SOTAs. Still, we need to point out that these system-level runtime comparisons are less significant due to variations in model choices and implementations.

Missing related work

Edit: Thank you for the suggestions. We have added these papers into related work.

Lack of comparison with recent methods in Sec. 5.4 "Text-to-Texture Generation"

DreamMat, Paint-it, Paint3D focuses on PBR texture and lighting disentanglement, which is on a different track to our 3D-Adapter. TexPainter is another I/O sync method. We have added results of TexPainter in the revised PDF.

In addition, for the Text-to-3D generation, [C1] released their code, but the authors did not compare with this closely related work.

Edit: EpiDiff is an image-to-3D model. We have tried the official code, but our 48GB A6000 GPU failed to run the inference, as the author suggested using an 80GB GPU. The core contribution of EpiDiff is improving the network architecture using epipolar attention, while our work focus on leveraging explicit 3D reconstruction and rendering, which is on a different track to EpiDiff.

Confusing notation of the proposed method in the tables

Many other methods are also combinations of existing multi-view diffusion models and reconstruction methods without explicitly naming them. Detailing all such methods equally would be too verbose given the space constraints.

Comments on the questions:

Have any similar results been discussed previously in different domains? It would be convincing to add references to discuss consistency across different works in different applications.

To our knowledge, few application-focused papers explore the theory behind linear combinations of score functions. We have added a citation to a concurrent work (Bradley & Nakkiran, Classifier-Free Guidance is a Predictor-Corrector), which examines similar phenomena in a different context.

How much training data should be used to train GRM? As stated in the training details of Sec. 4.1, the authors fine-tuned 2000 iterations with 16 objects in a single batch, i.e., 32000 objects. Considering that a single object yields many views, it seems to be a lot for fine-tuning. What is the minimal training data to make the proposed method work? This question can also be rephrased as why the authors picked 2000 and 4000 iterations for the Instant3D and Zero123++ cases, respectively.

The iteration numbers are set to ensure GRM performance saturates under the given schedules. GRM for Zero123++ requires more iterations because the released GRM only supports 4 views with a white background, while Zero123++ generates 6 views with a gray background, creating a larger domain gap. Using more iterations helps address this gap. Dataset sizes are detailed in Section 4.1: 47k for Instant3D and 80k for Zero123++. While we have not ablated dataset size, these numbers should have exceeded the minimal requirements.

Extending analysis: The authors showed why I/O sync may be bad, which is interesting. Then, another question naturally arises: why ControlNet-like feature addition (feedback augmentation) used in this work is effective? It is more interesting because the proposed method also does not guarantee a reduction of the gap in Eq. (7).

Thank you for the question. For 3D-Adapters with finetuned ControlNets, the diffusion loss forces the output $\hat{\mathbf{x}}_t$ to match the mean of the distribution $p(\mathbf{x}_0|\mathbf{x}_t)$, where $p(\mathbf{x}_0|\mathbf{x}_t)$ is a joint probability of multiple views. Without 3D feedback augmentation and ControlNet, it is more difficult for the model to learn the multi-view correlations in $p(\mathbf{x}_0|\mathbf{x}_t)$, while a 3D-aware branch makes it easier to do so. For 3D-Adapter with off-the-shelf tile and depth ControlNets, we empirically observe good results (since the tile ControlNet is robust to blurry conditions), although the accuracy has no perfect guarantee.

Lines 286-288: The authors mentioned two ControlNets for superresolution and depth. Does it mean the authors use both ControlNet or is the depth ControlNet replaced with the super-resolution ControlNet? Then, how can we feed depth rendering to the superresolution ControlNet encoder without training?

We use both ControlNets. The tile superresolution ControlNet takes the rendered RGB as input, while the depth ControlNet takes the rendered depth as input.

Line 287: Please provide the URL for the superresolution Control Net as a footnote to give proper credit and to enhance the reproducibility.

The tile ControlNet is a part of the official ControlNet repository from the ControlNet authors.

Line 485 - "texture field optimization": What is texture field optimization referred to? Is the texture field optimization applicable to the other competing methods, TEXTure, Text2Tex, and SyncMVD, in Table 6? Was it applied to them already?

After the final denoising step, occlusion may leave some object surfaces untextured when using texture backprojection. To address this, we use InstantNGP as a volumetric RGB representation, enabling smooth interpolation for unseen regions by optimizing it to match the final denoised views. TexPainter already employs this approach and SyncMVD could potentially adopt it. We don’t think this isthe primary factor affecting quality (TexPainter is clearly worse than SyncMVD and our I/O sync baseline due to other implementation reasons).

Thank you for the reply. The comments you read are outdated and have been updated. Please check the latest version on openreview.

Regarding Epidiff:

EpiDiff is an image-to-3D model. We have tried the official code, but our 48GB A6000 GPU failed to run the inference, as the author suggested using an 80GB GPU. The core contribution of EpiDiff is improving the network architecture using epipolar attention, while our work focus on leveraging explicit 3D reconstruction and rendering, which is on a different track to EpiDiff.

We have added CRM and InstantMesh as suggested by Reviewer TS5P. Note that the best methods we have compared with (One2345++, InstantMesh, GRM) all use Zero123++ as the multi-view model, stressing the importance of the base model. EpiDiff adopts a entirely new base model, which introduces lots of uncertainties. For text-to-3D, we will further add 3DTopia for comparison.

PBR texture generation is a non-trivial task. While it is entirely feasible to use pretrained diffusion models to generate PBR materials, this requires careful handling of PBR differentiable renderers and lighting. Therefore, adapting our own method to generate PBR texture is beyond the scope of our work (which already has a very large scope). Comparing our methods with PBR texture methods isn't really meaningful because our method (and the existing baselines we have compared) renders the lightingless albedo while the PBR methods need to render their results under specific lightings, so the condition isn't the same.

Thus, mentioning that it is not the authors' priority is irresponsible.

We are being very responsible not to mislead readers with unfair comparisons. Even if we finally get the comparison done, the result won't indicate which one of EpiDiff and 3D-Adapter is superior, because the base model we use are very different.

For the EpiDiff, well.. this reviewer strongly disagrees about while our work focus on leveraging explicit 3D reconstruction and rendering, which is on a **different track** to EpiDiff. The level and way in which multi-view consistency is introduced are comparable, where both methods propose modules conditioning multi-view consistency. Thus, closely related work.

For the PBR texture, I know the limitations of the proposed method the authors just mentioned (the proposed method cannot deal with view-dependent photometric characteristics due to the requirement of 3D reconstruction from multi-view images), and that's the point I pointed out in the initial review. That is something that needs to be discussed as a limitation, not something that can be rebutted.

Considering missing important references related to multi-view consistency, texture generation applications, and theory, the way the reviewer approaches could appear to be faithless. (While I didn't mention it intentionally, the theory of Appendix A is straightly deducible from the reference [Bradley & Nakkiran]. In this case, the authors should have cited that concurrent work at the initial submission.)

Update: Epidiff also does not provide a full 3D generation solution in their official codebase (https://github.com/huanngzh/EpiDiff). Only the multiview diffusion model is provided, which is insufficient to reproduce the 3D generation results. All the methods we have compared with are complete 3D generation solutions (with NeRF, GS, or Mesh representations), and the final rendered images are evaluated. Again, this shows that the focus of EpiDiff is to provide a better multi-view model, instead of building a complete 3D generation pipeline.

We understand your concerns about the PBR texture comparison. We will address this as a limitation in Section 5.4 in our next update.

Regarding [Bradley & Nakkiran], first of all, it is an concurrent work. Secondly, our appendix was written in April 2024, way before the release of [Bradley & Nakkiran] (Aug 2024). We have already added that reference in the revised version, but we do not understand why it is necessary to cite a concurrent work which focuses on CFG instead of 3D generation.

We acknowledge that the initial submission has a lot of missing references, which have been added in the revised version.

Thank you for the comments! We have uploaded a revised PDF, which hopefully addresses some of the clarity issues. The following are comments in response to the weaknesses.

By integrating depth conditioning via ControlNet...

Correction: 3D-Adapter employs RGB+depth ControlNets. In our texture generation benchmark, all baselines already use depth ControlNet; the improvement in quality comes from the additional RGB feedback provided by the 3D-Adapter.

To clarify, our concept of "geometry consistency" goes beyond object surface geometry (depth-related). It involves "ensuring precise 2D–3D alignment of local features and maintaining geometric plausibility." The RGB ControlNet provides essential feedback to ensure that the texture and silhouette of the object are aligned across different views.

The paper's scope is broad, which obscures the clarity of its main contributions and makes it challenging to pinpoint specific innovations with reusable community value.

We have revised the manuscript, which hopefully better clarifies the scope and contributions.

• We propose 3D-Adapter, which enables high-quality 3D generation with enhanced multi-view geometry consistency by integrating a 3D feedback module into a base image diffusion model.
• We demonstrate that 3D-Adapter is compatible with various base models and reconstruction methods, making it highly adaptable to a range of tasks.
• We conduct extensive experiments to show that 3D-Adapter improves geometry consistency while preserving visual quality, outperforming previous methods on text-to-3D, image-to-3D, and text-to-texture tasks.

Again, the main contribution of this paper is the high-level design of 3D-Adapter. This is not just one specific model, and we need to demonstrate its compatibility with multiple base models and reconstruction methods to show its generalizability.

Some comparisons and insights appear irrelevant, e.g. the text-to-avatar section, as the paper primarily addresses general 3D consistency issues rather than avatar-specific issues.

The focus of this section is not on text-to-avatar tasks (hence, we do not claim SOTA or compare with domain-specific methods) but on demonstrating the compatibility of 3D-Adapter with optimization-based 3D reconstruction methods and off-the-shelf ControlNets. Comparing 3D-Adapter with 2-stage and I/O sync baselines across various setups is critical, as different methods excel in different contexts. For example, I/O sync performs poorly in GRM-based text-to-3D setups but is competitive in optimization-based avatar generation and texture reprojection tasks, which is explained in Appendix A.2.

Additionally, these setups provide valuable insights, as noted by Reviewer RiGq in their comments on optimization-based 3D-Adapters.

Introducing per-step 3D depth Control Net prediction comes with computational overhead. The paper could explore strategies to mitigate this, such as reducing the frequency of applying the adapter or skipping initial steps. Since per-step VAE decoding is probably the main bottleneck, adopting a lightweight VAE alternative, e.g. TinyAE, could yield substantial speed gains.

This overhead is not unique to our 3D-Adapter, it also affects I/O sync methods. Acceleration strategies like skipping steps or using lightweight VAEs are orthogonal to our contributions and applicable to both 3D-Adapter and I/O sync. While they would improve efficiency, it is an engineering effort beyond the scope of our main contributions.

Some design choices around the ControlNet addition appear understudied. Introducing depth ControlNet itself may add a substantial 3D prior, but this effect is not adequately considered and is instead attributed mainly to ControlNet's role in denoising.

Again, we use RGB+D ControlNets, not just depth ControlNets. The main idea of 3D-Adapter requires the rendered images to be encoded and fused into the features of the base model without disrupting the original model topology, and ControlNet is an intuitive choice since its zero initialization is designed to minimize the impact to the base model.

Quantitatively, the ControlNet's effectiveness can be directly reflected by comparing B0 and C0 in table 1 (both use the same finetuned GRM). For texture and avatar generation, disabling ControlNet feedback is equivalent to the two-stage baseline and we have shown the differences.

Additionally, in the GRM branch, ControlNet is trained from scratch, which seems counterintuitive and is not supported with ablation studies.

Correction: The ControlNet is not trained from scratch but fine-tuned from the base model encoder, following standard ControlNet training. This preserves most of the model's knowledge and doesn't require extensive training. Additionally, for Instant3D and Zero123++, pretrained ControlNets are unavailable. We attempted to use existing Stable Diffusion ControlNets, but they performed poorly.

Comments on the questions:

What is the rationale behind the significant drop in metrics in Table 1 when adding IO sync and GRM finetuning to the two-stage baseline?

After adding I/O sync (A1), the problem is that the original GRM is incompatible with coarse intermediate outputs, and will lead to severe error accumulation. After finetuning the GRM (A2), the appearance metrics becomes worse but the geometry metric MDD improves significantly. The reason is explained in Appendix A2 (edited):

While I/O sync works reasonably on our texture generation benchmark, our text-to-3D model using I/O sync (A2 in Table 1 and Fig. 3) exhibits significant quality degradation due to mode collapse. We believe the main reasons are twofold. First, the base model Instant3D generates a very sparse set of only four views, which are hard to synchronize. Second, our fine-tuned GRM reconstructor is trained using the depth loss to suppress surface fuzziness, which has a negative impact when its sharp renderings $\tilde{\mathbf{x}}_t$ are used as diffusion output. This is because a well-trained diffusion model should actually predict blurry outputs $\hat{\mathbf{x}}_t$ in the early denoising stage as the mean of the distribution $p(\mathbf{x}_0|\mathbf{x}_t)$. Only in the late stage should $\hat{\mathbf{x}}_t$ be sharp and crisp, as shown in Fig. 8.

In light of this, in the revised PDF, we added a stronger I/O sync baseline (A3) with the dynamic blending technique.

As shown in Table 1 and Fig. 3, dynamic I/O sync demonstrates significant improvements in visual quality over vanilla I/O sync. Its MDD score indicates that its geometry consistency lies between vanilla I/O sync and 3D-Adapter. However, the visual quality of dynamic I/O sync is still clearly below that of the two-stage method and 3D-Adapter. While it is possible to tune a better blending weight $\lambda_t^\text{sync}$ , we believe it is very difficult to reduce the gap due to the aforementioned challenges brought by our model setup.

Edited: Please also note that I/O sync itself is not typically the first choice for 3D generation tasks. For example, in text-to-3D and image-to-3D tasks, most state-of-the-art methods (e.g., GRM, CRM, InstantMesh) employ two-stage approaches. I/O sync is more commonly used in texture generation tasks (e.g., SyncMVD, TexPainter), where our own I/O sync implementation outperforms others. Here, its drawbacks are thoroughly analyzed both theoretically in Appendix A.1 (where the linearity assumption holds due to linear texture blending) and empirically, as demonstrated in the experiments in Table 5.

Also GRM finetuning does not appear to be evaluated in the 3D-Adapter case.

Edited: 3D-Adapter uses the finetuned GRM by default. As shown in the I/O sync experiments (A2 vs A1), only the finetuned GRM improves geometry quality (indicated by the MDD metric and visualized in Fig. 8). Additionally, C0 vs A0 also reveals the contribution of the finetuned GRM in the 2-stage setting (C0 removes the feedback and effectively becomes 2-stage). Given these evidences, testing 3D-Adapter without GRM finetuning is unnecessary. However, if you're interested, we can test this setting for the rebuttal.

The SOTA selection in Table 2 seems debatable; could the authors clarify the choice of methods over more recent approaches?

Edited: LGM and GRM are already the latest ECCV 2024 SOTAs. In the revised PDF, we also added a new comparison (3DTopia). In general, text-to-3D is a less crowded field compared to image-to-3D, thus more SOTAs are found in the image-to-3D benchmark (Table 3).

How does the output quality compare to I/O sync + tiled ControlNet, and how critical is ControlNet's role during denoising relative to the 3D prior that could be introduced simply by depth conditioning in IO mode?

We added an experiment in Table 5 combining 3D-Adapter with I/O sync for texture generation. As expected, this combination performs worse than 3D-Adapter alone due to error accumulation and blur introduced by I/O sync (shown in Figure 5 and indicated by the drop in Aesthetic score). We tested this combination in the text-to-texture setup because I/O sync performs well here, whereas it significantly degrades text-to-3D results, making such a combination impractical. Again, this highlights the importance of testing 3D-Adapter across different setups, even if it adds complexity.