Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: More results of perspective input.
We appreciate this suggestion and have accordingly expanded our results section. In Figure 2 of the global response, we include more results on GSO datasets and cases from the Internet. Compared with Wonder3D and more recent Unique3D, our method demonstrates superior performance in mitigating distortion and reconstructing plausible 3D shapes. As per your recommendation, we will incorporate more results into the main paper.

Q2: The presentation about EFReg is explicit. Is the better performance from EFReg or preprocessing of training data?
In Fig.7 of the main paper, we evaluate the effectiveness via qualitative comparison by removing EFReg. Notably, without EFReg, the resulting shape is distorted and fails to generate reasonable novel views in the canonical camera setting. To provide quantitative support for these observations, we additionally report the Chamfer Distance in Tab.1. We use orthogonal renderings at an elevation of 0° as the reference (last column) and vary the elevation from −10° to 40° and select focal lengths from {35, 50, 85, 105, 135, ∞} to assess the system’s robustness to distortions. These results consistently confirm that EFReg significantly contributes to robust pose estimation and enhances overall reconstruction accuracy.

Table 1: Ablation of EFReg on GSO datasets with various elevation (α) and focal lengths (f). We report the Chamfer Distance (↓).

Q3: Missing metrics of PSNR and SSIM.
We apologize for the missing reports of PSNR and SSIM. Following your valuable suggestion, we have now incorporated these results in Tab.2 to provide a more comprehensive evaluation following your suggestion. Our method significantly outperforms others, consistent with our performance on other geometry metrics.

Table 2: Quantitative evaluation of SSIM and PSNR.

Q4: Inaccurate claim of 'the first method to solve perspective distortion'.
We acknowledge that our initial statement requires refinement. The study in CVPR'23 attempts to predict the distortion in category-specific image reconstruction. VQA-Diff only explores vehicle distortion in Autonomous Driving. In contrast to them, our study first considers distortion artifacts for general 3D object generation. We will revise the statement in the manuscript.

Q5: Does RMA lead to performance degradation compared with dense/epipolar MV attention?
Theoretically, row-wise, epipolar, and dense MV attentions are equivalent in our orthogonal setup. However, both dense MV attention and epipolar attention consume a large amount of GPU memory in the training, which cannot scale up to high resolution 512$\times$512 and largely reduces the training batch size. As observed by Zero123 and other works, a large training batch size is very important for training a high-quality diffusion model. Due to the memory limitation, we cannot conduct a fair comparison experiment to train the model with dense MV attention or epipolar line attention.

Q6: Era3D does not fully address the distortion issue. It should be included as a limitation.
Thanks for your kind reminder. We use the commonly used focus lengths for training Era3D, which significantly mitigates perspective distortions. We acknowledge that our work offers a promising avenue for addressing this challenge rather than fully resolving the issue. We will clarify this point and discuss it in the limitation section.

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Row-wise attention is just epipolar attention and is not the contribution of this paper.
We respectfully disagree with this characterization. As discussed in Line290 of the submission, the vanilla Eipolar attention adopted by previous methods significantly increases memory and time consumption due to their need for sampling points along the epipolar lines, which makes them even slower than the dense multiview attention. Era3D is the first attempt to simplify epipolar attention, which makes it extremely efficient for generating high-quality 3D objects. Thus, we consider it a core contribution of our work.

Q2: Lack of comparison with SV3D and Unique3D.
While we acknowledge the importance of providing comprehensive comparisons, Unique3D and SV3D are unpublished technical reports on arXiv. Both of them are concurrent works that released their codes near or after our submission. Despite it, we provide an additional comparison with Unique3D in Fig.2 of global response, which shows that Unique3D still suffers from perspective distortion while our method greatly alleviates this problem. We appreciate the reviewer's feedback regarding these comparisons.

Q3: There are no other supplementary materials other than the video and response to the reviewer's comment regarding the violation of the double-blind policy.
We respectfully disagree with the assessment. We strictly follow the NeurIPS 2024 guideline to attach the supplementary materials at the end of the main paper, including additional experiments and descriptions of our method. We also included an anonymous link at the end of the abstract to show our results without compromising the double-blind review process. We did not encourage or invite reviewers to search for our submission, and thus, we maintain that we have adhered to the double-blind reviewing policy as stipulated by the conference guidelines. We appreciate the reviewer's feedback and clarification on this matter.

Q4: Selection of features for EFReg.
We apologize for any lack of clarity in our presentation. For EFReg prediction, we utilize the final layer feature map of the intermediate UNet block with the lowest resolution since it could provide high-level global information.

Q5: Can the proposed method directly generate perspective images?
Our method is designed for 3D generation, which generates images around an object with orthogonal cameras. Thus, generating perspective images falls outside the scope of our paper. Moreover, generating perspective images will introduce additional challenges, such as varying scale prediction across multiple views, which would significantly increase the complexity of 3D generation. That's why existing works, including ours, focus on orthogonal setups rather than perspective ones.

Q6: Not sure the novelty of the paper is enough for publication.
As elaborated in Q1, our row-wise attention is distinct from the custom epipolar attention in previous studies. The proposed row-wise attention demonstrates a significant enhancement in training efficiency. SV3D is our concurrent work and requires considerably more resources for training. Therefore, we restate our contributions as follows:

1. Era3D is the first method that tries to solve the distortion artifacts in general 3D generation tasks;
2. We design the novel EFReg to enable diffusion models to take distorted images as inputs while outputting the orthogonal images on the canonical camera setting;
3. We propose row-wise multiview attention, an efficient attention layer for high-resolution multiview image generation;

We believe our designs could advance the field and inspire other 3D generation models.

Thank you for your valuable feedback! We try to address your issues in the following discussion.

Q1: Proof of Row-wise.
Please refer to the 'A.2 Proof of Proposition 1' section (on Page 18) in our initial submission for the detailed proof.

Q2: Selection of features for EFReg.
As mentioned in response to Q4, we select 'the final layer feature map of the intermediate UNet block' rather than 'the final layer feature map of the whole UNet'. Therefore, the feature has the lowest resolution.

Q3: Works on orthogonal setups.
The works on orthogonal setups include but are not limited to:

• (ICLR'2024) MVDream: Multi-view Diffusion for 3D Generation.
• (Arixv'2024) Unique3D: Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image.
• (ECCV'2024) CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model.
• (CVPR'2024) EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors.
• (CVPR'2024) Wonder3d: Single image to 3d using cross-domain diffusion.

We can further address unclear explanations and remaining concerns if any.

Once more, we appreciate the time and effort you've dedicated to our paper.

Thank you for your valuable time and insightful comments! We have tried to address your concerns in the updated manuscript and our rebuttal text:

Q1: Comparison with LRM-based methods.
Tab.1 showcases the comparisons of Era3D with OpenLRM and CRM on the GSO dataset, which shows that Era3D exhibits better performance than these two methods. Our method can be incorporated with LRM-based methods because recent LRM-based methods rely on multiview generation as the input while Era3D can provide generated multiview images.

Table 1: Quantitative comparison with LRM-based methods:

Q2: Missing details of inference time and hyperparameters.
The whole process requires approximately 4 minutes, comprising 13 seconds for the multiview diffusion, 3 minutes for the NeuS reconstruction, and 10 seconds for the texture refinement. Our diffusion model employs a 6-layer MLP for pose estimation. We use the same hyperparameters as SD2.1, except what is explicitly stated in the 'Implementation Details' section. For the NeuS reconstruction, we use the same settings as Wonder3D. During texture refinement, we optimize the appearance for 200 iterations with a resolution of 1024 and a learning rate of 1e-3. We will add these details to the revision.

Q3: Absence of robustness testing.
We comprehensively evaluate the robustness of our pose prediction on the GSO dataset and the generation quality for in-the-wild objects in Tab.5, Tab.6, and Fig.11 of ' Supplementary Material'. We will also release our code for public testing and evaluation.

Q4: Why select generation viewpoints of elevation 0? Would this affect the reconstruction of the bottom?
In contrast to the setup with various elevations employed in Zero123++, Era3D is trained to generate six views at an elevation of 0. This choice is because row-wise attention requires a fixed elevation. Intuitively, models perceive an elevation of 0 more readily than a random elevation. While this setting may affect the objects with planar bottoms, such cases are uncommon. Note that the prior works such as SyncDreamer and Wonder3D also use similar pre-defined viewpoints.

Q5: Why only generate 6 views?
Generating dense views necessitates substantial memory for training. Fig.1 in global response illustrates sparse-view reconstruction results using 2, 4, and 6 views. Our method, utilizing 6 views, consistently produces complete and plausible 3D shapes.

Q6: Insufficient ablation of EFReg.
We utilize the intermediate feature map with the lowest resolution in UNet for EFReg prediction since it could provide high-level global information. The effectiveness of EFReg is evaluated qualitatively in Fig.7 of the main paper. To provide quantitative support, we additionally report the Chamfer Distance in Tab.2. These results confirm that EFReg facilitates robust pose estimation and enhances reconstruction accuracy.

Table 2: Ablation of EFReg on GSO datasets with various elevation (α) and focal lengths (f). We report the Chamfer Distance (↓).

Q7: Provide additional visual comparison with baselines in the demo video.
We appreciate the reviewer's suggestion. We will incorporate comparisons in the demo video to provide a more comprehensive visual presentation of our method's performance relative to existing approaches.

Q8: Confusing claim about computation complexity reduction of row-wise attention.
Compared to Dense MV attention, our row-wise attention reduces the computation complexity by 12 times. We will clarify this in the Abstract. In Tab.3, we list the memory usage and running time of each part of our pipeline, in which other parts include self attn, cross attn, and feed-forward layers. Notably, Dense MV attention layers constitute approximately 60% of the memory footprint and 75% of the running time in the overall pipeline. Our row-wise attention substantially mitigates these computational demands, with particularly remarkable improvements in execution time.

Table 3: Memory usage and running time of the pipeline with 512 resolution and xFormer.

I greatly appreciate the author’s detailed rebuttal, which effectively addressed nearly all of my concerns. However, based on my understanding of the field and feedback from another reviewer, I have some reservations about the paper’s novelty. Given the substantial body of existing literature in this area, I remain cautious about the contribution of this paper. Therefore, I will maintain my current score of 6.

Dear reviewers, as we approach the final two days, please take a moment to review the author's responses and join the discussion. Thank you!