Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering

We sincerely thank the reviewer for insightful feedback and comments.

1. Setting the Hyperparameters R and alpha.

All hyperparameters were tuned based on a small random set of objects. Due to limited computational resources, the tuning process was not exhaustive. We found that the method's performance is not highly sensitive to these parameters.

• Cross-step weight (alpha): We experimented with values ranging from 0.1 to 0.9 in increments of 0.1. Alpha determines the weight assigned to previous predictions in the cross-step aggregation. In Zero123-XL, early predictions were generally reliable, so a larger weight yielded better results. We maintained the same parameters for Zero-1-to-3. In new experiments conducted for the rebuttal, we applied our method to additional models such as ControlNet and MVDream. In both cases, a smaller weight generally produced better outcomes.

• Resampling iterations (R): We observed that the model's performance is relatively insensitive to the choice of R, with values between 4 and 8 yielding similar results. While some objects benefited from larger values (~10), the overall improvement was minimal. Additionally, as shown in Figure 8 in the paper's appendix, large values of R (~15-20) can reduce diversity. For additional experiments with models like ControlNet and MVDream, we fixed R at 5 and did not test other values.

2. Additional Computational Cost of Resampling.

Resampling iterations are computationally equivalent to adding more denoising steps, as both linearly increase the number of function evaluations (NFE). In the paper, we addressed this by counting the NFE and keeping it comparable to the base model to ensure a fair comparison. Our chosen value for R, and the specific timesteps where we applied it, resulted in mapping 26 denoising time steps to a total of 66 NFE. For further details, please refer to our response #1 to reviewer UpfM, where we discuss the individual computational overhead of each module.

3. The Extent of Quantitative Improvement. Please refer to General Comment #4 for a detailed discussion of the quantitative improvements.

4. Evaluating on Additional Dataset.

We evaluated our approach on the RTMV benchmark [1], an out-of-distribution dataset consisting of 3D scenes, each contains 20 random objects. Here, we present the results for Zero123-XL, using the same hyper-parameters reported in the paper. Our evaluation shows improvements across all metrics compared to baselines. GSO and RTMV are the common datasets used for evaluation in Zero1-to-3 and its follow-ups.

5. Including all objects in the ablation study.

Thank you for your suggestion. In response, we have conducted the ablation study across all test objects, focusing on Zero123-XL. The new results corroborate the trends observed in our initial submission. We will ensure that the comprehensive evaluation is included in the camera-ready version.

6. The usage of Mutual Self-Attention in Zero-to-Hero.

We acknowledge that the formulation of MSA was not introduced in our work. We have cited prior works such as MasaCtrl in our paper, and we will include the additional references suggested by the reviewer. However, we wish to highlight the distinctions between previous works and our unique application of this technique. To our knowledge, most prior works utilizing MSA fall into two categories:

• Training-free usage of MSA (e.g., MasaCtrl): In these works, MSA is typically applied after the general structure of the target is formed to transfer appearance details from the input to the target. In this scenario, MSA does not contribute to the initial structure formation of the target.
• Training or fine-tuning models with MSA layers (e.g., Consistent123, Tune-A-Video): These works incorporate MSA within the training or fine-tuning process.

Our approach is distinct as we employ MSA in a training-free manner, but crucially, we apply it from the beginning of the denoising process until the target structure stabilizes—a phase we term "Early-Stage Shape Guidance" in our paper. This early application of MSA leads to more stable results and often prevents the model from generating out-of-distribution outputs (like the leftmost and rightmost chairs in Figure 1 in the paper).

To illustrate the impact of early-stage MSA on structure, we conducted a simple experiment using Zero123-XL with 50 DDIM steps. We measured the effect of activating MSA at different stages of denoising on Intersection over Union (IoU) metric, noting that image quality metrics improved similarly. Our approach demonstrated a significant improvement in the structural integrity of the image. We believe that applying MSA in the later stages of the denoising process introduces a bias towards the input images, which can disrupt the shape and appearance.

7. The Contribution of MSA vs. AMF.

MSA and AMF address different artifacts and complement each other. MSA's greater impact on metrics does not diminish AMF's value. The combined methods show consistent improvements. For more details, please see General Comment #1 and Figure 1 (right) in the supplementary materials.

[1] RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis.

We sincerely thank the reviewer for insightful feedback and comments.

1. Additional Qualitative Results.

We kindly refer the reviewer to General Comment #1 and the supplementary material, where we have included further qualitative results.

2. Applicability of Zero-to-Hero in Multiview Diffusion.

Thank you for raising this critical point. While Zero-to-Hero was initially designed for single-view generative models, our Attention Map Filtering can indeed be extended to other diffusion models, including multiview synthesis. Based on your suggestion and that of reviewer qQr5, we have implemented the attention map filtering mechanism for MVDream and two pre-trained ControlNet models, observing significant improvements. Please refer to General Comment #2 for more details on these implementations.

3. Quantitative Evaluation — Additional Details.

For our quantitative evaluation, we rendered 8 random views of each object and used each view as a source. For each source view, we generated the remaining 7 views, resulting in a total of 56 generation tasks per object (each defined by a unique source-target pair). Each target view was generated 3 times with different seeds, yielding a total of 168 images per object. We then averaged the scores across all views and objects to ensure a comprehensive evaluation.

4. Effect of Zero-to-Hero on Generation Diversity.

In general, excessive use of attention map filtering might reduce generation diversity. We have analyzed this in the appendix of our paper (Figures 8 and 9). However, we find that responsible usage usually preserves diversity. For instance, in Figure 1 of the main paper, the generated chairs display diverse back sides, and the turtles in the third row further demonstrate the model's ability to produce varied results. The balance between diversity and fidelity is complex and warrants further study. Often, the "diversity" observed in the base model includes artifacts and deviations from the real-world distribution, as illustrated in Figure 1 of the main paper and Figure 1(left) in the supplementary material.

Thank you for your comprehensive rebuttal. The majority of my concerns have been addressed, particularly regarding the details of the Quantitative Evaluation, the applicability of Zero-to-Hero in Multiview Diffusion, and the additional qualitative results. However, I remain not entirely convinced about the purported balance between "diversity" and "fidelity". The attention map filtering is applied to a multi-view diffusion model, whose diversity should be largely preserved to maintain its non-deterministic nature as a diffusion model. Considering all these factors, I have decided to revise my score from borderline reject to borderline accept. I intend to consult other reviewers' discussions before making a final decision and I am more than happy to discuss with other reviewers and AC, SAC and PC about the diversity concern.

We sincerely thank the reviewer for insightful feedback and comments.

1. Effect of AMF on Generation Diversity.

We agree that excessive use of Attention Filtering (R>>1) may reduce generation diversity, as analyzed in the appendix. We will incorporate these insights into the limitations section. The balance between diversity and fidelity is a complex topic that warrants further study. Our findings indicate that "diversity" in the base model often includes artifacts and deviations from the real-world distribution, as illustrated in Figure 1 of the main paper and Figure 1 (left) of the rebuttal Supplementary Material. Responsible use of Attention Map Filtering usually helps preserving diversity while preventing results from deviating from the real distribution.

2. Generality and applicability to other diffusion models.

Thank you for highlighting this important point. Although our work focuses on novel view synthesis, we agree that the AMF module is potentially broadly applicable. This was indicated in the future work section. Inspired by your comment and by the request of reviewer 3GQx, we have extended Attention Map Filtering to other models beyond the task of novel view synthesis. Remarkably, we achieved noticeable improvements out-of-the-box in all cases. Please refer to General Comment #2 for more details.

3. Visual details in generated views.

We have not observed a general trend of lacking visual details in the generated views. Figures 1 in both the main paper and the supplementary materials demonstrate examples with fine details, such as the Android's antennas and the squirrel's eyes. The loss of the chicken's eye in one example may be due to resampling issues. While our pipeline usually mitigates the oversmoothing effect of vanilla resampling, it is not infallible.

4. Computational efficiency.

We acknowledge the reviewer's point regarding computational efficiency. Our primary goals were to improve quality and consistency while keeping generation times competitive to ensure applicability. We will clarify our claims in the paper and are open to further adjustments based on the reviewer's suggestions. An analysis of the computational cost of each module is provided in the response to reviewer UpfM.

We sincerely thank the reviewer for their insightful feedback and comments.

1. Inference cost analysis

Our proposed modules add a computational overhead to the base model. In the paper, we addressed this by counting the overall number of function evaluations (NFE) and keeping it on par with the base model to ensure a fair comparison. As requested, we now discuss the individual computational overhead of each module. We will incorporate this analysis into the manuscript for completeness.

(1) Resampling: Similar to the total number of denoising steps, T, the number of resampling iterations, R, linearly increases the NFE. Our chosen value for R and the timesteps at which we apply it resulted in mapping 26 denoising steps to a total of 66 NFE.

(2) MSA: The main additional cost of MSA is the necessity to generate the input view in addition to the target views, meaning the effective number of generated samples is increased by one.

(3) AMF: We implement attention map filtering in the upsampling part of the UNet and apply it during the early steps of the denoising process. We maintain two additional instances of each attention map: the attention map from the previous timestep (for cross-step updates) and the refined map from the current timestep (for in-step updates).

We provide a table showing the running times (in seconds) of Zero-1-to-3 and Zero-to-Hero for the same NFE (66). Overall, we manage to provide competitive running times. If both MSA and AMF are active (requiring the generation of the input), the running time is increased by approximately 1-1.5 seconds. If only AMF is active, the overhead is much smaller, averaging around 0.5 seconds.

2. Additional qualitative results.

Please refer to Figure 1 in the supplementary material for additional qualitative comparisons of view synthesis by our Zero-to-Hero and the baseline Zero-123-XL. As can be seen, the improvement demonstrated by our method is consistent across objects and seeds. Further qualitative results demonstrating our method on other base models are discussed in General Comment #1.

3. The extent of quantitative improvement.

We have provided a clarification in General Comment #4.

4. Qualitative demonstration of MSA and AMF.

We appreciate this suggestion. Indeed, the difference in quantitative improvement between these two modules does not faithfully reflect their effect. We thus chose to share the response with all reviewers in General Comment #1 and Figure 1 (right) in the supplementary material.

5. The effect of MSA and AMF on zero123 and zero123-XL.

This is a good observation. Firstly, regarding the different effects of MSA and AMF, please refer to General Comment #3. Here, we address the reviewer's inquiry regarding the difference in the modules' effects on both base models. Our MSA (MSA with early termination) generally improves performance in both base models. As pointed out by the reviewer, in Zero-1-to-3, MSA did not improve PSNR and IoU while boosting SSIM and LPIPS. In Zero123-XL, the improvement was consistent across all metrics. A potential explanation for this difference may lie in the inherent bias of MSA towards the input view. Roughly speaking, MSA copies source details to improve generation consistency. Thus, too much of it might impair the results as it would lead to a bias towards the appearance and pose of the input image. To control this effect, we introduced early stopping. The termination step in our work was chosen based on Zero123-XL (where MSA consistently improved all metrics). While we found that the same parameters generalized well for the overall performance of Zero-1-to-3 without further tuning (the combined MSA and AMF improved all metrics and addressed the same issues as Zero123-XL in visual results), the ablation reveals that the parameters may not be optimal for the MSA effect alone in Zero-1-to-3.

6. Choice of resampling steps R.

We found that the model is not very sensitive to the choice of R, and values within the range of 4-8 provide similar results. While some objects benefited from larger values (~10), the overall improvement was minimal. Additionally, we found (as shown in Figure 8 in the appendix of the paper) that large values of R (~15-20) can limit diversity.