Harnessing Attention Prior for Reference-based Multi-view Image Synthesis

The paper needs to improve readability by structuring the content more logically.

1. The definition is a bit confusing. Why do we need the concept of local synthesis and global synthesis for novel view synthesis? What’s the core challenge? Besides, the claimed local synthesis is actually the reference-guided inpainting task, and the proposed ARCI is built upon an inpainting model, which makes the target more like inpainting.  

2. The presentation is not clear enough. for example, in Figure 1, the authors should explain what the purple mask means. What is the input of the model, the green bounding box or the purple one? The inputs and outputs look very different in (a)-(d).

3. The authors claim in the introduction that they can use efficient task and view prompt tuning for novel view synthesis with frozen SD, However, in the experiments, the authors finetuned the whole stable diffusion and reported some results with Lora and fully finetuned results. I think the authors should grandly review and improve their presentation and make them clearer.

4. The comparison in Figure 5 is not fair. The authors should report the original results of Zero123 instead of re-training it with a suboptimal training setting. Besides, zero123 is able to take specific camera poses to generate the novel view, what’s the input used here for both zero123 and ARCI? The authors should make sure all the models take appropriate inputs.

5. Some important experiments are missing. The authors should conduct an ablation study on adaptive masking. For comparison with zero-123 for novel view synthesis, the authors should show comparisons on GSO and RTMV following zero-123. For reference-based inpainting, the authors should also show comparison results by using some very different references following the setting of Paint-by-Example to verify the effectiveness of the ref-inpainting technique for open-domain cases.

6. What’s the used task and view prompt? Specifically, do the task prompts indicate local or global? How to set the view prompts? Rotation angles or front/side/back view? If the model tasks as input an image beyond these views, then how does the ARCI work?

1. The writing is too bad. The paper has many confused words, unclear explanation, and unconvincing arguments. Here are some examples:
   • Confused words. In abstract, "Our contributions of ARCI, built upon the Stable Diffusion fine-tuned for text-guided inpainting, include skillfully handling difficult multi-view synthesis tasks with off-the-shelf T2I models, introducing task and view-specific prompt tuning for generative control, achieving end-to-end Ref-inpainting, and implementing block causal masking for autoregressive NVS. " The sentences indicates there are 3 contribution: 1) handling difficult multi-view synthesis tasks with off-the-shelf T2I models, 2) introducing task and view-specific prompt tuning for generative control, achieving end-to-end Ref-inpainting, 3) Implementing block causal masking for autoregressive NVS. The phrase 'block causal masking' is first introduced here, which confuses readers. In addition, why implementing this can be counted as a contribution? I suggest the author explain what the block causal masking is first.
   • Unclear explanation. In introduction, "This task can be broadly categorized into two facets: local and global multi-view image synthesis from reference images. " If I understand Figure 1 correctly, there seems to be 2 tasks, ref-impainting and novel view synthesis here, but the author write one task with 2 facets. I think this is a confusing definition.
   • Unconvincing arguments. In introduction, 'They struggle to capture fine-grained correlations, including object orientations and precise object locations, between reference and target images. These nuanced details are pivotal for tasks such as multi-view generation, as exemplified by Ref-inpainting.' Is there any reference paper or experimental results supporting this arguments? The above shows 3 typical examples and there are many more here and there. The paper writing does not satifiy ICLR bar and requires major re-writing.
2. What's the motivation of this paper? 'However, adapting them for multi-view synthesis is challenging due to the intricate correlations between reference and target images.' This sentence is correct, this is not the problem of current methods, e.g. Zero-1-to-3. Why does this proposed method better than zero-1-to-3. I suggest the author to write the motivation clearly in abstract.
3. In related work, 'Compared with these aforementioned manners, the proposed ARCI enjoys both spatial modeling capability and computational efficiency.' What is spatial modeling capability? Please define spatial modeling capability.
4. In method, please define and formulate the two tasks first.
5. ATTENTION REACTIVATED CONTEXTUAL INPAINTING is not intuitive. It's hard to image the technique by the name. I recommentd to use more intuitive method name.
6. The method design seems to be complicated or the method writing is bad, which make the method design look complicated.

• The biggest issue of the paper is the writing quality, which makes the paper very hard to follow. Details are listed below.

  1. The introduction is extremely long and poorly organized. Many points are made, but I cannot find a precise sentence that emphasizes the essential contribution of the paper. The two applications (Ref-inpainting and NVS) seem to stem from the unified ARCI approach, but the introduction always tries to separate them when discussing their challenges and claiming improvements.

  2. While the introduction makes separate claims for the two tasks, the descriptions of the two tasks in Sec.3 are heavily entangled, making it hard to clearly understand how the model works for each task.

  3. Fig.1 to Fig.3 are very difficult to parse. The texts in the figures are too small. The inputs and outputs for each task are not clearly explained. The captions are not self-contained, and it is also very hard to link them to certain parts of the main text.

• There seems to be no quantitative evaluation for multiview image generation -- Table 2 of the main paper only provides results with a single target view. Since the paper claims improvements in multiview image generation, it is important to formally evaluate the consistency of the generated multiview images.

• The proposed ARCI is limited by the autoregressive generation design and cannot produce many multiview images. While a potential tradeoff is to constrain the length of the condition, it would definitely sacrifice the quality/consistency of the generated images. Note that an important goal of multiview image generation is to extract the 3D object/geometry. Both the number of views and the multiview consistency are important when exporting a 3D model from the generated images. The proposed designs are suboptimal compared to recent works such as SyncDreamer and MVDiffusion, which can generate 16 or more views in parallel.

• Although the authors claim that ARCI outperforms Zero123 in novel view synthesis (NVS), Zero123 itself is not merely an NVS model. Zero123 can serve as a general diffusion model backbone with 3D shape/global priors, which facilitates many other approaches via fine-tuning or score distillation (e.g., Magic123, One-2-3-45, SyncDreamer, etc). It is true that ARCI outperforms Zero123 in NVS, especially when the training budget is limited, but the scope of ARCI is much narrower than Zero123 given the more complicated designs.