Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach

Thank you for the time, thorough comments, and nice suggestions. We are pleased that you acknowledged the novel idea, technical soundness, and effective experiments.

Q1: Some unclear explanations and descriptions.

A1: Thanks for your detailed review, here are our clarifications.

Q1.a: It is unclear why these two types of methods (GAN or MAE) need to run multiple times.

A1.a: (1) For previous methods, in their training stage, they outpaint images from the given central image 128x128 to 192x192, and the training settings are fixed. As a result, there are 2.25 ($\frac{128}{192}^2$) times more contents are generated.

(2) However, If we want to generate 57.6 ($\approx(2.25)^{5}$) times more contents, we have to forward the pretrained generative network 5 times ($1$->$f(x,)$->$2.25$-$f(x,)$->$(2.25)^{2}$->resize->$f(x,)$->$(2.25)^{3}$->resize)->$f(x,)$->$(2.25)^{4}$->resize->$f(x,)$->$(2.25)^{5}$).

(3) The reason for categorizing the existing methods into GAN-based and MAE-based is because GAN-based methods are difficult to converge and MAE-based methods are relatively more efficient due to the dropping of masked patches.

Q1.b: The authors fail to clearly explain why previous methods cannot outpaint images with arbitrary and continuous multiples.

A1.b: Arbitrary and continuous: For previous methods, they train their model by inputting the central sub-image, and the outpainting multiples (e.g., 1.5x) are pre-fixed in the training stage. Then, in the sampling stage, if we want to outpaint the image 1.7x, they have to retrain their model again. However, our PQDiff is much more flexible. Once PQDiff finishes training, PQDiff can generate images with any multiples.

Q1.c: Why relative positional queries can reduce the number of diffusion steps?

A1.c: The relative positional queries do not reduce the number of diffusions timesteps, but reduce the number of iterations for large multiple settings. As described in A1.a, previous methods have to forward the network 5 times to obtain 57.6x times more content. In our methods, for any outpainting multiples $n$, we only forward the network one times $1$->$f(x)$->$(2.25)^{n}$. Note that $f(x)$ includes all the diffusion steps.

Q1.d: In section 3, the authors say that they set h1 = h2 = h and w1 = w2 = w for notation simplicity. It looks improper.

A1.d: Thanks for your nice suggestions. Actually, the regions of the target view are larger than the regions of the anchor view. $h1 = h2 = h$ and $w1 = w2 = w$ are because we use the resize operation to conveniently feed the two views into transformers. We have corrected this expression in our new version.

Q1.e: The mismatch of the training and sampling in Figure 2 and Eq.1.

A1.e: There are some misunderstandings in the training stage. Actually, in the training stage, we feed the $z_{a}$, noised $z_{b_{t}}$ and relative positional embeddings as input. Please note $z_{b_{t}}$ is relatively equivalent to $x_{t}$ in the original diffusion model, which can be calculated by diffusion forward process (see line 9 in Alg.1 in Appendix B). Then, for the sampling stage, the inputs are the given sub-image (anchor image) $z_{a}$ , a randomly initialized Gaussian noise, and relative positional embeddings. For better understanding, we have modified the Fig.2. Please check our new version.

Q1.f: In Eq.2, it seems Xb0 should be Zb0 In Eq.2, it seems Xb0 should be Zb0

A1.f: Thanks for your detailed review. We have corrected the typo in our new version.

Q2: How does this one-to-many mapping influence the model?

A2: We are sorry for the confusion of the figure, and actually, the anchor-to-target is a one-to-one mapping. Since the relative position is one of the core parts of our method, we hope our clarification could help your understanding:

Instead of using image-level relative position, we will record the top-left positions for each patch of the target view, which means that the target view is determined. Take an example:

in the training stage, we take the anchor view as the starting point, suppose the positions of the anchor patches are:

Q3: In the sampling stage, if the input noise, image, and relative position are all fixed, how to ensure the outpainting ratios on the four edges are the same as what we expected?

A3: Due to the randomness of the relative positions between the anchor view and target view in the training stage, the model can learn almost all the positional relations between two views. Then, in the sampling stage, for example, if the position input sub-image is:

targetPos=\begin{pmatrix} (-h,-w)&(-h,w)&(-h,3w)\\ (h,-w)&(h,w)&(h,3w)\\ (3h,-w)&(3h,w)&(3h,3w)\ \end{pmatrix}.

Thanks for your detailed response. My confusions have been clarified. I have raised the my rating to 6.

Regarding my question 1.c, I understand that the relative positional queries are not used to reduce diffusion timesteps. But the related statement in the 4th paragraph in Introduction is really misleading. I suggest to explain these different "step" conceptions clearer in this paragraph.

Thank you for the quick reply and nice suggestions!

We have revised our new version by using timestep and step to distinguish the "diffusion step" and the "iteration" in our new revised version. Besides, we also add a footnote on page 2 in our new version for better understanding.

Please let us know if you have any further questions.

Thank you for the time, thorough comments, and nice suggestions. We are pleased to clarify your questions step-by-step.

Q1: Further discuss the potential of their method to other potential applications beyond outpainting to further expand the potential impact of this work.

A1: Thanks for your suggestions. We have updated the potential applications in our new version (page 22). We believe our method could be applied in image-inpainting, and super-resolutions if we properly change the relative positional embeddings in the training and sampling stages.

Q2: Are there other methods using positional query? As current existing work part does not mention them (if there were) too much.

A2: There are a few works learning about how to construct useful positional embeddings in transformers [1, 2, 3]. However, the motivation behind PQDiff is different from their methods. In the image generation area, to our best knowledge, we are the first to propose relative positional queries (RPE beyond the original region) to outpaint images arbitrarily and continuously.

Please let us know if there are further questions.

[1] Shiv V, Quirk C. Novel positional encodings to enable tree-based transformers[J]. NeurIPS 2019.

[2] Chi T C, Fan T H, Ramadge P J, et al. Kerple: Kernelized relative positional embedding for length extrapolation[J]. NeurIPS 2022.

[3] Su J, Lu Y, Pan S, et al. Roformer: Enhanced transformer with rotary position embedding[J]. arXiv preprint arXiv:2104.09864, 2021.

Reviewer HK74, did the authors' rebuttal had addressed your concerns?

Please reply and post your final decision as well.

AC

Thanks for your constructive suggestions. Your endorsement of our method and experiments gives us significant encouragement. Here are our clarifications.

Q1: Is it precise to say that PQDiff handles inputs with various sizes and ratios while the output size is fixed?

A1: Yes. As we adopt diffusion models, the output sizes are fixed. If the users want to generate images with 10x resolutions, PQDiff has to leverage an additional decoder (e.g., super-resolution).

Q2: Can the author provide results on more multiple besides (2.5x, 5x, 11.7x) used in other sota to justify their claim of continuous multiples. My suggestion would be a figure of FID over continuous multiples or spectrums of outpainting images with increasing multiples.

A2: Thanks for your constructive suggestions. We have added the figure over continuous multiples with increasing multiples in our new version (see Fig.7 in the Appendix).

Q3: What is the quantitative evaluation of Sec 4.2 Outpainting in an arbitrary position, i.e. FID under different positions? Does the model prefer certain positions?

A3: We conduct a set of experiments when outpainting with arbitrary positions. Specifically, we randomly crop the anchor view and target view, and by using RandomCropResize. The crop ratios of the anchor view and target view are $(0.4, 0.4)$ and $(0.9, 0.9)$, respectively. This setting aims to keep PQDiff generating at least 2.25 times contents (only when the target view contains the anchor view, the generated region are the same with centered outpainting. Otherwise, PQDiff will generate more regions than centered outpainting). Here are the results:

Please note that due to the random location, it's difficult to add "copy" operations for arbitrarily outpainting. We find PQDiff (arbitrary position) obtains a bit lower IS score than PQDiff (Center position), which is because the generated region of PQDiff (arbitrary position) is larger than PQDiff (Center position) on average.

Q4: The generated and input images in Supp. Sec. J (Fig. 11, 12, ...) can be put side by side for clearer comparison.

A4: Thank you for your suggestion. We have put the Sec.K (Appendix) side by side for clearer comparison in our new version.

Please let us know if there are further questions.

Acknowledged we have received your response. We will take those into consideration.

Thanks, AC