Image Inpainting via Tractable Steering of Diffusion Models

We believe that the author should explain in the introduction and method sections whether the method proposed is training or non-training. … I cannot fully understand the details of the method proposed in this paper (e.g., how were the weights in the PC obtained)

Thank you for pointing out aspects of the paper that were not explained clearly in the main text. The PC used in the proposed inpainting algorithm needs to be first trained (unconditionally) on the datasets (e.g., CelebA-HQ, ImageNet, and LSUN-Bedroom) using the MLE objective: $\max \sum_{\mathbf{x} \in \mathcal{D}} \log p_{n} (\mathbf{x})$, where $p_{n}$ is the PC and $\mathcal{D}$ is the dataset.

Following prior art, a natural way to optimize PC parameters w.r.t. the MLE objective is by the Expectation-Maximization algorithm since PCs can be viewed as a latent variable model with a hierarchically nested latent space. The implementation of the EM algorithm is very similar to the backpropagation algorithm since the feedforward computation following Definition 1 can be treated as a computational graph. To update the parameters, we first run the standard backpropagation algorithm to compute the gradient of all parameters (w.r.t. the MLE objective, which is the output of the feedforward PC computation). Suppose the gradient of $\theta_{n,c}$ is $g_{n,c}$, for every sum node $n$, we update all its child parameters by

$\theta_{n,c} \leftarrow (1 - \alpha) \cdot \theta_{n,c} + \alpha \cdot \frac{g_{n,c} \cdot \theta_{n,c}}{\sum_{m \in \mathtt{in}(n)} g_{n,m} \cdot \theta_{n,m}},$

where $\alpha$ is the step size. In addition to the EM update, we follow prior arts and use the LVD technique to initialize the parameters. The full details of the learning process of PCs are included in Appendix C of the revised paper. At the end of Section 4.1, we also added a paragraph to explain the PC learning pipeline and points to the Appendix for more details.

We are happy to make further clarifications if there are other unclear parts in the paper.

LPIPS alone may not be sufficient to assess the quality of generated images.

We conduct a user study based on all three datasets and two masks (“Expand1” and the newly added “Wide” masks adopted from [1,2]). Detailed settings and results are added in the updated paper (Appendix E.1). We copy the result table from the paper:

|      Dataset & Mask      |   Tiramisu   | CoPaint | RePaint | DPS |

| CelebAHQ & Expand1 | Reference |      22      |      34      |  14   |

| CelebAHQ & Wide       | Reference |      26      |      30      |   22   |

| ImageNet & Expand1 | Reference |      32      |      20      |   24   |

| ImageNet & Wide       | Reference |      14      |       6       |   12   |

| LSUN & Expand1         | Reference |      18      |       2       |   8   |

| LSUN & Wide                | Reference |        4        |       6       |   -6   |

We use the vote difference (%) metric, which is the percentage of votes to Tiramisu subtracted by that of the baseline. A positive vote difference value means images generated by Tiramisu are preferred compared to the baselines, while a negative value suggests that the baseline is better than Tiramisu.

We compare against the three strongest baselines (CoPaint, RePaint, and DPS) as their average LPIPS scores are not significantly worse than Tiramisu. As shown in the table, in all but one task the vote difference score is positive (and often quite large), which indicates the superiority of Tiramisu compared to the baselines.

We also tried the metrics suggested by the reviewer. However, FID is not specific to image pairs and is less informative for image inpainting tasks; when using the official U-IDS metric, we found that in most cases we got a U-IDS close to 0 and the results are not very informative.

We thank the reviewer for their constructive feedback.

How about the potential of this method for general conditional generation?

There are various conditional/constrained image generation tasks that could be done by (variants of) the proposed method. As a first attempt at using TPMs (PCs) for conditional image generation tasks, this paper uses image inpainting to demonstrate the effectiveness and potential of the proposed method. In the following, we list several classes of conditional generation tasks that could be accomplished by the proposed method or some (non-trivial) extension of our method.

• Other types of constraints belong to the independent soft evidence constraint family (formally defined in Section 4.2). For example, the constraint of image super-resolution and deblurring could be written in the form of such constraints and thus can be tackled by the proposed method.

• Other inverse problems that can be formulated into “simple” constraints could potentially be tackled by a generalized version of the proposed method thanks to the ability of PCs to efficiently condition on certain logical constraints. For example, in image coloring, the constraint would be that for every pixel, its R, G, B value have a fixed weighted sum (specified by the input gray-scale image).

• Controlled generation tasks without formally described conditions. For example, the semantic fusion task described in Section 6.3 aims to “fuse” the semantics of several reference images while maintaining the “naturalness” of the resultant image.

Here, more experimental analysis of the TPM steps are expected, including the effect on generation quality, semantic coherence, and computational efficiency.

Thank you for the suggestion. In the updated paper (Section 6.2, the “computational efficiency” paragraph), we provide an ablation study over the % of denoising steps using PCs, and report the LPIPS score and runtime-per-image for each case. We adopt the “CelebA-HQ + the Expand1 mask” task. As shown in Figure 4 of the updated paper (for convenience, we copied the results in the figure to the table below) as we engage the PC in more denoising steps, the LPIPS score first decreases and then increases. Therefore, incorporating PCs in a moderate amount of steps gives the best performance (around 20% in this case). In terms of computational efficiency, the additional computation cost is around 10% when we engage the PC for the first 20% denoising steps and around 25% when the PC is used in the first 50% denoising steps.

One explanation for the diminishing performance gain when using PCs in too many denoising steps is that the later stages of denoising are mainly responsible for refining the details in the image. And the guidance of PCs could be less effective in this aspect. This could be the consequence of using latent space PC (latent space generated by VQ-GAN). In the future, we will explore possibilities to train good PCs directly in the pixel space. This could lead to better performance in terms of refining details in generated images.

| % denoising steps using PC |     5     |    10    |     20    |    30    |    40    |    50    |

|                     LPIPS                     | 0.470 | 0.459 |   0.454 | 0.456 | 0.464   | 0.476 |

|                  Runtime                  |   109   |   111   |   115   |   118   |  121   |   125   |

Limitations and Failure Cases: A discussion of the method's limitations and failure cases are not addressed in this research, which would be beneficial for the application of this work.

As described in the response to the above question, engaging the guidance of PCs in too many denoising steps could lead to worse performance. This suggests that the current method may not be good at refining details. Therefore, Tiramisu could be less effective when dealing with small-hole inpainting.

Another potential weakness of the proposed method is the expressiveness of PCs. While we are able to train good PCs on datasets such as ImageNet 256*256, it is not sure whether this could be generalized to higher-dimensional images.

We thank the reviewer for their constructive feedback.

So instead of claiming 10% additional computation overhead, a detailed analysis is more helpful.

Thank you for the suggestion. In the updated paper (Section 6.2, the “computational efficiency” paragraph), we provide an ablation study over the % of denoising steps using PCs, and report the LPIPS score and runtime-per-image for each case. We adopt the “CelebA-HQ + the Expand1 mask” task. As shown in Figure 4 of the updated paper (for convenience, we copied the results in the figure to the table below) as we engage the PC in more denoising steps, the LPIPS score first decreases and then increases. Therefore, incorporating PCs in a moderate amount of steps gives the best performance (around 20% in this case). In terms of computational efficiency, the additional computation cost is around 10% when we engage the PC for the first 20% of the denoising steps and around 25% when the PC is used in the first 50% of the denoising steps.

One explanation for the diminishing performance gain when using PCs in too many denoising steps is that the later stages of denoising are mainly responsible for refining the details in the image. And the guidance of PCs could be less effective in this aspect. This could be the consequence of using latent space PC (latent space generated by VQ-GAN). In the future, we will explore possibilities to train good PCs directly in the pixel space. This could lead to better performance in terms of refining details in generated images.

| % denoising steps using PC |     5     |    10    |     20    |    30    |    40    |    50    |

|                     LPIPS                     | 0.470 | 0.459 |   0.454 | 0.456 | 0.464   | 0.476 |

|                  Runtime                  |   109   |   111   |   115   |   118   |  121   |   125   |

We thank the reviewer for their insightful comments.

It would be ideal to see more comparisons of such masks in arbitrary shapes

Thank you for the suggestion. We additionally adopted a set of randomly generated “wide” masks following [1] and [2]. The masks are generated by uniformly sampling from polygonal chains dilated by a high random width and rectangles of arbitrary aspect ratios. We downloaded a set of 100 such masks provided by [1] and ran all algorithms with the same set of hyperparameters used in other tasks (no hyperparameter tuning specific to these masks is done). We have added the full results in the updated paper (Table 1) and below is a summary of the new results:

| Dataset & Mask     | Tiramisu | CoPaint | RePaint | DDNM | DDRM |   DPS   | Resampling |

| CelebAHQ & Wide |   0.069     | 0.072    |   0.075   |   0.112   | 0.132   | 0.078 |   0.128         |

| ImageNet & Wide |   0.125     | 0.128    |   0.127   |   0.198   | 0.197   | 0.132 |   0.196         |

| LSUN & Wide         |   0.116     | 0.115    |   0.124   |   0.135   | 0.204   | 0.108 |   0.202         |

Tiramisu achieves the best LPIPS score on CelebA-HQ and ImageNet, and is behind DPS in LSUN-Bedroom. Overall, Tiramisu achieves the best LPIPS score on 14 out of 21 tasks. Since the LPIPS scores alone may not fully justify the performance of Tiramisu, please also refer to the response to the next question (about user study).

a user study would be beneficial in this case as previous works such as [1][2] perform

We conduct a user study based on all three datasets and two masks (“Expand1” and the newly added “Wide” masks adopted from [1,2]). Detailed settings and results are added in the updated paper (Appendix E.1). We copy the result table from the paper:

|      Dataset & Mask      |   Tiramisu   | CoPaint | RePaint | DPS |

| CelebAHQ & Expand1 | Reference |      22      |      34      |  14   |

| CelebAHQ & Wide       | Reference |      26      |      30      |   22   |

| ImageNet & Expand1 | Reference |      32      |      20      |   24   |

| ImageNet & Wide       | Reference |      14      |       6       |   12   |

| LSUN & Expand1         | Reference |      18      |       2       |   8   |

| LSUN & Wide                | Reference |        4        |       6       |   -6   |

We use the vote difference (%) metric, which is the percentage of votes to Tiramisu subtracted by that of the baseline. A positive vote difference value means images generated by Tiramisu are preferred compared to the baselines, while a negative value suggests that the baseline is better than Tiramisu.

We compare against the three strongest baselines (CoPaint, RePaint, and DPS) as their average LPIPS scores are not significantly worse than Tiramisu. As shown in the table, in all but one task the vote difference score is positive (and often quite large), which indicates the superiority of Tiramisu compared to the baselines.