A Novel Evaluation Framework for Image Inpainting via Multi-Pass Self-Consistency

Regarding Motivation:

1. The authors argue that the primary flaw in current evaluation methods (MSE, PSNR, SSIM, LPIPS, FID) is their reliance on unmasked images. This assertion lacks rigor. For instance, metrics like MSE and PSNR do not inherently depend on unmasked pixels, especially considering the common post-inpainting operation: $\hat{\mathbf{X}} = \hat{\mathbf{X}} * (1 - \mathbf{M}) + \mathbf{X} \odot \mathbf{M}$.
2. The assertion in Section 1, Para. 2, that unmasked images might not always be accessible in practical scenarios. It seems unlikely to happen under current inpainting settings. Could the authors provide several examples to support this claim?
3. The definition of "potential bias," as mentioned in Sec. 1, Para. 3, remains unclear. Even with the aid of Figure 1, the nature of this bias is not immediately evident.

Regarding Results:

1. As per Table 4, the values derived from the proposed metrics are strikingly similar, particularly for smaller mask ratios. This is in contrast to human scores, which exhibit significant variance.

1. The motivation is to achieve a high level of consistency through a profound understanding of the generated content. However, the design of patch masks is to focus on the restoration of the low-level details, which often rely on the local similarity priors, rather than the high-level understanding. Moreover, the performance of the second inpainted images depends on the second inpainting network, which highly affects the evaluation.
2. The validation in Table 1 is unreasonable. First, original-first and original-second have the same task, which is to recover the content similar to the original image. Thus, their performance of different methods is consistent, but not the persistence of biases in the metric. Second, for the difference in First-Second, it may be due to the different inpainting mechanisms of CNN-based methods and Diffusion-based methods. It cannot show the superiority of the proposed metric. Third, it is unclear the meanings of 1-to-2, 2-to-0, 2-to-0.
3. Overall, the contributions are limited, and more reasonable motivation and theoretical and experimental validations are needed.

• 1. The presentation in Sec. 4 does not highlight the contribution of the paper. See Q1 and the Suggestion.
• 2. The robustness of the evaluation framework is not sound. See Q2, and Q3.

• The analysis about why the proposed framework works for evaluating the image inpainting algorithms is confusing. It seems that the framework tries to use the first inpainting method to recover the masked regions from the unmasked regions of the original image with a normal mask, and then employ another random inpainting method to recover the patch-masked regions from the unmasked regions of the first inpainted image with multiple patch masks that are different from the first normal mask. But why it works for better evaluating the first inpainting method is not clear.
• The evaluation on the proposed benchmark is not convincing. The experiments are insufficient to validate the efficacy of the proposed metric, since not only the dataset but also the method are not comprehensive enough. And also some details are not clear, for example, what is the second inpainting method for these experiments.
• The ablation study is not sufficient. There are many components that may influence the performance but are not analyzed, for example, what about to choose different second inpainting method, what about to choose different sub-metric, what about to choose different dataset, what about to choose different mask type, etc.
• It seems that the Perceptual Metric in Figure 2 should connect the Second Inpainted Images with the Inpainted Image but not the Images Masked by Second Masks.