SAIR: LEARNING SEMANTIC-AWARE IMPLICIT REPRESENTATION

Thanks for your review. We have modified and updated our manuscript based on your suggestions.

Q1. real-life application cases can be shown, e.g. recovering objects that are occluded or blurred via dramatic camera motions.

A1. Our method can be used in many real-life applications, such as object removal. To illustrate its effectiveness, we present several examples in the newly added section 'Real-life Applications'.

Q2. Computation cost, compare the inferring and training cost with previous methods. (as it seems the two-module framework may be costly.)

A2. Our model takes 0.043s to process one image. The one-module framework LIIF's inference time is 0.027s. The incurred increase in time cost remains within an acceptable range.

Q3. Compare the proposed method that leverages coordinate-based implicit representation over diffusion-model ones for image inpainting.

A3. For image inpainting, our proposed method outperforms diffusion-based methods in three aspects. First, inference speed. The inference speed of our method (0.043s/image) is faster than that of the diffusion-based method, such as stable diffusion [1] (12s/image). Second, arbitrary resolution. Our method can generate images at arbitrary resolutions during inference, a capability lacking in diffusion-based models. Third, high fidelity. Diffusion-based models prioritize naturalness over fidelity, resulting in generated results that deviate significantly from the ground truth. We compare our method with stable diffusion [1] on the CelebAHQ dataset under the same setting, the results are PSNR$\uparrow$ 37.97 (ours) vs. 37.60 (stable diffusion) and L1$\downarrow$ 0.01 (ours) vs. 0.032 (stable diffusion).

[1] Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models. (CVPR 2022)

Q4. Can the proposed method apply to any in-the-wild images? Show some samples.

A4. Our method can be used to process the in-the-wild images. The results are shown in section 'Real-life applications'.

Thanks for your review. We have modified and updated our manuscript based on your suggestions, like citing all mentioned papers.

Q1. The novelty of the approach is limited since the improvement mainly comes from the rich representations of the clip embeddings which has been used in other tasks.

A1: Our contribution extends beyond the utilization of CLIP embeddings. In the image inpainting task, the input image is masked and the CLIP embedding is damaged. We address this challenge by introducing SIR, which can reconstruct and resize the damaged and low-resolution embeddings simultaneously. Previous methods only need to resize undamaged embeddings.

Q2. I believe a similar text-based alignment can be made with the implicit representations of LIIF.

A2: After removing SIR module, our method is similar to the baseline LIIF. We broaden the application of implicit neural function to image inpainting tasks, leveraging semantic features for enhanced performance.

Q3. Lack of comparison with recent approaches like NUWA-LIP.

A3: Several recent CLIP-based approaches, such as NUWA-LIP (CVPR 2023), necessitate a 'Guidance Text' to describe the input image for generation. However, datasets like CelebAHQ and ADE20K lack such descriptions. Those methods rely on the image-level semantic descriptions, while we build the continuous pixel-wise semantic representation.

Q4. How does the authors' approach compare against powerful generative models like diffusion model which are excellent at image impainting as well.

Q4. For image inpainting, our proposed method outperforms diffusion-based methods in three aspects. First, inference speed. The inference speed of our method (0.043s/image) is faster than that of the diffusion-based method, such as stable diffusion [1] (12s/image). Second, arbitrary resolution. Our method can generate images at arbitrary resolutions during inference, a capability lacking in diffusion-based models. Third, high fidelity. Diffusion-based models prioritize naturalness over fidelity, resulting in generated results that deviate significantly from the ground truth. We compare our method with stable diffusion [1] on the CelebAHQ dataset under the same setting, the results are PSNR$\uparrow$ 37.97 (ours) vs. 37.60 (stable diffusion) and L1$\downarrow$ 0.01 (ours) vs. 0.032 (stable diffusion).

[1] Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models. (CVPR 2022)

Thanks for your review. We have modified and updated our manuscript based on your suggestions.

Q1. About Table 5, how to evaluate ADE20K mIoU only with AppEncoder alone without SIR.

A1. In Table 5 'with/without SIR', we want to shighlight the reconstruction proficiency of the SIR model in semantic segmentation task. We do not use APPENCODER in this section. The model only using APPENCODER is similar to the baseline LIIF.

Q2. Compare SAIR without CLIP and with other networks other than CLIP.

A2. In the section 'Using other semantic embeddings', we substitute the CLIP encoder with a pre-trained SAM encoder [1]. Tthe text-aligned embeddings produced by CLIP yield superior results.

[1] Kirillov et al. Segment anything.

Q3. Are these semantic labels of CelebA and ADE20K used during training and testing?

A3. We do not use the dataset semantic labels, as the text-aligned embeddings are generated by CLIP encoder directly.

Q4. In Section 5.3, "we used CLIP_T to filter the image feature CLIP_I", the term filter is not very clear.

A4. In this section, we have image pixel features and text features for all dataset semantic labels. We assess the similarity between individual pixel features and semantic label features, and assign a label to each pixel based on the similarity.

Q5. Is there a loss for semantic feature reconstruction? If not, how could SIR reconstruct the semantic features, as stated in Section 5.3.

A5. Yes, we use a loss function to guarantee that SIR does not change the text-aligned feature space. We choose L1 loss to quantify the dissimilarity between the unmasked image's text-aligned feature and the SIR reconstructed feature without changing the resolution.

Q6. Compare to the LIIF on super-resolution tasks by setting up mask ratio 0%.

A6: We test the super-resolution ability of proposed method against LIIF. PSNR/SSIM results under different upsample ratios are shown in the table. Our method still gets promising results in SR task.

Thanks for your review. We have modified and updated our manuscript based on your suggestions.

Q1: These features are then processed by learnable θ, there is no guarantee that the embedding space is text-aligned.

A1.To ensure the preservation of the text-aligned feature space in SIR, we choose L1 loss to quantify the dissimilarity between the unmasked image's text-aligned feature and the SIR reconstructed feature without changing the resolution.

Q2. Why do the authors highlight the text-aligned embeddings? If I understand correctly, the embedding space is just an enhanced pixel-wise semantic feature space.

A2. Our text-aligned embedding seamlessly aligns with semantic label features, other semantic features lack this inherent compatibility. In the 'Using other semantic embeddings' section, we substantiate the superior efficacy of our embedding. And our contribution is not only using the clip representations, we also propose SIR to reconstruct the damaged and low-resolution embeddings for image inpainting task.

Q3. Why mapping the original CLIP feature space by $f_θ$ performs better than the original CLIP feature space?

A3. In Fig. 4, the initial CLIP feature derived from the pre-trained CLIP encoder encounters challenges in effectively handling the masked regions. Our introduced SIR module $f_θ$ proficiently reconstructs the semantic feature, showcasing a notable enhancement in performance.

Q4. Why using CLIP features instead of directly use the GT segmentation maps? (Details about how to implement 'models without SIR block'.)

A4. In the 'Models with/without SIR' comparison, we aim to demonstrate the reconstruction ability of SIR model in semantic segmentation task. The inputs are damaged images without GT masks. In the 'Without SIR' setting, we employ the CLIP encoder to obtain CLIP semantic features for segmentation. Conversely, in the 'With SIR' setting, we leverage SIR to reconstruct the CLIP features prior to segmentation. Evaluation is performed using the GT labels.

Q5. SAIR for other degraded images, like raining or noised images.

A5. We conduct experiments on adversarial defence task following DISCO [1]. To create training pairs, PGD ($\epsilon_\infty$ = 8/255 with step size is 2/255 and the number of steps 100 ) is used to attack a WideResNet28 on Cifar10 dataset. We evaluate both Standard Accuracy (SA) and Robust Accuracy (RA), where SA measures classification accuracy on clean examples, and RA measures accuracy on adversarial examples. The results in terms of $L_\infty$ norm affirm that our model demonstrates robust adversarial defense capabilities.

[1] Ho et al. Disco: Adversarial defense with local implicit functions. (Neurips 2022)

[2] Xu et al. Feature squeezing: Detecting adversarial examples in deep neural networks.

[3] Dziugaite et al. A study of the effect of compression on adversarial images.

[4] Xie et al. Mitigating adversarial effects through randomization. (ICLR 2018)

[5] Sun et al. Adversarial defense by stratified convolutional sparse coding. (CVPR 2019)