Prune and Repaint: Content-Aware Image Retargeting for any Ratio

• Weakness 1: Some detailed concepts are missed, such as $x_0$ in line 142.
  Response: Thanks for pointing it out. The center coordinate $x_0$ represents the center of the entire image. The height coordinate $H_{x_0}$ can be calculated as the average of the heights $H_i$ of all the salient pixels, where $i$ ranges from 0 to $n$, the total number of salient pixels:
  $H_{x_0} = \frac{\sum_{i=0}^{n} H_i}{n}.$ The width coordinate can be determined in a similar manner.

• Weakness 2: Lack of comparison with recent image edit/generation methods.
  Response: Thanks for your suggestion. The results compared with full-image repainting (FR) and InGAN are included in Tab. R1, Tab. R2, Fig. R1 and Fig. R2 in the PDF. Both qualitative and quantitative results demonstrate that our method significantly outperforms existing approaches by our proposed CSC to preserve the key information and AR to enhance the local smoothness.

• Weakness 3: Lack of explanations with corresponding visual samples to show the complementarity among the four different evaluation metrics in Table 2.
  Response: An example illustrating different degrees of content loss, deformation, distortion and aesthetics is presented in Fig. R3 of the PDF. Although the overall deformations in the third and fourth images are quite noticeable, the third image is very smooth locally, so it belongs to deformation rather than distortion, whereas the fourth image has both deformation and distortion. Further examples and scoring criteria will be included in the appendix.

• Weakness 4: The ARRD module in Figure 2 is not very clear.
  Response: Thanks for pointing out this issue, expanding the mask map is an optional step that will only be implemented when the retargeted image cannot accommodate salient objects. We will revise the flowchart with a dashed box and a legend to avoid this confusion.

• Question 1: Why AR restore the depth and proportion of the original image better than BR.
  Response: The expansion operation of AR reduces the seams that should have been removed from the foreground, thereby minimizing pixel displacement and alleviating misalignment of foreground objects. This allows for better preservation of the relative positions and depths of foreground objects.

• Question 2: Performance on the video.
  Response: An example of video retargeting is presented in Fig. R4 in the PDF. As expected, the lack of inter-frame consistency in the PruneRepaint approach leads to inconsistencies in generating objects in the background area.

• Question 3: The detailed difference between BR and AR in line 266.
  Response: To achieve background harmony and foreground preservation, BR identifies and repaints the background areas based on the saliency map, which does not affect the foreground and therefore fails to repair discontinuities in the foreground pixels. In contrast, AR is designed to achieve harmony across the entire image, which adaptively identifies areas where CSC removed more seams, thereby mitigating discontinuities in both the foreground and background.

• Question 4: Explain why the performance for different target ratios varies a lot.
  Response: Typically, the larger the aspect ratio difference between the retargeted image and the original image, the greater the loss in image saliency. We analyzed the RetargetMe dataset and found that the average aspect ratio is 0.7405, which is closest to 9/16 (0.5625), indicating minimal saliency loss at this ratio. As the aspect ratio deviates further from the original ratio, the Saliency Discard Ratio (SDR) increases.

• Question 5: How to decide threshold $\eta$ in line 172.
  Response: The threshold $\eta$ is the mean value of the entire saliency map.

• Weakness 1: Inference speed to be improved.
  Response: Thanks for your suggestion, we will further employ accelerated diffusion models to improve the inference speed.

• Weakness 2: Results on ratio 1:1 and 4:3 are not visualized.
  Response: Thanks for pointing out this issue, we present visualization results for only two extreme ratios to more clearly demonstrate the advantages of PrueRepaint as other ratios hold less difficulties. Limited by the page size, visualized results for the other two aspect ratios will further be included in the appendix.

• Weakness 3: Minor writing or grammar mistakes.
  Response: Thanks for your detailed review, we will carefully proof the manuscript.

• Weakness 4: About the text prompts.
  Response: Here we regard Image Retargeting as an image-to-image generation task and do not utilize additional text prompts to assist in the task. The positive text prompts used in the experiment are the default 'best quality, high quality,' while the negative prompts are 'monochrome, lowres, bad anatomy, worst quality, low quality.' The weight of the image prompt is set to 1, meaning that the text prompts contribute minimally.

• Question 1: The meaning of $W_s$ in equation (5).
  Response: $W_s$ is different from the image width $W$. $W_s$ denotes the saliency width which is defined in Eq. (4). We will add the description on $W_s$ in Eq (2).

• Question 2: How to implement $W_s$ in metric SDR? Is it to sum the width of all lines in the saliency map and output the maximum one as the width of the saliency map? Why do not use the metrics in salient object detection such as F-measure or use IOU to evaluate?
  Response:

  • For the first and second questions, I'm afraid there might be a slight misunderstanding. $W_s$ sums the width of all lines in the saliency map and takes their union rather than the maximum. In the implementation, we use a list of zeros with length $W$ to store whether each column has salient pixels. If a column has a salient pixel, we set the corresponding entry to 1. Finally, summing the list gives the salient width.
  • For the last question, both saliency detection metrics and IOU are only suitable for evaluating two images with completely identical resolutions. However, image retargeting changes the aspect ratio as well as resolutions of the original image. Therefore, we proposed a simple metric that provides a rough estimate of saliency preservation. Thank you for your inspiring question, we will explore more retargeting metrics in our future research.

• Question 3: The weights of different subjective metrics.
  Response: We agree with you that different people and tasks have different biases towards subjective metrics, thus specific weights can be tailored to specific needs. We take averages just to compare the overall performance in a more intuitive and concise way.

• Weakness 1: Lack of comparison with more retargeting methods.
  Response: Thanks for your advice. We have added experiments on InGAN as well as full-image repainting (FR) in Fig. R1, R2 and Tab. R1 in the PDF. Both the quantitative and qualitative comparisons show the large superiority of our method.

• Weakness 2: Suggestions to solve the distorted regions with local correlations.
  Response: Yes, thanks for your constructive suggestion. Although our method has achieved significant improvement on both preserving semantic completeness and avoiding local artifacts, it is indeed a promising solution to further improve local smoothness by strengthening local correlations. However, the spatial misalignment between the original and retargeted images makes this local consistency constraint difficult to be implemented. We will study this problem carefully in our future work.

• Weakness 3: Lack of a larger high-quality dataset.
  Response: Thanks for your advice, we are actually working on the retargeting dataset. We collect images with a wider variety of objects, input ratios, foreground scales, and object layouts for comprehensive evaluation.

• Question 1: Explain the limitation of ARRD.
  Response: The reason is that ARRD searches the local pixel displacement area without global understanding. For example, in comparing the 'Original image' and '+CSC+AR' images in the second row of Figure 6, some seams passing through the streetlight were removed, causing misalignment. Ideally, the entire streetlight should be repainted, but AR only repaints the pixels near the deleted seams in the middle, resulting in a streetlight that remains misaligned in the generated image.

• Question 2: Explain why the results of '+CSC' in Table 3 and ‘background repainting’ in Table 4 are the same.
  Response: The BR and AR are all implemented on the base of CSC. As BR identifies the background based on the saliency map for repainting, it will not change the saliency regions, thus they hold the same score.

• Question 3: About text prompts.
  Response:

  • For the first question, we regard Image Retargeting as an image-to-image generation task. Moreover, designing specific prompts for each image is quite expensive thus we do not utilize additional text prompts to assist in the task. The positive text prompts used in the experiment are the default 'best quality, high quality,' while the negative prompts are 'monochrome, lowres, bad anatomy, worst quality, low quality.' The weight of the image prompt is set to 1, meaning that the text prompts contribute minimally.
  • For the second question, text prompts are supposed to be captions that describe an image. However, prompts such as 'keep key semantics, preserve the main structure' are typically not captions of images. As a result, they are meaningless in guiding the generation process.

• Question 4: The reasons for selecting the reported ratios.
  Response: Most existing devices have aspect ratios ranging from 16:9 to 9:16. The models that handle extreme ratios tend to perform better on the more common ratios as well. Therefore, we only select 4:3 and 1:1 as the middle-ground aspect ratios to test. Among them, 16:9 and 4:3 are common aspect ratios for computers and televisions, 1:1 is a preferred image and video size for social media (such as Instagram), and 9:16 is commonly seen on smartphone screens.

Thank you for your detailed rebuttal and for thoroughly addressing all of my concerns. I appreciate the additional experiments and explanations you provided, particularly regarding the comparisons with retargeting methods, the handling of local correlations, and the dataset enhancement.

Given the improvements and clarifications, I will maintain my initial positive rating.

• Weakness 1: No novelty in the method.
  Response:

  1. Seam-carving is a semantic-agnostic approach that often results in severe foreground loss and distortion (see Fig. 4, 5 and 7). In contrast, our proposed content-aware seam-carving (CSC) incorporates semantic awareness to preserve key objects, leveraging visual saliency cues and a careful integration strategy (see section 3.2).
  2. Additionally, to address the inconsistent style and structure issues in previous global generation methods, we introduce adaptive repainting (AR). AR adaptively identifies discordant artifacts with a masking strategy and adjusts the mask to accommodate any aspect ratio.
  3. The qualitative and quantitative comparisons in Sections 4.3 and 4.4 demonstrate the significant superiority of our method over previous approaches, highlighting the advantages of our designs.

• Weakness 2: One page of the paper is direct equations from Seam Carving.
  Response: We respectfully disagree that CSC is merely a rehash of seam-carving (SC). SC determines the importance of different structures in an image using low-level gradient information, which often leads to key foreground loss or deformation. To address SC's lack of semantic awareness, we introduce high-level saliency priors alongside spatial priors (see lines 138 to 148), as presented in Eq. (2). Additionally, a tolerable saliency loss ratio $\lambda$ (see lines 154 to 161) is set to avoid excessive loss of the foreground in extreme ratios and to accommodate potential image expansion operations in AR. Our CSC successfully preserves key information. As demonstrated by the results in Tab. 3 and Figs. 6 and 7, our proposed CSC method achieves significant improvements over traditional SC.

• Weakness 3: The application is quite dated and has not made any significant through the methods proposed.
  Response:

  1. Regarding applications, the reviewer may refer to popular tools like Adobe Photoshop's "Content-Aware Scaling" and Instagram's "Auto Crop", which improve the work efficiency of media professionals and enhance the entertainment experience of the general public.
  2. Our method significantly advances the two key aspects of retargeting - semantic completeness and local smoothness. This greatly enhances the user experience when deploying retargeting in real-world applications.

• Weakness 4: The writing is hazy, and there is barely any motivation about the approach and the application.
  Response: Our motivation is to address two challenging and key issues in image retargeting tasks: preserving the main information and avoiding artifacts, as clearly stated in line 25 of the manuscript. CSC and AR are proposed to solve them respectively.