High-Resolution Image Harmonization with Adaptive-Interval Color Transformation

Thanks for your positive evaluation and valuable suggestions.

W1 & Q1: Limited novelty of using two 3D LUTs for color transformation, which is a combination of ideas from existing LUT-based image enhancement methods [47, 48] but lacks necessary discussions and experimental verifications..
R1 & A1: Regarding the novelty of using two 3D LUTs for color transformation, we request the reviewer to refer to our response to Q2 & Q3 of Reviewer pwQP due to the space limitation.

Our method is not a combination of Adaint [47] and Seplut [48]. In fact, simply combining them faces the challenges of limited color transformation capability and overlooking diverse local contexts. Our AICT addresses these challenges by dynamically predicting entire LUTs for pixel-wise sampling interval adjustment and color transformations. We also conduct experimental comparison between our method and Adaint and Seplut as well as their combination. As shown in the table below, our AICT achieves the best performance in terms of all metrics.

W2: Explain “the non-linearities” and show the visual results.
R2: Due to resolution differences, predicted 3D LUTs cannot align with the original input image. Consequently, linear interpolation based on a few sampling points is performed within the 3D LUT to obtain output color values, leading to "non-linear" transformations. To illustrate this advantage, we present the color distribution in local areas of high-resolution harmonized and ground truth images in Figure 3 of the PDF attached in the author rebuttal, which shows that the pixel values predicted by our method are closer to the real color distribution.

W3: “lightweight and computationally efficient” needs to be tuned down.
R3: Thanks for this suggestion. We will tune down our claim in the revised manuscript.

W4 & Q4: Explain novelty of the 'global consistent weight method'.
R4 & A4: We request the reviewer to refer to our response to Q4 & Q5 & Q6 of the Reviewer pwQP due to space limitation.

W5: There are a few weak points regarding the paper writing.
R5: Thanks for this suggestion. We will improve the paper writing in the revised manuscript.

W6: The visual results do not show significant improvements.
R6: To better show significant improvements, we quantify and visualize the error between the harmonized and ground truth images as shown in Figure 1 of the PDF attached in the author rebuttal. We also present additional visual results in Figure 2 of the PDF.

Q2: Report and explain the results of incorporating more 3D LUTs.
A2: We report more results in the table below. For each color channel, we use only 1 LUT for color transformation ('Single') and cascade 2 and 3 LUTs for adaptive interval learning ('Int x 2', 'Int x 3') and color transformation ('Tra x 2', 'Tra x 3'). We also cascade 4 and 6 LUTs for alternating adaptive interval learning and color transformation ('Alt x 2', 'Alt x 3'). Performance improves slightly with more LUTs but plateaus at three, due to increased training difficulty.

Q3: Explain why foreground-normalized MSE loss [34] is more important
A3: The foreground-normalized MSE loss, widely used in methods like PCT-Net and DCCF, prevents instability in images with very small objects. For ablation studies, we set Amin to 100, 5000, and 10000, denoted as 'Amin=100', 'Amin=5000', and 'Amin=10000'. In our AICT, Amin is set to 1000. As shown in the table, when Amin is 100, performance doesn't improve due to the scarcity of smaller targets. Increasing Amin from 1000 to 10000 causes instability in training on small targets, resulting in decreased performance.

Q5: Report non-reference image quality metrics (e.g., NIQE and LPIPS) for Figure 9.
A5: Since LPIPS is a full-reference metric, we only report NIQE in the table below. Our method obtains the best results in the first two examples and the second-best result in the last example.

Q6: Provide intermediate visual results of the ablated models
A6: The intermediate visual results are shown in Figure 4 of the PDF attached in the author rebuttal.

Q7: Explain why the channel-crossing strategy fails to improve performance
A7: Our parameter model uses an RGB image to predict parameter maps, which inherently include information from all color channels. The channel-crossing strategy introduces redundancy without improving performance.

Q8: Use SSIM as the evaluation metric
A8: SSIM is actually not a commonly used metric for image harmonization tasks. Following this suggestion, we have used SSIM as an evaluation metric. As shown in the table, our AICT achieves a competitive SSIM score of 0.9910.

Thanks for your positive evaluation and valuable suggestions.

Q1: The authors should conduct experiments using the training/test set of HAdobe5k which focuses on high-resolution, and report the results under different resolutions (1024, 2048, etc). More baselines should be compared in this setting.
A1: Thanks for this suggestion. We have reported the results under the resolutions 1024x1024 and 2048x2048 using the training/test set of HAdobe5k in Tables 2 and 7 of the submitted manuscript. Following this suggestion, we present comparison with two additional baselines: AdaInt [1] and SepLUT [2]. As shown in the tables below, our method outperforms AdaInt and SepLUT in terms of all metrics.

[1] Yang, Canqian, et al. AdaInt: Learning adaptive intervals for 3D lookup tables on real-time image enhancement. CVPR 2022.

[2] Yang, Canqian, et al. Seplut: Separable image-adaptive lookup tables for real-time image enhancement. ECCV 2022.

Q2: The authors should test on high-resolution real composite images and conduct user study.
A2: Thanks for this suggestion. We test on a high-resolution real composite image dataset [3] with resolutions ranging from 1024 × 1024 to 6016 × 4000. We randomly select 20 images for the user study. In the study, 20 volunteers independently rank the predictions from 1 to 3 based on visual quality, considering color and luminance consistency. Scores of 3, 2, and 1 are assigned for ranks 1, 2, and 3, respectively. The mean scores for each method are presented in the table below. As we can see that our method achieves the highest score.

[3] Li Niu, et al. Deep image harmonization with globally guided feature transformation and relation distillation. ICCV 2023.

Q3: The authors are suggested to evaluate the method on ccHarmony dataset, which can reflect the illumination variation more faithfully.
A3: Thanks for this suggestion. Actually, we have evaluated the method on the ccHarmony dataset and presented the results in Table 3 of the submitted manuscript. Our method achieves the best results in terms of MSE, PSNR, and SSIM metrics, which demonstrates the effectiveness of the proposed method when handling illumination variation.

Q4: The authors should discuss the limitation of the proposed method and show some failure cases.
A4: Thanks for this suggestion. We have discussed the limitations of the proposed method and shown some failure cases in Section A.4 of the supplementary materials of the submitted paper.

Q5: More efficiency analyses (FLOPs, memory, time) should be provided.
A5: Thanks for this suggestion. In the submitted manuscript, we have provided the FLOPs, memory cost, and inference time on the HAdobe5k dataset for different resolutions (1024 x 1024 and 2048 x 2048). Please refer to Tables 8 and 9 of the submitted manuscript.

Thanks for your positive evaluation and valuable suggestions.

Q1:The authors abuse notation regarding 3DLUTs in color image processing.
A1: We thank the reviewer very much for pointing out the inaccurate usage of the term 3DLUTs. Following the reviewer's suggestion, we will rename the term of 3DLUTs to define our approach and rewrite the corresponding parts of the paper.

Q2: Metrics are insufficient. The authors need to add further metrics such as LPIPS, and DeltaE as perceptually-based full-referenced metrics, and NIQE as a non-reference one.
A2: Thanks for this suggestion. Following this suggestion, we use LPIPS, DeltaE, and NIQE metrics to evaluate the performance of the compared methods on the iHarmony4 dataset. The table below shows the quantitative results. As we can see that our method achieves the best results in terms of LPIPS and DeltaE and the third best results in terms of NIQE. It should be noted that NIQE is a no-reference metric that does not assess color or semantic context continuity. Therefore, it is not suitable for evaluating the performance of image harmonization.

Q3: The paper will be of much larger interest if the authors present the results of a user study, for example as it was done in PCT-Net (reference 12 in this paper) or in [2].
A3: Thanks for this suggestion. We randomly select 20 high-resolution real composite images from the GiftNet dataset. In the study, 20 volunteers independently rank the predictions from 1 to 3 based on visual quality, considering color and luminance consistency. Scores of 3, 2, and 1 are assigned for ranks 1, 2, and 3, respectively. The mean scores for each method are presented in the table below. As we can see our method achieves the highest score.

Q4: Figures 1,3, and 4 are versions of the same general idea of Figure. It will be beneficial for the paper if the authors can present just 1 Figure in which all the small differences are included.
A4: Thanks for this suggestion. We will combine Figures 1, 3, and 4 into a single comprehensive figure that highlights all the small differences in the camera-ready version.

Q5: Also, when reading this paper, I got the feeling that: "This idea is mostly AdaInt + Mask". I may have lost some important component here, so please guide me through why my feeling may not be correct.
A5: We are sorry that our presentation let the reviewer misunderstand that our idea is mostly AdaInt + Mask. In fact, our method is significantly distinct from AdaInt in three aspects. Firstly, AdaInt predicts only a few weight parameters to fuse several fixed LUTs obtained through training, which limits its expressive capacity. In contrast, our method dynamically predicts entire LUTs based on the input, resulting in significantly greater expressive power. Secondly, AdaInt uses LUTs to perform global RGB-to-RGB transformations, which overlook diverse local contexts. In contrast, our method uses the color and position of each pixel as inputs for the LUTs to perform pixel-wise color transformations. Thirdly, AdaInt applies adaptive interval learning to each lattice dimension of the 3D LUTs for global sampling interval adjustment, which limits the expressiveness of the 3D LUTs. In contrast, our method achieves pixel-wise sampling interval adjustment to model local non-linearities of the color transformation.

Thanks for your positive evaluation and valuable suggestions.

Q1: The effectiveness of AICT for ultra high-resolution images is unclear and should be validated with appropriate benchmarks.
A1: Thanks for this suggestion. In Table 1 of the paper, we have reported the results on the HAdobe5k dataset with resolutions ranging from 312 × 230 to 6048 × 4032. Following the reviewer's suggestion, we collect two ultra high-resolution benchmarks (over4K and over5K) by selecting images with resolutions exceeding 4K (4096 × 2160) and 5K (5120 × 2880) from the HAdobe5k dataset. The compared results on the table below show that our method achieve the best performance on the over4K benchmark in terms of all metrics. On the over5K benchmark, our method achieves the best performance in terms of MSE and PSNR and the second-best performance in terms of fMSE and SSIM.

Q2 & Q3: Explain how AICT's specific LUT-based approach is distinct from prior methods.
A2 & A3: Compared with prior methods, our method has three distinct differences. Firstly, unlike previous methods that use fixed weight parameters to fuse pre-trained LUTs, our method dynamically predicts entire LUTs based on the input. Secondly, Previous methods use LUTs for global RGB-to-RGB transformations, which often overlook local context. In contrast, our method uses each pixel’s color and position as inputs for LUTs, enabling pixel-wise color transformations. Thirdly, previous methods apply adaptive interval learning and separable lookup tables to each lattice dimension of the 3D LUTs for global sampling interval adjustment, which limits the expressiveness of the 3D LUTs. In contrast, our method achieves pixel-wise sampling interval adjustment to model local non-linearities of the color transformation.

Q4 & Q5 & Q6: More discussion is needed on how AICT's weights differ and what additional capabilities they enable.
A4 & A5 & A6: Thanks for this suggestion. When a foreground is inserted into a new background, its average brightness is influenced by the background lighting. Predicting spatially consistent weights for global color adjustment can enhance adaptive interval learning. To further discuss AICT's weights, we conduct ablation studies by sequentially removing the parameters: first for the R channel ('w/o R'), then for both R and G channels ('w/o RG'), and finally all parameters ('w/o weight'). We also explore pixel-level weights ('spatial'). As shown in the first table below, increasing the number of removed parameters decreases performance, highlighting the role of weight learning module. The results of "spatial" show that learning spatially-varying weights does not enhance performance, as LUTs handle pixel-wise color transformations. To illustrate the role of the weight learning module in global color adjustment, we calculate the error between the average color values of harmonized and ground truth images. As shown in the second table below, our method performs better.

Q7: Cascade LUTs need ablation studies
A7: Following this suggestion, we report more results in the table below. For each color channel, we use only 1 LUT for color transformation ('Single') and cascade 2 and 3 LUTs for adaptive interval learning ('Int x 2', 'Int x 3') and color transformation ('Tra x 2', 'Tra x 3'). We also cascade 4 and 6 LUTs for alternating adaptive interval learning and color transformation ('Alt x 2', 'Alt x 3'). Performance improves slightly with more LUTs but plateaus at three, due to increased training difficulty.

Q8: The user study results are not reported.
A8: Thanks for this suggestion. We randomly select 20 high-resolution real composite images from the GiftNet dataset. In the user study, 20 volunteers independently rank the predictions from 1 to 3 based on visual quality. Scores of 3, 2, and 1 are assigned for ranks 1, 2, and 3, respectively. The mean scores for each method are presented in the table below. As we can see our method achieves the highest score.

Q9: More analysis on what types of images/scenarios benefit most from AICT.
A9: We present more results in Figure 2 in the PDF attached in the author rebuttal. As we can see that our AICT method particularly benefits images with rich textures, such as fur and hair. This is because AICT achieves non-uniform sampling intervals, allowing for better modeling of local non-linearities in color transformations.