LIME-Eval: Rethinking Low-light Image Enhancement Evaluation via Object Detection

Q6. Evaluation of more tasks

Our method does have the potential for more restoration/enhancement tasks. In fact, we synthesized 8 types of degradation. (Figure 5 exhibits an example of this), which include noise, blurriness, color bias, and exposure shift. These degradations damage the performance of detection and as a result, present a larger energy value. To preliminarily see the potential for more restoration/enhancement tasks, due to the short period of rebuttal, we here exhibit cases from dehazing and real-world SR in Fig.10. From the figure, one can find that our method can distinguish clearer results from degraded ones. We leave a detailed investigation to more restoration/enhancement tasks for future work. We added an analysis of this in the appendix and revised the conclusion/future work part as well.

We hope our responses can address your concerns. Please don't hesitate to respond if you have further concerns.

References: [1] W. Grathwohl et al., Your classifier is secretly an energy-based model and you should treat it like one. In ICLR 2020 [2] R. Peng et al., Energy-based Automated Model Evaluation, In ICLR 2024

We deeply appreciate the time and effort you invested in reviewing our manuscript and your positive feedback on our work. Your comments are invaluable, and we have thoroughly addressed them to improve the quality of our paper. Below, we present our detailed responses to your concerns.

Q1. Quantization.

The quantization process described in Sec. 3 involves converting the output of an enhancer into unsigned integers within the range of 0-255. For the "Enhancement first" scheme, we pre-enhanced all outputs and saved the enhanced images before training. As a result, the value of inputs to enhancers is discretized into 256 distinct choices. In contrast, the "Augmentation First" scheme uses the intermediate output of enhancers, which remains continuous. To ensure this difference has minimal impact on the results, we conducted an experiment referred to as "Bread-round". In the rebuttal revision, we have provided a clearer explanation of this process.

Q2. Rationale of energy modeling, and more details on background prediction and object prediction.

Previous studies [1,2] have demonstrated that classifiers can be interpreted as Energy-Based Models (EBMs), exhibiting an intuitive property: correctly classified samples are associated with lower energy values, whereas misclassified ones are assigned higher energies. Leveraging this insight, we employ energy-based modeling to intimate the accuracy of object detection systems.

In an object detector, we observe that the correctly classified areas are sparse, only with a minimum difference among clear and damaged images, but the logits in those less confident areas are more sensitive to degradations. Modern object detectors, particularly one-stage models (and to some extent, two-stage detectors), typically feature output layers composed of dual classification heads: one for object classification and the other for background classification (there also exists a localization head; however, it addresses location regression rather than classification). Within the detection pipeline, the model determines whether a data point corresponds to a valid object using the background classification score, which ranges from [0, 1]. A higher score indicates greater confidence in treating the data point as part of the foreground. Subsequently, the class with the highest probability is selected as the final classification output.

To address regions characterized by lower confidence in the background score, we propose a reweighting strategy that amplifies the influence of the classification score in these areas. Specifically, we apply the formulation $(1 - x_{bg}) \cdot x_{cls}$, where $x_{bg}$ represents the background score and $x_{cls}$ denotes the classification score. As this adjustment reduces the overall score magnitude, we introduce a square-root transformation to balance the scaling, yielding a refined confidence measure expressed as $\sqrt{(1 - x_{bg}) \cdot x_{cls}}$. We have refined the manuscript for a clearer presentation of our method.

Q3. Additional algorithm than IQA for a more convincing conclusion in Chapter 5.2.

Thanks for your advice. We reorganize the experiments and bring more non-reference metrics for evaluation in Table 2. In the revised version, we involve 5 image quality assessment methods (BRISQUE, NIQE, LIQE, MUSIQ, and ClipIQA), an image aesthetic assessment method NIMA, a color assessment method DeT, and a lightness assessment method LOE to make a more convincing conclusion.

Q4. Report experiments on more detectors/evaluation metrics in Chapter 5.3. Thanks for your precious advice. We reorganized the experiments in Table 3 and adopted another traditional detector YOLOv8, and an open-vocabulary detector YOLO-World for a direct evaluation.

Q5. References to certain terms and methods.

Thanks for your suggestion. In the rebuttal version, we have done thorough proofreading and added related references to terms/methods.

Thanks for the time and effort spent on our work.

There seems to be some misunderstanding, thus we would like to clarify the contribution of our work in this response.

Our work begins with a preliminary experiment to investigate which evaluation scheme fits best for human perceptual. The result shows that directly evaluating enhanced low-light images with normal-light pre-trained enhancers gives a better understanding of human perceptual, rather than training on the enhanced images.

To confirm the advantage of this method, we add more low-light enhancement methods, launch the online benchmark platform, LIME-bench, to collect pairwise user preference, and most importantly, come up with the conclusion that the direct evaluation scheme does have a strong correlation to human preference. We also discuss the factor that forms discrepancies between machine and human preference.

Finally, we proposed a label-free and reference-free metric for evaluating low-light enhancers, namely LIME-eval. Since it requires no detection labels, it can be used on natural images, not limited to annotated detection datasets.

We highlighted the contribution of our work to avoid possible confusion, on the pipeline and technical contribution of this manuscript in the rebuttal revision.

We respond to specific concerns as follows:

Q1. Augmentation for Qualitative comparisons on ExDark

We agree with the opinion that augmented images are not legitimate inputs for qualitative comparison. However, we do NOT use augmentation on any qualitative samples of ExDark, the input is fed as is in all of the qualitative samples. By posing these images, we would like to give a qualitative view of how this enhancer performs in real-world samples.

Q2. How is LIME-Eval integrated into Retinexformer?

As a loss term. Since it requires no object/ground-truth annotation, we just train Retinexformer with their released codebase with exactly the same data they used, but add LIME-eval as an additional loss term.

Q3: Concerns for training with target detector.

Please note that we are not discussing retraining enhancers, but object detectors in Table 1. Since the retrained Retinexformer has never been back-propagated with detection labels or images from the detection dataset, it meets no contradiction with what we have stated in Section 3.

We hope our responses can address your concerns. Please don't hesitate to respond if you have further concerns.

We would like to express our gratitude for the time and effort spent on reviewing our manuscript and for your appreciation of our work. Your comments are valuable for us, and we have accordingly revised the manuscript in Sec. 4. of the rebuttal revision. Please find the response to the concern below.

Q: Explain background prediction.

For modern object detectors, particularly one-stage detectors (and to some extent, two-stage detectors), feature output layers are typically composed of dual classification heads: one for object classification and the other for background classification (there also exists a localization head; however, it addresses location regression rather than classification). Within the detection pipeline, the model determines whether a data point corresponds to a valid object using the background classification score, which ranges from [0, 1]. A higher score indicates greater confidence in treating the data point as part of an object. Subsequently, the class with the highest logit is selected as the final classification output.

A neat property of energy-based methods is that correctly classified samples are associated with lower energy values, whereas misclassified ones are of higher energies. In an object detector, we observe that the correctly classified areas are sparse, only with a minimum difference among clear and damaged images, but the logits in those less confident areas are more sensitive to degradations. To address regions characterized by lower confidence in the background score, we propose a re-weighting strategy that amplifies the influence of the classification score in these areas. Specifically, we apply the formulation $(1 - x_{bg}) \cdot x_{cls}$, where $x_{bg}$ represents the background score and $x_{cls}$ denotes the classification score. As this adjustment reduces the overall score magnitude, we introduce a square-root transformation to balance the scaling, yielding a refined confidence measure expressed as $\sqrt{(1 - x_{bg}) \cdot x_{cls}}$.

We hope our responses can address your concerns. Please don't hesitate to respond if you have further concerns.

We sincerely appreciate the time and effort you dedicated to reviewing our manuscript. We are particularly delighted by the insightful nature of your comments, which have greatly inspired us to refine and improve our work in the rebuttal revision. Please find the response to the concern below.

Q1: What is the training recipe for the detectors? Does the color jitter and normalization apply to the models?

ImageNet normalization is not applied during training. As suggested by the authors of YOLOX, we adopt a simple division of 255.0. Since we adopt pre-trained YOLOX models on the COCO dataset for direct evaluation, we align the training recipe of YOLOX to keep consistency during comparisons. The data augmentation pipeline of YOLOX involves two stages: In the first stage (270 epochs), mosaic, MixUp, color jitter in HSV color space, and geometric augmentations are used, while in the second stage (30 epochs), mosaic is turned off to make a better alignment to the real-world images. To make a clearer view of the training setting of the models in Table 1, We have added further training details of the detectors to Appendix A.2.2 in the rebuttal revision.

To investigate what effect it could bring, we retrained a YOLOX-x detector on the COCO dataset, but without color jitter. This results in a slight performance degradation from 51.1 to 50.8 on the validation split of the COCO dataset. The performance of all detectors, except NeRCO, drops.

We then conducted a correlation study to investigate what it could affect, as in Appendix B.2 of the rebuttal revision. In general, detectors without color jitter adhere better to human preference. The only outlier is IAT, which delivers less visually pleasant, but extremely clear low-light enhancement results that benefit detection. We found that a linear ensemble of the mAP from two detectors (0.25 * w/o color jitter + 0.75 * color jitter) yields an even better consensus with human perception, raising the correlation from 0.703 to 0.718. This demonstrates that detectors without color jitter adhere better to human preference generally. Ensembling with detectors under various augmentation schemes may further boost the perceptual correlation, which we leaves for future work.

Q2: Evaluation with face detectors.

It is positive that our method can be applied to other detection datasets, such as the Dark Face dataset, as the reviewer mentioned. To demonstrate this, we have pre-trained the model with WIDER FACE and tested the model with enhanced DARK FACE results. Due to the short period of the discussion phase, we exhibit visual results for the dark in Appendix B.3 of the rebuttal revision. Effectively, though, it is limited to face images while facing challenges to evaluate natural images without human faces. To see this, we evaluate our face detector with our LIME-Bench dataset, which contains few human faces. The correlation drops from 0.593 to 0.496, demonstrating its limitation in evaluating images in the wild. Learning on the ExDark dataset leads to an awareness of objections to the model, which can be generalized better in real-world scenarios. In future work, we are planning to evaluate with open-vocabulary detectors, which can detect arbitrary objects. Thanks again for your insightful comments.

Q3: Add related work to Low-light object detection without enhancement.

Thanks for your suggestion. In the revision, we have added a paragraph in the Related Work section to describe representative existing schemes in low-light object detection without enhancement.