Real-Time Scene-Adaptive Tone Mapping for High-Dynamic Range Object Detection

Dear Reviewer zmDQ,

We'd like to thank you for your positive feedback and address the concerns raised in your comments.

Concerns

Q1. Why does training a detector based on HDR inputs lead to unstable gradient propagation, while training an HDR-to-LDR network does not exhibit such issues?

1.The 24-bit HDR (140dB) images exhibit an extreme dynamic range compared to 12-bit LDR images (70dB), with a ratio of approximately 10^7. Therefore, directly feeding HDR images into the network can result in extreme gradient fluctuations, potentially causing the loss function to become NaN.

2.Both HDR-to-LDR models and handcrafted tone mapping algorithms can compress the dynamic range while retaining as many details as possible, helping to stabilize the training process. The difference lies in the fact that the effectiveness of HDR-to-LDR models is influenced by network design and the training process. This is why we pretrain the tone mapper network initially. Traditional tone mapping algorithms, on the other hand, do not require pretraining of parameters, allowing them to converge stably.

Q2.Has there been any experimental analysis on the training stability with HDR inputs?

We use a reinahrd tone mapping [1] method to compress the dynamic range and quantize it into different bit-depths before feeding it into the detector. Experiments show that 14-bit data yields the best performance, while 8-bit data yields the worst. A wider dynamic range preserves more useful details, but under extreme lighting conditions, it can render most details indistinguishable, making it challenging for neural networks to extract meaningful feature.

Q3. What is the size the predicted weight map in low-resolution ? Would this patch-wise weight map affect the scale-invariance of subsequent processing?

Our method predicts the scale map $K$ in a low-resolution space (256x256) and then upsamples it to the original resolution using bilinear interpolation. This scale map does not affect the scale-invariant property of the tone mapper, which is determined by the network design. Bias-free CNNs (with BN and ReLU) generally exhibit scale invariance.

Q4. The authors may consider comparing with HDRNet.

We compared HDRNet[2] and the results are as follows.

HDRNet (from scratch) refers to a cascade with a detector trained from scratch, while HDRNet (pretrained) is pretrained on the RoD dataset (HDR-LDR pairs).

[1] Reinhard, Erik, et al. "Photographic tone reproduction for digital images." Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 2023. 661-670.

[2] Gharbi, Michaël, et al. "Deep bilateral learning for real-time image enhancement." ACM Transactions on Graphics (TOG) 36.4 (2017): 1-12.

Dear Reviewer zmDQ

The discussion period is ending soon. We hope this discussion has addressed your concern regarding training stability.

During the rebuttal phase, we used gradient variance, to represent training stability and modeled the relationship between gradient variance and dynamic range. We modeled the dynamic range using a histogram that spans from 0 to 1, such that the dynamic range $d_i=\frac{\mu_i+\sigma_i}{\mu_i-\sigma_i}>1$. Quantitative studies show that as $d_i$ increases, gradient variance also increases, leading to training instability.

I hope these responses address your concerns and provide some insights.

Best,

Authors

Dear Reviewer 4R6p,

Thank you very much for your insightful and valuable feedback. We believe there might be some misunderstandings, and we sincerely hope to clarify them and adequately address your concerns.

Misunderstandings

Q1. Evaluations without ISP (HDR RAW Data).

Actually, we have reported the performances of using HDR RAW data directly and HDR ISP in Table 1 and present the data point as follows.

For HDR RAW data, we cannot get normal results (0.049, 0.092, 0.032, 0.059) for training collapse. It is important to note that 24-bit HDR RAW data has an extremely large dynamic range. Directly using HDR RAW data can lead to gradient explosion, causing the loss function to become NaN, which makes the training process non-convergent.

Regarding HDR ISP, the results of other methods are significantly lower than those of our approach.

Furthermore, Reviewer 2qV1 acknowledged that our framework demonstrates improved performance compared to traditional ISP pipelines. Reviewer zmDQ regarded our method as having outstanding performance, notably outperforming existing state-of-the-art techniques.

Q2. Missing the details on whether and how they fine-tune the downstream network.

In this paper, we cascade the proposed tone mapping framework with the object detection network. The detection network is initialized with COCO pretrained weights and jointly trained with the tone mapping framework using detection loss in an end-to-end manner. Apart from this, we did not perform any fine-tuning on the detection network.

Concerns

Q3. It is unclear if the evaluation is fair.

Following the RoD dataset [1], we choose the RAODNet23 [1], IANet22 [2], which is most relevant works to ours

For handcrafted tone mapping methods, we select Mantiuk08 [4] and CLAHE [5], which are included in popular development packages and widely used in applications.

We also compare other types of comparative methods, including the Differentiable ISP Method, Low-Light Enhancement Method, and DNN-based ISP Network.

Among these, RAODNet23 [1] serves as the strong baseline method for the RoD dataset, and our method shows significant improvement in comparison (+3.7mAP).

Q4. The tone mapper offers little technical novelty.

The innovativeness of our design is reflected in the following three aspects:

1. The structure we proposed possesses scale-invariant properties, which are supported by mathematical theories. Tone Mapping is employed to compress the dynamic range and preserve details when converting from HDR to LDR. Theoretically, tone mapping necessitates the capacity to map HDR to LDR on a per - pixel basis under arbitrary luminance conditions. This implies that tone mapping must consider all scaling factors, with particular inclusion of the origin (mapping any value to zero). This realization has inspired us to design a scale-invariant tone mapper by eliminating all bias terms. We have proven that bias-free CNNs are scale-invariant, as shown in Eq. 8, and are also equivalent to local tone mapping, as demonstrated in Eq. 24 (in the Appendix).

2. Compared with other methods, the design of the tone mapper is efficient and concise, eschewing complex predefined models. In contrast to related works such as [1], [2], and [3], these approaches pre - define tone curves and estimate curve parameters to dynamically compress the dynamic range for each input. Moreover, the most recent tone-mapping methods, like [4] and [5], utilize Look-Up Tables (LUTs) and pyramid decomposition. This results in a more complex pipeline that is not suitable for deployment on edge devices.

3. Our method holds significant practical value. It exhibits high-speed performance and incurs a lower computational cost. All reviewers, including you, have acknowledged that our method is highly suitable for real-world applications.

[1] Xu, Ruikang, et al. "Toward raw object detection: A new benchmark and a new model." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[2] Liu, Wenyu, et al. "Image-adaptive YOLO for object detection in adverse weather conditions." Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 2. 2022.

[3] Li, Chongyi, Chunle Guo, and Chen Change Loy. "Learning to enhance low-light image via zero-reference deep curve estimation." IEEE transactions on pattern analysis and machine intelligence 44.8 (2021): 4225-4238.

[4] Jiang, Ting, et al. "Meflut: Unsupervised 1d lookup tables for multi-exposure image fusion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Zhang, Feng, et al. "Lookup table meets local laplacian filter: pyramid reconstruction network for tone mapping." Advances in Neural Information Processing Systems 36 (2023): 57558-57569.

Reviewer 4R6p

The discussion period is ending soon. We hope this discussion has addressed your concerns regarding evaluation fairness and novelty.

Regarding evaluation fairness, we provide the implementation details in Section 4 of this paper. All comparison methods are treated as tone mappers, cascaded with the detector and trained end-to-end using a detection loss. The tone mapper preprocesses the input HDR-RAW image before feeding it to the detector network.

The HDR-ISP method generates processed images offline before performing object detection. While the other methods run the detector jointly, processing images and applying detection online. We follow the standard strategy for detector training: "1x" schedule, meaning initialization with coco pretrained weights, 13 epochs of training, and multi-step learning decay. These settings are adopted from previous work.

Regarding novelty, we analyze the requirements of HDR tone mapping and design a tone mapper based on the principle of scale invariance. Although the proposed tone mapper is straightforward, it is highly efficient, balancing performance and speed without unnecessary complexity. It can process 4K resolution input in real-time, a capability that comparison methods cannot achieve.

I hope these responses address your concerns and provide some insights.

Best,

Authors

Dear Reviewer 2qV1,

We'd like to thank you for your positive feedback and address the concerns raised in your comments

Concerns

Q1. Why is the learnable transformation function in Equation 4 designed in this particular form?

Since the input HDR RAW images are recorded as digital numbers, we need to make educated guesses about the minimum and maximum radiance values in the original scene. We introduce Eq. 4, where the bias term is constant to avoid excessively small values, and the scale map consists of learnable parameters that adjust the dynamic range. This approach maps HDR inputs into a unified luminance space through neural photometric calibration, addressing the large diversity in lighting conditions (e.g., day/night, mixed scenes).

Q2. There is some ambiguity in the model description regarding the use of "Tone Mapper."

Thank you for your suggestion. We will further clarify the description and distinguish the details between the tone mapper and neural photometric calibration in the paper. The tone mapping framework includes the scale-invariant tone mapper and neural photometric calibration, which work together for object detection. In Section 3.4, we pretrain only the scale-invariant tone mapper to warm up the parameters, enabling the tone mapper to gain the ability to compress the dynamic range. In Section 4.3, under the performance transfer strategy, we fine-tune the parameters of the tone mapping framework (tone mapper + neural photometric calibration) and the detection head to achieve rapid adaptation from the LDR RGB domain to the HDR RAW domain.

Q3. In the experiments of Table 5, how exactly was the scaling-invariant property removed?

Scale invariance means that the output changes in the same scale as the input. For HDR-to-LDR tone mapping, we argue that if the tone mapping network operates by scaling the HDR image onto a linear space of LDR images, then that space should encompass all rescalings of those images, especially including the origin. To achieve this, we design a bias-free CNN where, for any input $x$, the components (convolution, ReLU, and batch normalization) satisfy $f(ax)=af(x)$ for $x>0$. This ensures that any negative entries in the input are set to zero. Multiplying by a nonnegative constant does not change the sign of the entries in a vector. Thus, the bias-free network remains scale-invariant and can rescale the image without introducing any bias value. In contrast, a bias network would introduce additional background noise in the resulting images.

Q4. In Table 1, different algorithms should be clearly labeled with their respective grouping characteristics to better highlight the focus of each algorithm category.

Thank you for your suggestion. We will add the category group and modify Table 1 to the following style.

Dear Reviewer Vk4C,

Thank you for your feedback. We’d like to provide some contextual information to help you better understand our method and address your concerns.

Learning-based HDR Perception

(1) HDR data contains richer tones and more details, which have the potential to boost various computer vision tasks. However, neural networks are usually designed for LDR input, which requires compressing the dynamic range before directly utilizing HDR data.

(2) There are many ways to compress the dynamic range, with tone mapping being the most common. Handcrafted tone mapping methods are based on human visual perception, while deep learning-based tone mapping is guided by the loss function.

Concerns

Q1. Why is joint training of the tone mapper and object detector preferable compared to using a modular tone mapping approach followed by detection?

The HDR images present an extreme dynamic range compared to LDR images. For example, 24-bit HDR contains about 1e7 times the dynamic range of 12-bit LDR, directly feeding these HDR images into neural network can lead to gradient explosion. Therefore, tone mapping is necessary to compress the dynamic range initially. Most modular tone mapping methods are crafted to adhere to human visual perception, but this alignment may not always be conducive to meeting the demands of downstream high-level tasks. Since a DNN-based tone mapper is cascaded with an object detector, which optimizes the model parameters driven by detection loss, it can generate the most suitable image for enhanced detection performance. Besides, some of these DNN-based tone mappers are plug-and-play (such as our method) and can be cascaded with any object detector without any internal changes, which can also be seen as a modular approach.

Q2. Are there scenarios where the added complexity of joint training is justified, and if so, could the authors provide practical examples or empirical evidence illustrating these benefits?

In challenging imaging conditions (e.g., high dynamic range scenarios, rainy/haze weather), target detection requires a specialized image preprocessor. Some handcrafted image preprocessing methods provide limited generality, as they cannot generalize well across all examples. To address this, cascading a DNN-based image preprocessor (eg, tone mapper, dehaze network) with an object detector can improve performance, although it introduces additional complexity.

Dear reviewers, Thanks so much for reviewing the paper. The discussion period ends soon. To ensure enough time to discuss this with the authors, please actively engage in the discussions with them if you have not done so.