Test-Time Dynamic Image Fusion

We’d like to thank the reviewer for the valuable comments, the acknowledgment of our good results, and the simple but clear presentation. We provide detailed responses to the constructive comments.

• Weakness 1: More explanations for RD.

Thanks for the valuable comments. We have added experiments under noise conditions and with sources of different qualities. In addition, our RD's effect was validated on different methods and tasks. These experiments and the visualization of RD empirically demonstrate that the proposed dynamic weights (RDs) based on theoretical principles are effective in capturing and emphasizing the dominant regions of different sources, leading to outstanding fusion performance.

(i) Comparisons on different tasks over multiple baselines.

• For different scenarios, we have evaluated our TTD on four image fusion tasks: VIF (see Tab. 1), MIF (see Tab. 3-5), MEF (see Tab. 2), and MFF (see Tab. 2).
• For different baselines, we have applied our TTD to CDDFuse (CDDFuse+TTD), PIAFusion (PIAFusion+TTD), and IFCNN (IFCNN+TTD), separately. Results are given in Tab. 1-5.

(ii) The visualizations of RDs in various scenarios.

• RD is adaptable to the noise condition. We simulate a noisy situation in which the visible image quality is affected by contrast perturbation. As shown in Fig. A4, with the corruption severity level increasing, the dominant regions of visible modality are gradually reduced, while the unchanged infrared modality gains an increasing RD. Our RD effectively perceives the dominance changes.

• RD is adaptable to different data qualities. a) To simulate the malfunction of sensors in a real scenario, we masked the infrared image randomly. As shown in Fig. A2, the RD of the region being masked is apparently smaller than the surrounding area, while that of the same region in infrared image is relatively greater. b) Furthermore, the quality of images also changes with illumination. As shown in Fig. 6 (see Appendix C.1), we visualized the RDs of the samples at different times in the same scenario. As it changes from day to night, the dominance of visible images gradually decreases, while the dominance of infrared images increases.

• RD is adaptable to different tasks. We have presented visualizations of RDs obtained using CDD+TTD for VIF and MIF tasks, and IFCNN+TTD for MEF and MFF tasks, as CDDFuse is a VIF-specific method while IFCNN is a unified approach. As shown in Fig. 1, our RD reflects the dominance of each source on different tasks adaptively.

(iii) RD's effectiveness over varying performance baseline. We further conduct additional experiments to combine TTD with baselines of varying performance. As shown in Tab. A2 of global rebuttal PDF, our TTD significantly improves these baselines with different performance. This validates that our TTD effectively improves baselines in different scenarios and with different performances.

• Weakness 2: More explanations for normalizing the weights.

Thanks for the valuable comment. In Eq. 7, we performed a softmax normalization on $w$ as the sum of $w$ being 1 is a prerequisite for deriving the upper bound of generalization error (GError). As shown in Fig. A1 of global rebuttal PDF, we normalized the weight maps of different sources at the same positions.

• Weakness 3: More explanations for the initialization.

Thanks for the constructive comment. The initialization of $w$, which is multiplied with unimodal features, only participates in the uni-source reconstruction process to adapt the uni-source features to the feature space of the baseline. If the initialization of $w$ is different from that of the baseline in the reconstruction process, the feature distribution will deviate from the baseline's feature space, affecting the reconstruction performance. Accordingly, we have added experiments with different initialization weights during uni-source reconstruction to explore the effect of different initializations. According to Tab. A3 of global rebuttal PDF, we set $w$ the same as the baseline.

Besides, unlike traditional test-time adaptation methods [20][21], our TTD does not require fine-tuning the network. In the fusion process, $w$ can be obtained by a single calculation step (Eq. 6). Theoretically, the fused image can be regarded as a linear combination of $M$ uni-source components. We reveal that the fusion model's upper bound of GError is composed of the distance between the uni-source image and each uni-source component as well as the correlation between fusion weight and uni-source component reconstruction loss according to Eq. 4. As the model is frozen in the test time, the essence of reducing GError lies in the negative correlation between fusion weight and uni-source component reconstruction loss. TTD performs fusion by the dynamic weight negatively correlated to the reconstruction loss according to Eq. 6 by a single calculation step without any training or fine-tuning, reducing the generalization error and achieving robust results.

• Weakness 4: More explanations for the framework.

Thanks for your comments. We draw a more detailed pipeline for inference in Fig. A3.

In stage 1 (dashed line), we feed each uni-source image individually into the frozen encoder and decoder to acquire the respective decomposed uni-source components. Then, we construct the RD according to Eq. 6.

In stage 2 (solid line), we feed multi-source images into the encoder and get their corresponding features. Then, we fuse features by multiplying the RDs to the respective features and adding them up. Finally, the fused feature is fed into the decoder for the final fusion results.

Please kindly note that the fusion of different sources only occurs at the fusion layer of the baseline without interactions between features from different layers. We will release all our code for reliable reproduction.

Thanks a lot for your reply. We are delighted to have addressed your concerns. We also appreciate your insightful comments, which have greatly improved our work and inspired us to research more.

We sincerely thank the reviewer for your valuable comments and appreciate your recognition of the theoretical justification, meaningful and sufficient experiments as well as the superiority of our work. We believe the constructive feedback will improve the paper and increase its potential impact on the community.

• Weakness 1: The details presentation and explanation about test-time adaption are not very clear.

Thanks for your constructive comment. To illustrate the TTD workflow more clearly, we draw a new detailed pipeline during inference in Fig. A3 (see the global rebuttal PDF).

In stage 1 (dashed line), we input each uni-source image individually into the frozen encoder and decoder to acquire the respective decomposed uni-source components. Then, we construct the RD which is negatively correlated to the distance between the uni-source component and its original image according to Eq.6.

In stage 2 (solid line), multiple source images are fed into the encoder at the same time and get their corresponding features. Next, we get the fused feature by multiplying the RDs as weights to their respective features and adding them up. Finally, the fused feature is input into the decoder and the fused image is obtained.

• Question 1: Is the idea that adding up every uni-source data linearly to get fused image reasonable enough? Are there any information only found with several data neglected in the whole process?

Thanks for your valuable comment.

(i) Most recent approaches [18][19] imply multi-source image fusion aims at integrating comprehensive information from different source data. The key challenge lies in capturing the effective component of each uni-source data. To address this problem, we theoretically propose a method that can effectively extract single-source information.

• Theoretically, the fused image can be regarded as a linear combination of $M$ uni-source components. We reveal that the model's upper bound of generalization error in the image fusion task is composed of the distance between the uni-source image and each uni-source component as well as the correlation between fusion weight and uni-source component reconstruction loss according to Eq. 4. As the model's encoder and decoder are frozen in the test time, the essence of reducing generalization error lies in the negative correlation between fusion weights and uni-source component reconstruction loss. TTD performs fusion by the dynamic weight negatively correlated to the reconstruction loss, achieving a reduction in the generalization error compared with the baseline.

• Empirically, the fusion weight, e.g. RD (defined in Eq. 6), since fusion models are trained to extract complementary information from each source, the decomposed components of fusion images represent the effective information from the source data. Thus, the uni-source components can be estimated from source data using the fusion model, with the losses representing the deficiencies of the source in constructing fusion images. Negatively correlated to the reconstruction loss, the RD effectively demonstrates the dominance of each source and highlights the dominant regions as a fusion weight.

(ii) With the neglect of other source data, the uni-source reconstruction can be regarded as the corresponding decomposed component of the fusion image, it represents the effective information from the source data, thus the loss between it and source data reflects the Relative Dominability (RD) of the uni-source data in constructing the fusion image, i.e., we leverage the impact of the missing modality on the loss to perceive the RD of that source in the fusion image and use the RD as fusion weight. This aligns with the theoretical guarantee of image fusion (see (i)): the key to reducing the generalization error in image fusion tasks lies in the negative correlation between fusion weights and uni-source component reconstruction loss.

• Question 2: Mathematical formulas in Appendix A.1 are a little confused. More parentheses ought to be used to present clear explanations about the scopes of every mathematical symbols.

Thanks for the valuable suggestions. Here we provide a detailed explanation of the proof in Appendix A.1. At the second equals sign in Eq. 10, the covariance term is derived from the covariance formula: $\mathbb E[XY]=\mathbb E[X]+\mathbb E[Y]+Cov(X, Y)$. At the second inequality sign in Eq. 10, the expression is expanded by grouping $\Vert D(E^{(i)}(x^{(i)})) − x^{(m)}\Vert$ as like terms and expanding the summation notation. Then it is simplified by combining like terms using $w^{(i)}$. At the third inequality sign in Eq. 10, the property of the norm is used, i.e. $\left\|a\right\|-\left\|b\right\|\leq\left\|a+b\right\|$, to achieve further simplification. Due to character limitations, a more detailed derivation and more easily understandable parentheses have been revised and added in Appendix A.1.

• Question 3: Is the TTD applied to every combination from sources? Most test-time adaption methods only need a few data to fine-tune the model. But this paper seems to apply TTD to all source data.

Thank you for your valuable questions. Most test-time adaptation methods [20][21] require a small amount of data to fine-tune the model. Unlike traditional test-time adaptation methods, our TTD does not require fine-tuning any network parameters. For each sample, the negative correlation between the fusion weight $w$ and the uni-source reconstruction loss can reduce the generalization error upper bound according to the theoretical analysis in the answer to question 2. TTD performs fusion by the dynamic weight negatively correlated to the reconstruction loss referring to Eq.6 through a single calculation step without any training or fine-tuning, reducing the generalization error and achieving robust results.

Dear Reviewer gbUk, I have noticed that you have not yet responded to the authors' rebuttal. I kindly urge you to engage in a discussion with the authors at your earliest convenience to help advance the review process.

We would like to thank the reviewer for the thoughtful and thorough comments on our paper as well as for recognizing our theoretical guarantee, the simple but effective framework of our TTD, the detailed and rich experiments, and the competitive effect compared to SoTAs. We will also make an effort to increase clarity throughout.

• Weakness 1: Authors do not mention the change in model inference efficiency.

Thanks for the constructive comment. The inference time of TTD is dependent on the inference time of the baseline. Since TTD is executed in two stages: in the first stage, we calculate the uni-source reconstruction loss and then compute the fusion weights; in the second stage, we perform the fusion based on the weights. As baselines perform the static fusion, the inference time of TTD is approximately double that of the baseline. We measured the average processing time per image on the test set of the LLVIP dataset. The results of the inference time over multiple models are given in Tab. A4 (see the global rebuttal PDF).

• Weakness 2: The paper assumes that the decoder of the fusion model is a CNN model, which introduces certain limitations. If the model were a Transformer or Diffusion model, would TDD still be effective?

Thanks for the suggestive comment. Although we assume that the decoder of the fusion model is a CNN model, we applied TTD on both CNN-based (PIAFusion, IFCNN) and Transformer-based (CDDFuse) models, and achieved competitive results, the results are given in Tab. 1 and Tab. 2. In addition, we apply our TTD to a diffusion-based image fusion method (Dif-Fusion [1]). The experimental results are given as follows, demonstrating that TTD can be applied to baselines with various network structures and improve their performance on multiple metrics.

• Weakness 3: For specific fusion tasks, it is recommended to compare specific methods rather than generalized methods, e.g., multifocus image fusion tasks should compare multifocus image fusion methods.

Thanks for the constructive suggestion. We have added comparisons of our TTD with methods specifically applicable to the multi-exposure or multi-focus task, and the results in Tab. A1 in the global rebuttal PDF shows that our method can outperform these specific methods.

• Weakness 4: Ablation lacks enough persuasiveness.

Thanks for the valuable comment. Our TTD is a simple but effective method with a straightforward structure, and we analyze the effectiveness of the TTD from different aspects in our paper. We have summarized these ablated analyses here:

(i) Ablation study on different baselines: see Sec. 4.2, Tab. 1, and Tab. 2.

(ii) Ablation study on the correlation between weight and loss: see Sec. 5.1 and Fig. 5.

(iii) Ablation study on the ways to obtain weight: see Sec. 5.3 and Fig. 5.

In addition, we added more ablation experiments here:

(iv) Ablation study on different forms of fusion weights. We compared different forms of fusion weight: $w=0.5$ (baseline),$w = Softmax(-\ell)$, $w = Softmax(Sigmoid(-\ell))$, $Softmax(e^{-\ell})$ over IFCNN on the LLVIP dataset, results are given as follows, it shows that forms of fusion can be flexible to achieve the negative correlation between weight and reconstruction loss.

(v) Ablation study on the normalization of the weights: we compared three forms of normalization over IFCNN on the LLVIP dataset, results are given as follows, indicating that as a premise of the generalization theory (see Theorem 3.1), the normalization of the weights is necessary and the ways to normalize have little impact on our method.

Overall, we performed complete ablation analyses to validate the effectiveness of TTD (i), the necessity of the negative correlation between fusion weight and reconstruction loss (ii), the expandability of ways to obtain fusion weight (iii), the flexibility in the form of weights (iv), the significance of normalization (v).

• Weakness 5: Table I and II are written inconsistently e.g. TDD and Ours.

Thanks for the detailed comments. We have modified the claims to avoid inaccurate descriptions, and we have thoroughly reviewed the entire manuscript and corrected these errors.

Thanks a lot for your positive feedback. Your insightful comments have greatly improved our work. We sincerely appreciate your support.

We thank the reviewer for recognizing our effectiveness on multiple tasks, theoretical guarantee, and well presentation. We appreciate your support and constructive suggestions and address your concerns as follows.

• Weakness 1: The adaptation method heavily relies on the performance of the baseline model, which can be ineffective when the model performance is poor.

Thanks for your comment. Please kindly note that our performance is related to the baseline model as we claimed in the Limitations (Line 479-Line 480), however,  it does not imply that our approach would be ineffective on weak-performance baselines. Based on the generalization theory, our TTD explicitly reduces the upper bound of the generalization error of models regardless of the baselines' performance. We have also added experiments to further clarify our effectiveness on the baseline with different performances. We have elaborated on this in two facts:

(i) Theoretically, referring to the deduce in Sec. 3, the fused image can be regarded as a linear combination of multiple uni-source components, i.e. the uni-source reconstruction. We reveal that the model's upper bound of generalization error in the image fusion task is composed of the distance between the uni-source image and each uni-source component as well as the correlation between fusion weight and uni-source component reconstruction loss according to Eq. 4. As the model's encoder and decoder are frozen in the test time, the essence of reducing generalization error lies in the negative correlation between fusion weights and uni-source component reconstruction loss. In static image fusion (baseline), the correlation between weight and reconstruction loss is 0. In contrast, the dynamic weight in our TTD is negatively correlated to the reconstruction loss, achieving a reduction in the generalization error compared with the baseline and improving the baseline's performance effectively. Thus, the key to TTD functioning is independent of the performance of the baseline theoretically.

(ii) Experimentally , in our paper, we have applied TTD to various baselines with different capabilities and all achieved consistent enhancement, TTD can even further improve the performance when combined with current state-of-the-art methods. To further validate that our TTD is effective on models with different performances, we conducted additional experiments to apply TTD on models with varying performance levels by adding random Gaussian noise to the pre-trained model (IFCNN) parameters. The results on the LLVIP dataset are given in Tab. A2 (see the global rebuttal PDF), showing that the performance of the baseline decreases with increasing noise added to it. As a comparison, our TTD effectively improves all these baselines' performance, indicating the effectiveness and generalizability of our TTD on various baselines with different performances.

• Weakness 2 : The improvement over the non-adaptive baseline is minor.

Thanks for your comment. Please kindly note that our method is independent of the baseline's adaptability, which of TTD is reflected in its ability to dynamically adjust the fusion weights to each sample.

(i) The adaptability of TTD. As analyzed in Question 1, TTD can theoretically reduce the image fusion model's generalization error by constructing the negative correlation between fusion weight and uni-source component reconstruction loss. For each sample, we derive a pixel-level Relative Dominablity (RD) as the dynamic fusion weight, and RD is negatively correlated with uni-source component reconstruction loss according to Eq. 6. Experimental results in Tab. 1-5, the visualization in Fig. 1 and Fig. 6 show that RD-based dynamic fusion weight effectively captures the dominance of each source in image fusion and enhances its advantages in the fused images. Overall, this adaptability of TTD refers to its adaptive fusion weights instead of the baseline.

(ii) The improvements over various baselines are not minor. We applied our TTD on three baselines with different performances, and we also conducted extensive experiments on multi-modal, multi-exposure, and multi-focus datasets. The superior performance across diverse metrics demonstrates the effectiveness and applicability of our approach. We further perform additional experiments to combine TTD with baseline models of varying performance. As shown in Tab. A2 (see the global rebuttal PDF), our TTD significantly improves these baselines with different performances. This validates that our TTD effectively improves the various baselines in different scenarios as well as with different performances.

Dear Reviewer gBb1, I have noticed that you have not yet responded to the authors' rebuttal. I kindly urge you to engage in a discussion with the authors at your earliest convenience to help advance the review process.