Revisiting Generative Infrared and Visible Image Fusion Based on Human Cognitive Laws

We sincerely thank you for the valuable comments and suggestions. We have carefully revised the manuscript according to your feedback. All modifications have been incorporated into the revised version. Below, we respond to each of your concerns in detail.

Response to Weakness 1:

In our framework, we deliberately apply transformation only to the infrared image while keeping the visible image unchanged. This design choice is primarily based on two considerations:

(1) Modality discrepancy: Infrared images generally contain less structural detail and more semantic ambiguity compared to visible images due to inherent sensor limitations. Therefore, aligning the infrared modality towards the visible one — which typically exhibits more stable textures and clearer semantic content — helps to reduce cross-modal discrepancies and facilitates the learning of more consistent joint representations.

(2) Computational efficiency and robustness: Aligning both modalities simultaneously would require computing and optimizing two separate transport plans, increasing training complexity and introducing potential inconsistencies between mappings. A one-sided transformation from infrared to visible ensures efficiency and stability during optimization.

In practice, we compute a regularized optimal transport plan using the following formulation:

$$\mathbf{P}^* = \arg\min_{\mathbf{P} \in \mathcal{U}(r, c)} \sum_{i,j} P_{ij} C_{ij} + \varepsilon \sum_{i,j} P_{ij} \log P_{ij},$$

where $\mathcal{U}(r, c)$ denotes the set of doubly stochastic matrices with row and column marginals $r$ and $c$, $C_{ij}$ is the cost matrix computed by squared Euclidean distances between flattened pixel values of the infrared and visible images, and $\varepsilon$ is a regularization coefficient. After obtaining the transport plan $\mathbf{P}^*$, the infrared image $X \in \mathbb{R}^{B \times C \times H \times W}$ is first flattened to $X_{\text{flat}} \in \mathbb{R}^{B \times N \times C}$, where $N = H \times W$, and transformed by:

$$X' = \mathbf{P}^* \cdot X_{\text{flat}},$$

followed by reshaping back to $\mathbb{R}^{B \times C \times H \times W}$. The transformed infrared image $X'$ and the original visible image $Y$ are then fed into the variational bottleneck encoder for subsequent fusion modeling. We have added this explanation and implementation clarification in the revised manuscript for better clarity.

Response to Weakness 2:

Thank you for raising the questions regarding the derivation and effectiveness of the mask weights $M_s$ described in line 174. To clarify, the mask weights $M_s$ are computed as $M_s = \mathrm{sigmoid}(w_s)$, where $w_s \in \mathbb{R}^{1 \times C \times 1 \times 1}$ is a learnable parameter introduced at each encoder scale. This parameter is initialized from a standard Gaussian distribution using "torch.randn()" and registered as an “nn.Parameter". The sigmoid activation ensures that the mask values remain within a continuous range of $(0, 1)$, enabling soft, differentiable modulation of feature importance.

Response to Weakness 3:

We sincerely thank you for highlighting the importance of clarifying the construction process of the masks $M_{\text{heat}}$ and $M_{\text{stru}}$. These masks are explicit, non-learnable spatial priors designed based on statistically meaningful physical patterns inherent in the input modalities.

*$M_{\text{heat}}$ is derived from the thermal intensity distribution of the infrared image. Specifically, we apply a $3 \times 3$ average pooling to obtain a local heat response map, then compare each pixel to the global mean value to construct a binary mask—pixels above the mean are set to 1, others to 0. This explicitly highlights thermally significant regions and models localized heat saliency.

*$M_{\text{stru}}$ captures structural saliency in the visible image by applying a Laplacian operator to extract high-frequency edge responses (i.e., local second-order derivatives). This structural response map reflects intensity variations along object boundaries. We binarize it via thresholding at the mean, thereby identifying regions with prominent structural features.

Both masks are dynamically updated at each reverse diffusion timestep, providing a time-varying guidance mechanism that adaptively emphasizes thermal or structural cues depending on the generation stage. This design enhances modality complementarity while reinforcing physical consistency and interpretability throughout the sampling trajectory.

Response to Weakness 4:

To address your concern, we conduct a detailed ablation study covering all three types of masks involved in our framework.

Ablation 1 — W/O $M_s$:

We remove the $M_s$ from the latent representation and observe clear performance degradation across multiple perceptual and structural metrics, confirming that $M_s$ is essential for filtering informative latent variables and reducing redundancy.

Ablation 2 — W/O $M_{\text{heat}}$ & $M_{\text{stru}}$:

We replace $M_{\text{heat}}$ and $M_{\text{stru}}$ with all-one masks, effectively removing their spatial selectivity. The performance drops consistently across key indicators related to detail and structure preservation (e.g., AG, SF, QSF), indicating that omitting spatial mask guidance weakens both visual quality and perceptual fidelity.

The results are summarized in the revised ablation table (see Appendix), which is reproduced below for clarity. These findings demonstrate that each mask plays an indispensable role in its corresponding mechanism: $M_s$ in the bottleneck pathway, and $M_{\text{heat}}$, $M_{\text{stru}}$ in physically-guided image generation.

The best results are highlighted in bold.

Response to Weakness 5:

We have revised the layout of Table 5 to ensure that the last column is properly aligned and fits within the page boundaries. Additionally, we have checked and adjusted the formatting of all other tables to ensure consistent alignment and presentation throughout the manuscript.

Response to Question a:

We sincerely thank you for raising this important point. In response, we have conducted additional experiments on the MFNet dataset. As shown in the results table below, our method retains strong robustness and generalization ability even under degraded sensor conditions.

The best results are highlighted in bold, while the second-best results are marked with a † symbol.

Response to Question b:

We sincerely apologize for the omission. We have added the citation to Clark's work in the revised manuscript (Line 134):

Clark, A. "Whatever next? Predictive brains, situated agents, and the future of cognitive science." Behavioral and Brain Sciences, 2013, 36(3): 181–204.

Response to Question c:

We appreciate your advice and have carefully addressed the concerns raised in the Weaknesses section.

We sincerely thank you for your constructive feedback and recognition of the contributions of our work. Below, we address each of the reviewer’s concerns in detail.

Response to Weakness:

To address the concern that parts of the theoretical presentation may hinder accessibility for a broader audience,

(1) We have carefully revised the manuscript based on your comments, improving unclear expressions and enhancing clarity throughout;

(2) We have added more references to support the foundational theories involved;

(3) We will release the complete code in due course to promote transparency and reproducibility.

Response to Question 1:

Thank you for the comment regarding the mask weights $M_s$ in Eq.~(6), which represent a critical component of the proposed design. At each encoder scale, a structurally defined learnable parameter $w_s \in \mathbb{R}^{1 \times C \times 1 \times 1}$, where $C$ denotes the number of feature channels, is introduced. The parameter is initialized from a standard Gaussian distribution using “torch.randn()" and registered as an "nn.Parameter" in the model. During forward propagation, $M_s = \mathrm{sigmoid}(w_s)$, forming a differentiable soft mask applied to the feature map. Throughout training, $w_s$ is optimized under the guidance of the information bottleneck loss $\mathcal{L}_{\text{VBE}}$, functioning as an auxiliary modulation mechanism that enhances the selective suppression of redundant information within the latent representation.

To address your concern, we conduct a dedicated ablation study to examine the individual and joint contributions of the mask mechanism and the information bottleneck loss. The results clearly show that removing either component degrades performance, while the best results are consistently achieved only when both are jointly applied. This highlights the necessity of their integration in realizing effective latent compression.

Response to Question 2:

Thank you for this helpful suggestion. We have added an overview of the entire HCLFuse process at the beginning of the Method section. In addition, we have included a pseudocode description of the complete workflow in the appendix to further enhance reproducibility and clarity. To facilitate your review, we present below the textual description as it appears in the Method section:

"In HCLFuse, let the infrared image domain be denoted as $X$ and the visible image domain as $Y$, with joint observations satisfying $(x, y) \sim p_{x,y}$. We construct a fusion mapping $F(X, Y) \rightarrow Z$ to generate the fused latent representation $z = F(x, y) \in Z$, where $Z \subseteq \mathbb{R}^d$ denotes the fusion space. Ideally, the fused representation should effectively preserve the complementary information from both modalities while suppressing redundant noise. To achieve this, HCLFuse first employs a multi-scale, mask-regulated variational bottleneck encoder to compress and model the latent representation $z$. Before encoding, an optimal transport-based mapping is applied to improve the optimization lower bound, encouraging $z$ to contain modality-discriminative content in the unsupervised setting. Subsequently, $z$ is refined through a reverse-time diffusion generation process, where physically guided constraints are dynamically injected at each denoising timestep to guide the evolution of latent features. Finally, the optimized latent representation is decoded to produce the fused image."

Response to Question 3:

Regarding your suggestion on the use of VIF, we agree that VIF is a widely adopted full-reference metric for perceptual quality assessment and aligns well with the "Human Cognitive Laws" emphasized in our title. In response, we have included additional experiments and reported VIF results for completeness. However, VIF is a full-reference metric that relies on source images, which conflicts with the goal of generative fusion. Our method is designed to synthesize richer and more complete information by integrating both modalities, and as illustrated in Figures 3, 4, and 6–9 in the paper, the fused outputs often surpass the original inputs in clarity, detail, and semantic structure. As a result, using source images as references tends to underestimate the perceptual quality of the generated fusion results.

To address this, we have further introduced two no-reference perceptual metrics—PIQE [IEEE NCC, 2015: 1–6.] and BRISQUE [IEEE TIP, 2012: 4695–4708.]—which are well-suited for evaluating visual quality without requiring reference images. They capture distortions and unnatural statistics in a way that correlates with human perception. As shown in the table below, although our method achieves a moderate score on the VIF metric, it demonstrates superior performance on both the PIQE and BRISQUE metrics, highlighting its perceptual quality in image generation.

The best results are highlighted in bold, while the second-best results are marked with a † symbol.

Response to Question 4:

We sincerely thank you for pointing out the lack of explicit validation for the heat conduction constraint. To address this concern, we have conducted a comprehensive ablation study targeting each physical constraint. The results are summarized in the updated Table:

The best results are highlighted in bold.

Experimental Analysis:

From A → B, introducing only the heat conduction constraint $\Phi_{\text{heat}}$ already improves several perceptual indicators (e.g., AG, EN, SF), and achieves the lowest Nabf and PIQE.

From B → C, adding the structure preservation constraint $\Phi_{\text{stru}}$ further enhances structural similarity (e.g.,SCD), showing that $\Phi_{\text{stru}}$ helps maintain sharpness and edge consistency.

From C → Ours, the inclusion of the physical consistency constraint $\Phi_{\text{con}}$ results in the best overall performance across almost all metrics, with notable gains in SD, AG, EN, SF, and VIF. This underlines the complementary and synergistic role of the three constraints.

These results collectively validate the effectiveness of each individual physical constraint, and further demonstrate that only the full combination of all three can maximize overall fusion performance.

Response to Question 5:

We sincerely thank you for pointing out the lack of clarity regarding DDIM. This is indeed an important component that was not sufficiently explained in the original version. DDIM (Denoising Diffusion Implicit Models) [Song et al., ICLR 2021] is a deterministic variant of the standard diffusion model DDPM(Denoising Diffusion Probabilistic Models) [NeurIPS, 2020: 6841–6853]. In our framework, DDIM serves as the core generative module. In the ablation setting "W/O DDIM", we remove the entire diffusion process and obtain the fused image directly through an encoder-decoder structure. As shown in Table 2 and Figure 5 in the paper, this leads to a loss of fine details and noticeable drops in several metrics, confirming that the generative process provides essential complementary information for high-quality fusion. Additional explanations and the relevant citation have been added in the revised manuscript for clarity.

We deeply appreciate your thoughtful and detailed review. In response, we have carefully revised the manuscript to address all your comments. A point-by-point response is provided below.

Response to Weakness 1:

We fully agree that comparing against more contemporary and competitive approaches is crucial to demonstrating the effectiveness of our proposed method. We have conducted extensive additional experiments to include the recent state-of-the-art methods mentioned in the review, such as TarDAL, DDFM, SegMIF, Text-IF, CCF, Text-DiFuse, and GIFNet. These methods represent leading advances from top-tier conferences. To present a more comprehensive comparison, we have included a more challenging dataset, MFNet, to further validate the robustness and generalization ability of our method on low-quality image pairs. The additional experimental results are shown in the table below. To improve clarity, we use bold to indicate the best result and † to mark the second-best. Overall, the proposed method consistently achieves superior performance across most metrics, confirming its robustness and generalization ability even under comparison with the latest and strongest baselines.

（Due to the rebuttal's character limit, we only present the fusion performance here, while the evaluation on the segmentation task is included in the revised manuscript. If needed, we would be happy to provide those results separately.）

(1)MSRS Dataset

(2)FMB Dataset

(3)TNO Dataset

(4)MFNet Dataset

Response to Weakness 2:

We have added a new ablation study that systematically isolates and analyzes each individual constraint. As shown in the table below, the progression from the baseline (without any physical guidance) to successively adding these constraints yields clear and interpretable results:

From A → B, introducing only $\Phi_{\text{heat}}$ significantly boosts perceptual indicators such as AG, EN, and SF, while also achieving the lowest Nabf. This demonstrates that enforcing thermal smoothness leads to tangible benefits even in isolation.

From B → C, adding $\Phi_{\text{stru}}$ leads to further improvements in structure-related metrics such as SCD, confirming its effectiveness in preserving edges and local details.

From C → Ours, incorporating the full $\Phi_{\text{con}}$ term further enhances global consistency and cross-modal balance, resulting in the best overall performance across nearly all metrics, including SD, AG, VIF, and QSF.

This layered ablation demonstrates that each constraint contributes uniquely to different aspects of fusion quality. More importantly, the full combination of all three constraints achieves the most balanced and robust results.

The best results are highlighted in bold.

Response to Weakness 3:

We appreciate your comment, which helped us correct this oversight. The descriptions of the three physical constraints in Equations (14)–(16) are indeed inaccurate. In fact, these constraints are applied sequentially in a cascaded manner. We have revised the manuscript to clearly state that the constraints are applied in order at each timestep:

z0_hat --> $\Phi_{\text{heat}}$ --> z0_hat^heat --> $\Phi_{\text{stru}}$ --> z0_hat^stru --> $\Phi_{\text{con}}$ --> z0_hat^phys

Response to Weakness 4:

In response to this suggestion, we have conducted a comprehensive study on the key hyperparameters used in our framework. Specifically, we divided them into three groups and performed targeted experiments to evaluate their influence on performance.

(1) Analysis of $\beta$ in Equation (5): We evaluated the trade-off coefficient $\beta$ (values: 0.1, 1, 10), which balances compression and information preservation in the variational bottleneck. The best performance occurred at $\beta = 1$, achieving an effective trade-off between KL regularization and reconstruction. Smaller $\beta$ led to under-regularization and instability, while larger values suppressed informative features.

(2) Analysis of the physical guidance weights $\lambda_{\text{heat}}$, $\lambda_{\text{stru}}$, and $\lambda_{\text{con}}$ in Equations (14–16): A grid search over values {0.05, 0.10, 0.15} for each $\lambda$ (with others fixed) yielded six combinations. The best result is with $(\lambda_{\text{heat}} = 0.05, \lambda_{\text{stru}} = 0.15, \lambda_{\text{con}} = 0.10)$, reflecting an effective balance: low $\lambda_{\text{heat}}$ prevents over-smoothing, high $\lambda_{\text{stru}}$ preserves structure, and moderate $\lambda_{\text{con}}$ promotes stable cross-modal fusion.

(3) Analysis of the decay factor $\gamma$ in Equation (17): We tested $\gamma = 1, 5, 10$ to examine its effect on time-dependent physical guidance. The optimal $\gamma = 5$ maintained sufficient guidance while preserving flexibility. $\gamma = 1$ was too weak, causing blurred structures; $\gamma = 10$ was too strong, degrading performance by over-constraining the model.

Due to the rebuttal's character limit, only a brief summary of the quantitative results is included here. And the complete ablation results are presented in the revised manuscript. If you would like to review the complete results, we would be happy to provide them in a follow-up.

Response to Question:

We have sincerely and thoroughly addressed all the concerns you raised. The manuscript has been carefully revised based on your suggestions, and we look forward to your further feedback.

We sincerely thank you for your insightful and constructive comments regarding the core contributions and scientific novelty of our work. Your thoughtful observations prompted us to more clearly articulate the essence of our contribution. We would like to clarify that the central innovation of this work does not lie in simply applying existing mathematical tools, but in redefining unsupervised image fusion under limited-data settings as a cognition-inspired generative problem. To this end, we present a coherent modeling strategy spanning theoretical motivation, architectural design, and mathematically grounded implementation. This reflects our shift from traditional heuristic fusion pipelines to an interpretable and principled generative approach guided by cognitive processes. Specifically, our innovations span three levels:

*A cognition-inspired image fusion framework is proposed by reformulating the fusion task as a generative problem, aiming to obtain high-quality fused images beyond the limitations of individual modalities.

*An unsupervised information mapping and quantization theory is proposed to characterize the trade-off between multi-source information compression and discriminability, based on which a multi-scale variational bottleneck encoder is designed to extract task-relevant low-level feature representations.

*A dynamically evolving physics-guided mechanism is incorporated into the diffusion process, where temporally progressive physical priors enhance structural perception and generalization, effectively reducing dependence on large-scale training data.

To address your concerns in detail, we respond to your comments point by point below.

Response to Weakness 1:

Regarding your concern about the use of the phrase “human cognitive laws” , we fully acknowledge that this wording may have unintentionally overstated our intention. We have revised the manuscript to clarify that our framework is guided by cognition-inspired modeling principles, rather than literal human cognitive laws. Specifically, we are inspired by how humans process multi-source information: filtering redundant content, attending to salient cues, and forming abstract internal representations. The Information Bottleneck (IB) principle models the selective compression process inherent in such cognitive abstraction, while Optimal Transport (OT) provides a tractable and unsupervised alternative objective that enables alignment and regularization of cross-modal information. This theoretical foundation is detailed in Theorem 1. Together, these tools are not themselves “cognitive laws,” but are used to instantiate a formal, interpretable, and optimizable framework that reflects cognition-like information processing. We hope this can clarifies our conceptual positioning.

Response to Weakness 2:

Thank you for raising this important point. We fully agree that individual components such as Laplacian smoothing have been used as traditional image priors. However, the novelty of our work does not lie in each constraint’s isolated form, but in how they are dynamically and sequentially embedded into the generative diffusion process through a mechanism we term Time-varying Physical Guidance (TPG). This mechanism introduces three physically-inspired constraints—heat conduction ($\Phi_{\text{heat}}$), structure preservation ($\Phi_{\text{stru}}$), and cross-modal consistency ($\Phi_{\text{con}}$)—that are:

Applied in sequence to the predicted sample at each reverse diffusion timestep;

Modulated by a decaying coefficient $\lambda(t)$ that mimics coarse-to-fine reasoning;

Designed to stabilize generation and guide physical structure, not merely smooth pixels.

This guidance process mimics the cognitive behavior of humans resolving ambiguous information by applying strong structural priors early and gradually relaxing them as certainty increases.

Response to Weakness 3:

While our Variational Bottleneck Encoder (VBE) adopts a variational formulation, it is not a simple extension of the conventional VAE. The key distinction lies in both its theoretical foundation and functional role. VBE is directly motivated by our theoretical result in Theorem 1, where we reformulate unsupervised image fusion as an information quantization problem, leading to a principled, task-aware latent abstraction objective that fundamentally differs from the reconstruction-driven design of standard VAEs.

More importantly, VBE serves as a functional bridge between mutual information optimization and the generative process. It provides the diffusion model with a structurally coherent and task-relevant latent space, effectively relieving the burden of handling complex cross-modal alignment and selection. This allows the diffusion model to focus on modeling high-quality fused outputs guided by the compressed priors. Notably, most existing diffusion-based methods focus solely on improving the architecture itself, while overlooking the task relevance and informational clarity of the latent inputs. Our approach addresses this upstream bottleneck optimizing the content of the latent space, thereby enhancing both the generative ability and fusion quality.

Supplementary Experiments:

To further support your concerns on originality and effectiveness, we have conducted comprehensive additional experiments:

(1) An ablation analysis of both the internal structure and overall design of VBE demonstrates its critical role in compact encoding;

(2)A fine-grained ablation study on each physical constraint confirms their individual benefits and complementary synergy;

(3)A hyperparameter study over $\beta$, $\lambda_{\text{heat/stru/con}}$, and decay factor $\gamma$ validates the robustness of our design choices;

(4) Comparative evaluations have been expanded by incorporating additional top-tier conference methods, resulting in a total of seventeen compared approaches (e.g., TarDAL [CVPR 2022], DDFM [ICCV 2023], Text-DiFuse [NeurIPS 2024], Text-IF [CVPR 2024], GIFNet [CVPR 2025]), and are conducted across twelve quality metrics, including spatial detail, structural preservation, and perceptual quality;

(5) The MFNet dataset has been additionally introduced, and the experimental results further verify the generalization ability of the proposed method under low-quality scenarios.

Thank you once again for your thoughtful comments, which have significantly improved our presentation and helped sharpen the contribution of our work. We hope the revised version addresses your concerns and clarifies the novelty and value of our approach.