Style-Coherent Multi-Modality Image Fusion

1. Disentanglement for some non-visual multi-modality tasks.

We add relevant Disentanglement methods for multi-modality tasks in the main paper Sec.2. It highlights the contributions of SCFNet in fusing multi-modal images based with distinct visual characteristic on style-content decomposition.

2. About Fig.1.

We add fused features in the main paper Fig.1. It reveals that fused features without style adjustments are smooth and lack details. Our SCFNet, by fusing style-coherent features, better captures details such as bridge structures and tree edges. Fig. 7 shows SCFNet producing fused images in various styles while preserving complete scene details, highlighting the effectiveness of our SCFNet in integrating content with style modulation.

3. About Fig.2.

The merging operator consists of concatenation followed by a convolutional layer, as shown in the supplementary Fig.1. We also add the description of FPE in the caption of Fig.2.

4. About the shared twin encoder branches.

The reason the twin encoder branches share the same structure and parameters is to reduce complexity and potential discrepancies that might arise from having separate configurations for each modality. We add this into the main paper.

5. The degree of style modification is gradually adjusted.

We conduct an analysis of the learnable $\lambda^1$ and $\lambda^2$ in Eq.5 and Eq.6. As shown in the following Tab.1, at the initial level of encoders, SNF tends to preserve more of the source modality style, maintaining the uniqueness of each modality. As the level deepens, the model increasingly relies on the fused features to enrich the representation of content. Furthermore, since the fused feature is encouraged to align with the visible image domain, $\lambda^1$ for visible tends to preserve the source modal amplitude more than $\lambda^2$ for infrared. We will include this analysis in the supplementary materials for the revision.

Table1:Parameters $\lambda$ of SNF in IVF task.

6. Pooling operations in Eq.7.

$W$ is learned to fuse style-normalized features. Through spatial squeezing, the fusion process minimally alters the features, thus preserving essential details and reducing computational load. The standard deviation of the fused features is adjusted using $W$ with $\text{std}(X_V)$ for alignment. By employing max pooling, the most prominent features are accentuated, whereas average pooling offers a comprehensive overview of features. This approach balances the focus on both critical and general details for distribution adjustment, optimizing the integration of infrared and visible features.

7. About Eq.8.

Eq. 8 is used to adjust the distributional characteristics of the style-normalized fused feature to align with the visible feature $\hat{X}_V^L$. The learnable weight $W$ is utilized to channel-wise fuse $SN(X_V)$ and $SN(X_I)$, and then align the fused feature with the variance of the style-normalized $\hat{X}_V^L$. The mean of the style-normalized fused feature is adjusted to match the mean of $\hat{X}_I^L$. This process ensures that the fused output retains crucial information from modalities, optimized for the characteristics of the visible domain. We revise the main paper for more clarity.

8. About entropy-aware \alpha.

An excessively large or small entropy-aware $\alpha$ damages the image priors of the supervision signal. Such constraints make the modal image with smaller variances effectively align with other source images, while ensuring that the adjusted supervision signal tends toward consistency, thereby ensuring the integrity of the supervision signal. We conduct an additional experiment by removing the constraints on $\alpha$. The results show a significant performance decrease on RoadScene dataset, as shown in the following Table 3. This is due to excessive modulation of the source images, which cannot provide effective supervision, thereby demonstrating the necessity of the constraint.

Table 3: Parameters $\alpha$ of adaptive reconstruction loss on RoadScene dataset.

9.Use Max(R(V), R(I)) for similarity loss.

We follow [1,2], using the maximum pixel values as the supervision signal in Eq. 11. As detailed in the supplementary material, Section 4.2, we further analyzed the operations used to form the supervision signals in the loss function. As shown in the following Table 4, replacing the current max operation with a mean operation, or using separate supervisions for $R(I)$ and $R(V)$, both led to decreased performance. The key reason is that when using the mean or separate supervisions, the network tends to blend the two input signals too smoothly. This excessive blending results in a loss of important content details in the fused output. In contrast, the max operation better preserves the distinct content characteristics from the input modalities, preventing this detail loss.

Table 4: Ablation studies of operations in Eq.11 on RoadScene dataset.

[1] Correlation-driven dual-branch feature decomposition for multi-modality image fusion. CVPR, 2023.

[2] Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network. Inf. Fusion, 2022.

10. About (f(V, I)-I) or (f(I, I)-I).

To avoid ambiguity during the fusion process, Our SCFNet prioritizes the domain with more information, the visible domain. The network learns the mapping of the visible image domain. Incorporating identity learning in the visible domain enhances the ability to capture visible features. However, introducing identity learning into the infrared image domain creates confusion during feature fusion, subsequently affecting the clarity of the fused images.

11. Minor comments.

We revise Eq. 9 and correct typos in the main paper.

4. About adaptive reconstruction loss with Eq. 9-11.

(1) Our proposed loss employs a learned linear rescaled function for the source input to form a unified and complete supervision signal. The visualization results in the supplementary Fig.3 demonstrate that the adaptive reconstruction loss function effectively guides the model to learn fusion, forming a supervisory signal that encompasses complete scene information. Models trained under the supervision of our proposed loss function reconstruct efficient fused images, especially for obscured objects, such as individuals concealed by smoke.

(2) The analysis of the $\beta$ in the adaptive reconstruction function shows that setting it to $\text{Mean}(V)$ improves alignment with the visible domain. As shown in the following Table1, this is more effective than the learnable setting or $\text{Mean}(I)$, especially when combined with the style-adjustment fusion (SAF) module.

Table 1: Ablation studies of $\beta$ in Eq.9 on RoadScene dataset.

(3) We follow [1,3], using the maximum pixel values as the supervision signal in Eq. 11. We further analyze the operation used to form the supervision signals in the loss function. As shown in the following Table2, Replacing the current max operation with a mean operation, or using separate supervisions for $R(I)$ and $R(V)$, both led to decreased performance. The key reason is that when using the mean or separate supervisions, the network tends to blend the two input signals too smoothly. This excessive blending results in a loss of important content details in the fused output. In contrast, the max operation better preserves the distinct content characteristics from the source modalities, preventing smoothing.

Table 2: Ablation studies of operations in Eq.11 on RoadScene dataset.

[3] Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network. Inf. Fusion, 2022.

5. About results of TarD and DeF on TNO.

We follow the dataset configurations[1] to train SCFNet and other models with published codes, tuning hyperparameters to obtain promising results. We find TarD with loss function weight $\beta$=0.05 performs better, with results on other results consistent with [1]. Additionally, for DeF trained on the COCO dataset [4], we finetune it on our training dataset, achieving superior performance on the TNO dataset.

[4] Microsoft coco: Common objects in context. ECCV, 2014.

1. The core idea and Eq. 8.

Our core idea for fusion is reflected not only in Eq. 8 but also in SNF, where complete phase fusion of style-normalized multi-modal features is demonstrated in the main paper Eq. 4-6.

More Analysis: Eq. 8 is used to adjust the style-normalized fused feature to align with the visible domain. The learnable $W$ is utilized to channel-wise fuse $SN(X_V)$ and $SN(X_I)$, and then align the fused feature with the $\text{Std}(\hat{X}_V^L)$. The mean of the fused feature is adjusted to match $\text{Mean}(\hat{X}_I^L)$. It ensures the fused feature is optimized for the characteristics of the visible domain. We revise the main paper for more clarity.

Experimental validation:

(1) For feature alignment with a specific domain, we present a visual comparison. Fig. 4 in the supplementary demonstrates that aligning modalities can reduce feature differences between modalities and does not lead to the loss of scene information.

(2) For the feature fusion process, we conduct both visual and quantitative comparisons. As shown in Tab. 4 of the main paper, skipping the alignment and directly fusing results in significantly reduced performance, demonstrating the necessity of alignment for effective fusion. The comparison of visual results in supplementary Fig. 4 shows that fused features with alignment preserve complete scene information without over-smoothing.

(3) For SAF aligning different modalities, we also conduct both visual and quantitative comparisons. The visualization results in supplementary Fig. 4 and the quantitative results in the following Table 1 highlight the advantages of selecting different modal domains for alignment. For IVF, although aligning with the visible domain provides more distinct details and contrast, maintaining excessively high contrast reduces visual performance. Therefore, prioritizing the information-abundant domain helps avoid the destruction of image priors caused by excessive modulation.

Table 1: Ablation studies of the selection of alignment across different domains in SAF on RoadScene dataset.

2. About decomposition approaches.

In Sec. 2 of the main paper, we add related works and provide analysis. CDD[1] employs shared and specific multi-modal feature decomposition, followed by separate fusions of specific and shared features. DIDFuse[3] uses four distinct encoders to extract scenario features and attribute latent representations, and then fuse each set. However, these methods, including DRF[3], do not address differences in feature characteristics, incomplete content complementarity, and ambiguity in the fusion outputs.

Specifically, (1) Decomposition: SCFNet is the first to leverage style learning to decompose into style-specific characteristics and content-invariant representations. (2) Fusion: With style-normalized features, SAF explicitly ensures that the content of the features maintains completeness and effective integration. (3) Alignment: SAF and the proposed loss function constrain the fused features and image domain alignment with a well-defined domain to ensure consistency.

From the results in the main paper, our SCF outperforms the latest decomposition methods, CDD [1] and DeF [2], further demonstrating the effectiveness of our decomposition and fusion approach for MMIF.

[1] Correlation-driven dual-branch feature decomposition for multi-modality image fusion. CVPR, 2023.

[2] Fusion from decomposition: A self-supervised decomposition approach for image fusion. ECCV, 2022.

[3] Disentangled representation for visible and infrared image fusion. IEEE Trans. Instrum. Meas.2021.

3. The design of the FPE and SNF.

To the best of our knowledge, our method is the first to directly address multi-modal inconsistencies based on style-learning theory. Our SNF module emphasizes integrating content and preserving details. To simultaneously handle different image features, our SNF module integrates content by mapping heterogeneous features to a style-independent space. It conducts content fusion and replacement to enhance dual-branch source content information and designs dynamic style adjustments to progressively refine feature expression and enhance details. These capabilities cannot be achieved through existing methods or previous works.

Furthermore, we do not claim FPE as our contribution. FPE enhances the extraction of feature frequencies, which aids in subsequent feature alignment and fusion processes. It is not the contributor to resolving the modality-specific visual characteristic differences

Dear Reviewer Exws,

Thank you once more for your insightful review.

As the author-reviewer discussion period is nearing its end, we would greatly value your feedback on whether our revisions and responses have adequately addressed your earlier concerns.

Should you have any further questions or additional feedback, please let us know, and we will address them promptly.

We appreciate your time and contributions.

Warm regards,

The Authors

More analysis and ablation studies for proposed adaptive reconstruction losses.

More analysis with visualization results and quantitative results on the proposed adaptive reconstruction loss are provided in the supplementary Sec.4. (1) Firstly, the visualization results in the supplementary Fig.3 demonstrate that the adaptive reconstruction loss function effectively guides the model to learn fusion, forming a complete supervisory signal that encompasses scene information. Models trained under the supervision of this proposed loss function can reconstruct efficient fused images, especially for obscured objects, such as individuals concealed by smoke.

(2) The analysis of the $\beta$ in the adaptive reconstruction function shows that setting it to $\text{Mean}(V)$ improves alignment with the visible domain. As shown in the following Table1, this is more effective than the learnable setting or $\text{Mean}(I)$, especially when combined with the style-adjustment fusion (SAF) module.

Table 1: Ablation studies of $\beta$ in Eq.9 on RoadScene dataset.

(3) We further analyze the operation used to form the supervision signals in the loss function. As shown in the following Table2, Replacing the current max operation with a mean operation, or using separate supervisions for $R(I)$ and $R(V)$, both led to decreased performance. The key reason is that when using the mean or separate supervisions, the network tends to blend the two input signals too smoothly. This excessive blending results in a loss of important content details in the fused output. In contrast, the max operation better preserves the distinct content characteristics from the source modalities, preventing smoothing.

Table 2: Ablation studies of operations in Eq.11 on RoadScene dataset.

(4) Additionally, we study the learnable $\alpha$, which is utilized to adjust the entropy of source images. An excessively large or small $\alpha$ damages the image priors of the supervision signal. We conduct an additional experiment by removing the constraints on $\alpha$. The results show a significant performance decrease on RoadScene dataset, as shown in the following Table 3. This is due to excessive modulation of the source images, which cannot provide effective supervision, thereby demonstrating the necessity of the constraint.

Table 3: Parameters $\alpha$ of adaptive reconstruction loss on RoadScene dataset.

1. This method has a preference for one modality in the fusion process, but how to choose the preference for two modalities whose dominance is not clear?

(1) SAF uses well-defined source distribution to guide the different features into a unified domain rather than the direct definition of the target domain. Fig. 7 in the main paper demonstrates that the fusion results do not completely depend on the aligned modality but instead more closely resemble its visual characteristics. It allows SAF to be more robust to variations in the source modalities.

(2) The fused results demonstrate that SAF aligns images of different modalities while preserving complete scene information, with differences in their visual characteristics. Fig. 4 in the supplementary shows that aligning with the infrared domain results in higher contrast, while aligning with the visible domain produces more visually satisfying images. This flexibility in visual characteristics further enhances the robustness of SAF to source modalities.

(3) SAF allows for selecting the alignment domain based on specific requirements. Aligning with the domain containing critical information helps preserve essential original details and avoids excessive modulation. For example, when fusing visible and infrared images for human observation, it is preferable to align with the visible domain. This ensures that the visual information is well represented in the fused images (see Fig. 4 in the supplementary).

2. About equation 8.

For SAF, Eq. 8 is used to adjust the distributional characteristics of the style-normalized fused feature to align with the visible feature $\hat{X}_V^L$. The learnable weight $W$ is utilized to channel-wise fuse $SN(X_V)$ and $SN(X_I)$, and then align the fused feature with the variance of the style-normalized $\hat{X}_V^L$. The mean of the style-normalized fused feature is adjusted to match the mean of $\hat{X}_I^L$. This process ensures that the fused output retains crucial information from modalities, optimized for the characteristics of the visible domain. We revise the main paper for more clarity.

3. More analysis on SAF.

We conduct further analysis on SAF as follows.

(1) For feature alignment with a specific domain, we present a visual comparison. Fig. 4 in the supplementary demonstrates that aligning modalities can reduce feature differences between modalities and does not lead to the loss of scene information.

(2) For the feature fusion process, we conduct both visual and quantitative comparisons. As shown in Tab. 4 of the main paper, skipping the alignment and directly fusing results in significantly reduced performance, demonstrating the necessity of alignment for effective fusion. The comparison of visual results in supplementary Fig. 4 shows that fused features with alignment preserve complete scene information without over-smoothing.

(3) For SAF aligning different modalities, we also conduct both visual and quantitative comparisons. The visualization results in supplementary Fig. 4 and the quantitative results in the following Table 1 highlight the advantages of selecting different modal domains for alignment. For IVF, although aligning with the visible domain provides more distinct details and contrast, maintaining excessively high contrast reduces visual performance. Therefore, prioritizing the information-abundant domain helps avoid the destruction of image priors caused by excessive modulation.

Table 1: Ablation studies of the selection of alignment across different domains in SAF on RoadScene dataset.

4. About proposed adaptive reconstruction loss.

Our proposed adaptive reconstruction loss function employs a learned linear rescaling function for the source input to form a unified and complete supervision signal. In Eq. 9, the learned linear rescaling function uses $\beta$ to align the center of the distribution of the visible image domain, while $\alpha$ aligns the variance of different images to adjust the entropy of the source images.

The visualization results in the supplementary Fig.3 demonstrate that the rescaled source images effectively guides the model to learn fusion, forming a complete supervisory signal that encompasses scene information. Models trained under the supervision of this proposed loss function can reconstruct efficient fused images, especially for obscured objects, such as individuals concealed by smoke.

We also conduct the analysis and ablation studies on the proposed loss and add them to the main paper and supplementary.

(1) We analyze the $\beta$ parameter in the adaptive reconstruction function. Table 2 below shows that setting it to $\text{Mean}(V)$ improves alignment with the visible domain when combined with the style-adjustment fusion (SAF) module. This avoids aligned domain conflicts, making it more stable and effective than the learnable setting or $\text{Mean}(I)$.

Table 2: Ablation studies of $\beta$ in Eq. 9 on RoadScene dataset.

(2) In the following response, we further analyze the $\alpha$ parameter. Table 4 below demonstrates the importance of the constraint in maintaining image priors and forming effective implicit supervision.

(3) We analyze the operation in the loss function. Table 3 below shows that the current $\text{Max}(\cdot)$ operation, compared to separate supervision and the mean operation, helps to obtain clearer supervision signals, thereby achieving promising results.

Table 3: Ablation studies of operations in Eq. 11 on RoadScene dataset.

5. Advantages of using information entropy-based clipping to constrain α.

The learnable parameter $\alpha$ is utilized to adjust the entropy of the image. An excessively large or small $\alpha$ damages the image priors of the supervision signal. Variance is an important indicator of the amount of information entropy. Such constraints make the source image with smaller variances effectively align with other source images. It ensures that the adjusted information entropy of supervision signals tends toward consistency, thereby ensuring the integrity of the supervision signal. We conduct an additional experiment by removing the constraints on $\alpha$. The results show a significant performance decrease on RoadScene dataset, as shown in the following Table 4. This is due to excessive modulation of the source images, which cannot provide effective supervision, thereby demonstrating the necessity of the constraint.

Table 4: Parameters $\alpha$ of adaptive reconstruction loss on RoadScene dataset.