Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model

Q1: Sampling interval and its impact.
Reply: In our method, image restoration and information integration are mutually coupled. This is reflected in the physical connection, where a fusion control module is embedded within the internal structure of the diffusion model. Once all the networks are trained, we can follow the standard diffusion model testing procedure, which involves performing T steps of continuous sampling. It is worth noting that information fusion needs to be performed at each sampling step. In this case, the only factor affecting the final fusion result is the number of sampling steps. More sampling steps mean better performance, but they also result in significant time consumption. Therefore, setting an appropriate number of sampling steps is a matter worth discussing.
In tasks where the ground truth is available, the number of sampling steps can be well determined by checking whether the generated results are sufficiently close to the ground truth. However, for the image fusion task, where ground-truth data do not exist, we rely on visual perception and multiple no-reference metrics to make the assessment. Specifically, we set the number of sampling steps to 2, 3, 4, 5, 10, 25, 500, and 1000, with qualitative and quantitative results shown in Figs. r7 and r8. Notably, each metric is normalized along the step dimension for easier presentation. It can be observed that as the number of steps increases, noise is gradually removed and the scene texture becomes increasingly refined. Corresponding to the quantitative results, 25 steps achieve good performance saturation, with subsequent increases in the number of steps resulting in only slight fluctuations in scores. Note that the only exception is AG, as it is affected by noise during the diffusion process. Therefore, in our experimental section, the number of sampling steps is set to 25.

Q2: Removal of degradation symbols $N$ in Eq. (7).
Reply: There are two conceptual differences that need to be clarified first. In all the equations, the symbol $N$ refers to the degradation from the source images, specifically including improper lighting, color distortion, and random noise. In contrast, the noise in the intermediate results obtained from continuous sampling of the diffusion model arises from the Gaussian noise assumption of the diffusion theory itself. Eqs. (2)-(6) represent the process of encoding, fusion, decoding, and the estimation of mean and variance for degraded multi-modal images. In this process, the objects being processed contain the degradation from source images, so they all include the condition $N$. Differently, Eq. (7) represents the intermediate result obtained after a single complete sampling step. Therefore, even though early sampling steps still contain Gaussian noise following the diffusion assumption (see Fig. r7), it does not need to include the symbol $N$. We will clarify their distinctions in the final version to avoid misunderstandings.

Q3: Concatenate order of image restoration algorithm.
Reply: In Table 2, we introduce three image restoration algorithms as preprocessing steps of other comparative image fusion methods, including CLIP-LIT, SDAP, and AWB. Among them, CLIP-LIT is a low-light enhancement algorithm, SDAP is a denoising algorithm, and AWB is a white balance algorithm. In the experiment, we follow the processing sequence of low-light enhancement first, followed by denoising, and finally white balance. The choice of this sequence is related to the dependencies among the three types of degradations we are focusing on. Specifically, in low-light images, both scene content and degradation present low-intensity properties, and the signal-to-noise ratio is low. The deep entanglement of noise, color distortion, and useful signals makes degradation removal more challenging. Thus, we first use CLIP-LIT to improve exposure, thereby reducing the difficulty of denoising and color correction. Furthermore, color correction based on white balance requires locating the white light source, and noise interferes with the accuracy of finding the light source. Therefore, we then perform SDAP to remove noise. After addressing exposure and noise, AWB is applied last to achieve color correction.

Q4: Verification of text control on the object detection.
Reply: Our method supports text control, enabling the enhancement of the salience of objects of interest based on instructions. Following the reviewer's suggestion, we further verify the semantic gain brought by text modulation on the object detection task. Specifically, the MSRS dataset [r1] is used, which includes pairs of infrared and visible images with two types of detection labels: person and car. Therefore, the text instruction is formulated as: "Please highlight the person and car," which guides our method to enhance the representation of these two types of objects in the fused image. Then, we adopt the YOLO-v5 detector to perform object detection on infrared images, visible images, and fused images generated by various image fusion methods. The visual results are presented in Fig. r6, in which more complete cars and people can be detected from our fused images while showing higher class confidence. Furthermore, we provide quantitative detection results in Table r5. It can be seen that the highest average accuracy is obtained from our fused images, demonstrating the benefits of text modulation. Overall, these results indicate that text control indeed provides significant semantic gains, benefiting downstream tasks.
[r1] PIAFusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 2022.

Q1: Clean data for the loss construction.
Reply: Constructing Eqs. (9) and (10) actually involves very stringent data requirements. Specifically, they require a pair of degraded multi-modal images describing the same scene, along with their corresponding clean versions. Unfortunately, such a dataset is currently not available. To alleviate this challenge, we adopt a two-step strategy. Specifically, we first pre-train the diffusion model to learn the image restoration capability. In this step, we only need degraded-clean image pairs, without the need for paired multi-modal images that describes the same scene. Once the diffusion model is trained, it can be used to process existing degraded multi-modal image fusion datasets, to generate the required clean multi-modal image pairs. At this point, all the data required for constructing Eqs. (9) and (10) has been obtained.

Q2: Source of the constrained fused image in Eq. (8).
Reply: Our method couples image restoration and information integration by inserting a fusion control module within the diffusion model. In other words, each step of sampling will be accompanied by an information fusion. Theoretically, the final fused image requires T steps of continuous iterative sampling to obtain. However, during training, it is inefficient to wait for T steps of sampling to obtain the result and then construct the loss functions of Eqs. (9) and (10). Therefore, we customize Eq. (8) based on the diffusion relationship. According to it, we can derive the corresponding fake final fused image from the results of any step of sampling and apply the corresponding fusion constraints.

Q3: Omission of degradation $N$ in Eq. (7).
Reply: Indeed, diffusion models often require a certain number of sampling steps to progressively remove the noise. However, it is needed to clarify that the degradation symbol $N$ in Eqs. (2)-(6) is not the same thing as the noise in the diffusion process. Specifically, the symbol $N$ refers to the degradation from the source images, specifically including improper lighting, color distortion, and random noise. They are used as the conditions of the diffusion model, included in the source image and fed into the denoising network. In contrast, the noise in the intermediate results obtained from continuous sampling of the diffusion model arises from the Gaussian noise assumption of the diffusion theory itself. Eqs. (2)-(6) represent the process of encoding, fusion, decoding, and the estimation of mean and variance for degraded multi-modal images. In this process, the objects being processed contain the degradation from source images, so they all include the condition $N$. Differently, Eq. (7) represents the intermediate result obtained after a single complete sampling step. Therefore, even though early sampling steps still contain Gaussian noise following the diffusion assumption (see Fig. r7), it does not need to include the symbol $N$. We will clarify their distinctions in the final version to avoid misunderstandings.

Q4: Generalization of semantic attributes.
Reply: Our method supports the use of textual instructions to re-modulate the information fusion process. Its purpose is to increase the salience of the object of interest, and enhance its presentation quality on the fused image. In our method, no specific downstream task is used to guide this remodulation process, so the gain in semantic attributes is not fixed to any particular task. In other words, beyond the semantic segmentation validation presented in the main text, the semantic attribute gain achieved through textual remodulation can certainly be generalized to other downstream tasks.
For proving this point, we implement application experiments on the object detection task. Specifically, the MSRS dataset [r1] is used, which includes pairs of infrared and visible images with two types of detection labels: person and car. Therefore, the text instruction is formulated as: "Please highlight the person and car," which guides our method to enhance the representation of these two types of objects in the fused image. Then, we adopt the YOLO-v5 detector to perform object detection on infrared images, visible images, and fused images generated by various image fusion methods. The visual results are presented in Fig. r6, in which more complete cars and people can be detected from our fused images while showing higher class confidence. Furthermore, we provide quantitative detection results in Table r5. It can be seen that the highest average accuracy is obtained from our fused images, demonstrating the benefits of text modulation. Overall, these results indicate that text control indeed provides significant semantic gains, benefiting downstream tasks.
[r1] PIAFusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 2022.

Q5: Typos.
Reply: Thanks for pointing out these issues. We will carefully correct all typos in the final version and further enhance the presentation of the figures and tables.

Q1: Dataset for training diffusion model.
Reply: In our work, acquiring image restoration capability depends on pre-training a conditional diffusion model, which needs paired clean and degraded data. The clean data are used to build the loss function for supervision, while the degraded data act as conditioning inputs for the denoising network. Therefore, we use existing supervised datasets and additionally simulate a portion of the data to meet the requirements of mixed degradation.
Our method primarily addresses three common types of degradation in the fusion scenario: improper lighting, color distortion, and noise. For improper lighting, we use 2,220 image pairs from the MIT-Adobe FiveK Dataset [r1], covering images with varying exposures and their corresponding ground truth manually adjusted by photography experts. For color distortion, we use 1,031 image pairs from the Rendered WB dataset [r2], including color-biased images under various light sources such as fluorescent, incandescent, and daylight, as well as corresponding reference images manually calibrated under the Adobe standard. For noise, we add Gaussian noise, pulse noise, Poisson noise, Rayleigh noise, and uniform noise to 2,220 clean images from the MIT-Adobe FiveK Dataset and 2,220 clean images from the MSRS dataset to obtain noised images. All these image pairs constitute the complete dataset for training our diffusion model, driving our model’s learning for compound degradation removal.
[r1] Learning photographic global tonal adjustment with a database of input/output image pairs. CVPR 2011.
[r2] When color constancy goes wrong: Correcting improperly white-balanced images. CVPR, 2019.
[r3] PIAFusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 2022.\

Q2: Expand application scenarios.
Reply: This is an insightful suggestion. Of course, our proposed method can be further generalized to other multi-modal image fusion scenarios, as its methodology is a general fusion paradigm. To prove this, we conduct comparative experiments in the near-infrared and visible image fusion scenario. The visual results are shown in Fig. r5, where our method effectively integrates texture details from the near-infrared band with those from the visible image, while preserving natural color attributes of the visible image. Notably, the inherent image restoration capability of our method allows it to produce vivid fused images in underexposed scenes without causing overexposure like MRFS, as seen in the results of the first row. Furthermore, the quantitative results in Table r4 show that our proposed method ranks first in three of all five metrics and second in the other two. Overall, our method can be generalized to near-infrared and visible image fusion scenario with promising performance.

Q3: Quantitative evaluation of polarization image fusion.
Reply: We conduct a quantitative assessment of the polarization image fusion, as reported in Table r4. Our method achieves the best scores in four of all five metrics, including EN, AG, SD, and SCD. These results indicate that our fused image contains the most information, the richest texture, the best contrast, and the most feature transfer from the source images. Overall, the quantitative results validate the advantages of our method in the polarization image fusion scenario, demonstrating its strong multi-modal fusion generalization capability.

Q4: Brightness-chrominance separation.
Reply: Image fusion requires a high level of color fidelity to the scene. Taking the infrared and visible image fusion as an example, the colors in the fused image are required to be as consistent as possible with those in the visible image. Therefore, by independently purifying and preserving the chrominance components in the visible image, our method can effectively and conveniently achieve color fidelity.
Next, we discuss why our method does not directly process three-channel images. Firstly, from the perspective of image restoration alone, directly processing color images is entirely feasible. However, our method requires embedding information fusion into the latent layers of the diffusion model used for image restoration. This means that features from the gray infrared image could potentially interfere with the color distribution of features from the visible image. In particular, this interference occurs in the highly nonlinear latent space, where some small changes can be amplified by the decoder to produce large color distortions. In this case, ensuring the expected color fidelity is very difficult. Second, the interference is directly related to the way multi-modal features are fused. In our method, we use a nonlinear neural network called the Fusion Control Module to perform information aggregation, which is guided to retain significant thermal radiation objects while preserving rich background textures. These two goals correspond to the similarity loss functions (see Eqs. (9) and (10)) based on the indicators of pixel intensity and gradient. Under such optimization guidance, it is difficult to avoid disrupting the color distribution in the features from the visible image. For verifying, we adapt our proposed method to directly process three-channel images without separating brightness and chrominance components, and the results are presented in Fig. r1. Clearly, color distortion occurs. Furthermore, we implement quantitative evaluation in Table r1. The direct processing strategy decreases the color score CIECAM16 and also negatively affects other metrics to varying degrees

Q5: Typos and underline on $AG$'s second place.
Reply: Thanks for pointing out these issues. We will carefully correct all typos in the final version and further improve the presentation of the figures and tables.

Q1: Discussion on the necessity of the brightness-chrominance separation.
Reply: Unlike the image generation task emphasizing diversity, image fusion demands high color fidelity. For instance, in infrared and visible image fusion, the fused image should closely match the colors of the visible image. To this end, our method independently purifies and preserves the chrominance components of the visible image for color fidelity.
While a direct processing approach without separating brightness and chrominance is appealing, it faces several challenges in our work. First, from image restoration alone, directly processing color images by the diffusion model is entirely feasible. However, our method integrates information fusion into the latent layers of the diffusion model, which can cause gray infrared features to interfere with the visible image’s color distribution. Such interference in the nonlinear latent space can cause the decoder to amplify small changes into significant color distortions. Second, the interference is related to the feature fusion way. Our method uses a nonlinear neural network, the Fusion Control Module, to aggregate features. This module preserves key thermal objects and background textures, guided by similarity loss functions based on pixel intensity and gradient (see Eqs. (9) and (10)). This optimization can disrupt color distribution due to direct pixel changes. To verify, we adapt our method to process three-channel images without separating brightness and chrominance, as shown in Fig. r1. This strategy causes noticeable color distortion. We also conduct a quantitative evaluation. Table r1 reveals that the direct processing strategy lowers the color score CIECAM16 and negatively impacts other metrics to varying extents.

Q2: InstructIR plus MaxFusion.
Reply: MaxFusion is designed for conditional image generation, requiring results to meet conditions like depth, skeleton, segmentation, and edges. Its strength lies in extending single-condition to multi-condition generation through feature fusion. However, MaxFusion is unsuitable for the image fusion task due to lower fidelity. Specifically, image fusion needs the fused image to maintain pixel-level consistency with multiple source images, preserving significant objects and sharp textures. In contrast, MaxFusion focuses on semantic consistency, producing diverse results with different styles. In Fig. r2, using MaxFusion for infrared and visible image fusion with infrared and visible images as conditions shows that it does not meet the goal of enhancing scene representation in image fusion.
Thus, we conduct comparisons using the reviewer-recommended InstructIR followed by several advanced image fusion methods. First, we input different text prompts into InstructIR to address improper lighting, noise, and color distortion. The restored images are then fused with advanced fusion methods. Results are shown in Fig. r3 and Table r2. Our method which implicitly integrates image restoration and fusion, shows better performance than these methods following a sequential strategy. In particular, our method can balance thermally salient object retaining and degradation removal, while competitors cannot.

Q3: No-training approach.
Reply: MaxFusion extends single-modal image generation models to multi-modal ones through feature-level fusion. Specifically, MaxFusion proposes a no-training fusion strategy, which uses the variance maps of the intermediate feature maps of the diffusion model to select or weighted sum multi-modal features, assuming that pixels with higher variance represent higher priority for condition control.
Differently, our method uses a neural network, the fusion control module (FCM), which is trained under the constraints of Eq. (11) to fuse multi-modal features aiming at preserving significant objects and textures. Of course, our method can also be extended to a non-training version by using a statistics-based fusion strategy. In the ablation study, we have evaluated three no-training fusion strategies: maximum, addition, and mean (see Table 5). These methods performed worse than our learnable FCM. Furthermore, we incorporate MaxFusion’s variance-based fusion strategy in our method (see Fig. r4 and Table r3), and it still falls short compared to our FCM. In future research, we will explore more powerful no-training fusion strategies, achieving performance comparable to retraining.

Q4: Discussion on the competitors.
Reply: Due to limited space, we discuss only a few newer competitors. TarDAL uses a GAN with dual discriminators to preserve objects and textures, and collaborates with object detection for semantic optimization. DeFusion employs a masked autoencoder to decouple unique and common features, achieving complementary feature aggregation. LRRNet introduces low-rank representation model-based networks and combines pixel-level and feature-level losses for multi-modal fusion. MRFS couples image fusion and segmentation at the feature level to improve fused image quality. However, these methods do not fully address composite degradation, resulting in lower robustness. Given the diffusion model’s strong generative capabilities, applying it to image fusion could address composite degradation issues, but the lack of clean references complicates this. DDFM uses source images to guide sampling direction for implicit fusion, but still cannot solve composite degradation due to the inability to retrain the diffusion model. Our method tackles this by first optimizing the diffusion model and then embedding a fusion module within it, achieving integration of image restoration and fusion. Additionally, our method includes a text control interface for further modulation and enhancement of objects. These designs make our method both robust and flexible.

Q4: Figures, tables, and typos.
Reply: We will carefully correct all typos in the final version and further improve the presentation of the figures and tables.