Reviewer cEzg

Thank you for your feedback.

A1: Regarding the typo in Equation (7) and the inconsistency with $\mathcal{L}_{\text {SSIM }}$ on line 179, we apologize for the oversight. We have revised line 179 to ensure consistency with the description in Equation (7).

A2: The dashed arrow in Figure 1 represents Dashed arrows indicating cutting off gradient flow. We have clarified this in the figure caption for better understanding.

A3: In the study of branches in the Object-Region-Pixel Phylogenetic Tree, we analyzed the performance degradation under settings 0, 1, 2, 3, and 4, and provided visual evidence. The fusion performance begins to decline after adding the fourth setting primarily due to the increased complexity and interaction among multiple branches. As more branches are added, the network may struggle to effectively balance and integrate pixel-level and object-level information, leading to a gradual decline in fusion performance.

A4: Regarding Figure 4, we acknowledge your suggestion to enlarge the highlighted areas where targets are circled. We have enhanced the clarity of these areas in Figure 4 to provide a clearer representation of the targets, similar to other figures in the manuscript.

Reviewer 6qWk

Thank you for your constructive insights.

A1: We acknowledge your point regarding comparing YOLOv5s with the latest version of YOLO. We chose YOLOv5s as it represents a well-established baseline in the field, and our focus was on evaluating the results of the detection accuracy of different fusion methods under the same detector. It is consistent with papers "CVPR2023 MetaFusion: Infrared and Visible Image Fusion via Meta-Feature Embedding from Object Detection" and "CVPR2022 Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection. However, we will consider including a comparison with the latest YOLO version in future work to provide a more comprehensive evaluation.

A2: Thank you for pointing out this omission. We apologize for the oversight. In Figure 1, the three backbone networks represent the same backbone. The parameters included are defined as the shared parameters for MF and OD networks. We will clarify this in the revised manuscript to ensure that other researchers have the necessary details for replication and comparison purposes.

A3: Thank you for your corrections; we have made the necessary revisions.

Reviewer G1Ro

Thank you for your feedback.

A1: Reviewer 6qWk and reviewer cEzg both affirmed the novelty of our paper. As reviewer cEzg commented, "The idea of simultaneously learning image fusion (MF) and object detection (OD) tasks to mutually benefit each other is intriguing and reasonable." It is important to note that the contribution of this paper lies in introducing the first end-to-end joint training paradigm in the fusion detection field. In current research, joint learning algorithms are an emerging hotspot, leveraging fusion networks and object detection synergies to enhance image informativeness and improve detection performance. However, existing optimization methods are typically non-end-to-end, relying on multiple steps, as shown in Figure 1. These approaches are cumbersome and reduce training efficiency. Therefore, we propose the first end-to-end synchronous joint optimization algorithm, E2E-MFD, which facilitates interaction between intrinsic features from both domains through synchronous joint optimization, enabling a streamlined, one-stage process.

Moreover, our end-to-end solution does not merely concatenate two tasks (modules) but involves dedicated module design. To harmonize fine-grained details with semantic information, we introduce the novel concept of an Object-Region-Pixel Phylogenetic Tree (ORPPT) coupled with a coarse-to-fine diffusion processing (CFDP) mechanism. Additionally, we find that the MF and OD tasks have naturally distinct optimization objectives. MF primarily focuses on capturing pixel-level relationships between image pairs, while OD integrates object semantics within diverse scene contexts. Therefore, there exists an inherent optimization barrier between these two tasks. To address this, we especially introduce the concept of gradient alignment in the multi-task learning field, proposing GMTA to align gradients for optimizing task dominance and resolving conflicting gradients in the end-to-end joint training process. As you mentioned, we ultimately propose a novel approach to learning image fusion and objection detection synchronously and jointly. We are the first to innovatively integrate image fusion and object detection into a single-stage, end-to-end framework, achieving SOTA results on multiple datasets.

A2: As pointed out by reviewer 6qWk, “This paper presents the first attempt to achieve simultaneous single-stage training of image fusion and object detection, and the results appear very promising. Inspired by multitask learning, the GMTA method is introduced to reasonably balance the loss functions, thereby converging the fusion detection weights to optimal points.” As discussed in A1, previous SOTA methods relied on non-end-to-end optimization approaches, dividing joint training into multiple steps, which led to complexity during training. These methods excessively emphasized leveraging OD information for MF enhancement, complicating parameter balancing and making them susceptible to local optima of individual tasks. Therefore, achieving a unified feature set that simultaneously satisfies the characteristics of each task through end-to-end training remains a formidable challenge. This paper introduces E2E-MFD, an end-to-end multimodal fusion detection algorithm. E2E-MFD aims to seamlessly integrate detailed image fusion and object detection from coarse to fine levels. We introduce the gradient alignment concept from the multi-task learning domain, aiming to eliminate conflicting gradients between object detection and multimodal fusion tasks through the design of GMTA optimization mode. By facilitating synchronous joint optimization and fostering interaction between intrinsic features from both tasks, E2E-MFD achieves a streamlined single-stage process in an end-to-end manner. In addition, reviewers bix9, 6qWk, and cEzg both acknowledged our writing presentation. We will strive to improve our writing skills to the best of our ability. Please help us identify which diagrams have caused you confusion. We will make every effort to revise the diagrams and provide comprehensive explanations.

A3: Our goal is to explore an end-to-end joint training approach. Thank you for recognizing the novelty of the design of node1 and node2. In this framework, Node 1 serves the object detection task, while Node 2 acts as a module for MF tasks, with personalized settings harnessing the respective roles of the nodes. As you mentioned, one of our primary contributions is introducing an end-to-end synchronous training paradigm for multimodal fusion detection, where synchronized joint optimization allows both tasks to complement each other synergistically. This collaboration enables MF to generate richer, more informative images, enhancing the performance of OD, which in turn provides valuable semantic insights to MF for accurate localization and identification of objects within scenes. However, it is crucial to address the issue of gradient conflicts in the joint optimization of fusion and detection tasks, known as task consistency. This challenge, though a common concern in multitask learning, is effectively tackled for the first time in the realm of multimodal fusion detection through end-to-end training. As reviewer cEzg noted, “An end-to-end fusion detection algorithm is proposed, effectively avoiding the local optimum problem encountered in multi-stage training models.”

Additionally, it should be noted that the training approach of Metafusion is not end-to-end. Our algorithm essentially represents an advanced end-to-end version of “metafusion,” (i.e., a non-end-to-end image fusion method) taken to its ultimate level of refinement.

Reviewer bix9

Thanks for your comments.

A1: This is a common default setting in the field such as "CVPR23 MetaFusion". V (brightness) channel can effectively measure the algorithm’s ability to handle low-light environments.

A2: We have made revision in Tab. 2. It's common to observe fluctuations in detection accuracy within a single category on M3FD dataset in Tab. R.1. Despite fluctuations, our algorithm significantly outperforms others in $\text{mAP}$.

A3: We utilize widely recognized datasets with diverse tasks, sufficient quantity, and complex environment. TNO and Roadscene are evaluation datasets for MF. TNO captures multispectral images day and night, while Roadscene comprises aligned image pairs from road scenes with vehicles and pedestrians. We also enrich experimental validation by integrating horizontal and oriented OD datasets. The M3FD dataset covers complex scenarios with diverse camera angles. Additionally, DroneVehicle dataset is considered for oriented OD with diverse scenes from an aerial viewpoint.

A4: The process of GMTA is illustrated in Section 3.4 in the paper and we detailed this section to eliminate the confusion. GMTA mitigates the undesired effects of the optimization barrier in task-shared parameters which are supposed to be balanced between the MF and OD tasks. Conflicts in the gradient matrix $\boldsymbol{G}$ are related to a negative cosine distance between gradient vectors ($<\boldsymbol{g}_u,\boldsymbol{g}_d><0$), while dominance is caused by an imbalance of their magnitudes ($\Vert \boldsymbol{g}_u \Vert \gg \Vert \boldsymbol{g}_d \Vert$ or $\Vert \boldsymbol{g}_u \Vert \ll \Vert \boldsymbol{g}_d \Vert$).

Learning from the Aligned-MTL (multi-task learning), the condition number is optimal ($\kappa(\boldsymbol{G})=1$) if and only if the gradients are orthogonal and equal in magnitude which means that the system of gradients has no dominance or conflicts: $\kappa(\boldsymbol{G})=1 \iff <\boldsymbol{g}_u,\boldsymbol{g}_d>=1.$

The final gradient linear system $\hat{\boldsymbol{G}}$ satisfies the optimal condition by the condition number. Thus, the feature learning constraint $\mathcal{S}(\boldsymbol{ \theta}^{\star})$ can be defined as the following optimization to eliminate training instability: $\min _{\hat{\boldsymbol{G}}}\|\boldsymbol{G}-\hat{\boldsymbol{G}}\|_F^2 \quad \text { s.t. }\kappa(\hat{\boldsymbol{G}})=1 \iff \min _{\hat{\boldsymbol{G}}}\|\boldsymbol{G}-\hat{\boldsymbol{G}}\|_F^2 \quad \text { s.t. }\hat{\boldsymbol{G}}^{\top} \hat{\boldsymbol{G}}=\boldsymbol{I}.$

The problem can be treated as a Procrustes problem and can be solved by performing a singular value decomposition (SVD) to $\boldsymbol{G}$ ($\boldsymbol{G}=\boldsymbol{U} \boldsymbol{\Sigma} \boldsymbol{V}^\top$) and rescaling singular values corresponding to principal components so that they are equal to the smallest singular value:

where, $(\boldsymbol{V}, \boldsymbol{\lambda})= eigh(\boldsymbol{G}^{\top}\boldsymbol{G}),$ $\boldsymbol{\Sigma}^{-1} = diag(\sqrt{1/\lambda_{max}},\sqrt{1/\lambda_{min}}).$ $eigh$ finds eigenvectors $\boldsymbol{V}$ and eigenvalues $\boldsymbol{\lambda}$, with $diag$ representing a diagonal matrix. $\lambda_{max}$ and $\lambda_{min}$ are the maximum and minimum eigenvalues of $\boldsymbol{\lambda}$. The stability criterion, defined by the condition number, sets a linear system to an arbitrary position scale. To resolve this ambiguity, we select the largest scale ensuring convergence to the optimum, corresponding to the minimal singular value of the initial gradient matrix: $\sigma=\sigma_{\min }(\boldsymbol{G})=\sqrt{\lambda_{min}}.$

A5: The GMTA process operates during the computation and updating stages of two gradients, approximately every $n$ iteration (gradient update), to balance the independence and coherence of various tasks. Tab. R.2 presents the ablation analysis of the $n$ parameter. Detail analysis is in Experiment Results part of Global Author Rebuttal.

A6: We have added ablation experiments on CFDP, involving whether to use CFDP and the number of proposed boxes, as shown in Tab. R.3. Detail analysis is in Experiment Results part of Global Author Rebuttal.

A7: The paper designs the first end-to-end joint training paradigm in MFD. E2E-MFD enhances interaction between intrinsic features through synchronous joint optimization, streamlining the process. Our solution goes beyond task concatenation, incorporating dedicated module design. Inspired by a phylogenetic tree analogy, we employ an ORPPT to extract features across multiple region scales. By utilizing PFMM and $L$ RFRM branches, we capture various granularities from coarse to fine. Tab. 5 validates our approach, while Fig. 6 underscores the importance of effectively integrating pixel-level and object-level details.

MF primarily focuses on pixel-level relationships between image pairs, while OD integrates object semantics within diverse scene contexts. This inherent optimization barrier between the tasks necessitates a solution. We propose GMTA, a gradient alignment concept within the multi-task learning framework, to optimize task dominance and resolve conflicting gradients in end-to-end joint training. Results in Tab. 4 demonstrate that GMTA, with shared weights, yields superior performance. Comparison with other methods in Tab. 6 further validates our approach. Fig. 5 illustrates how GMTA balances shared parameters between MF and OD, effectively mitigating gradient dominance and conflict.

A8: In Tab. R.4, three recent fusion SOTA methods (CVPR2024 SHIP, PR2024 CFNet, and PR2024 DSFusion) and three evaluation metrics (Qabf, PSNR, and SSIM) are incorporated to valid effectiveness of our method. Detailed analysis is illustrated in Author Rebuttal Experiment Results.