A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking

We sincerely thank the reviewer for the positive evaluation and appreciation of our VF-Bench benchmark, UniVF framework, and the proposed evaluation metrics. Below, we respond to each of the concerns raised in detail.

1. Q1: Fused video comparisons are missing
   R1: Please kindly refer to the supplementary material, where we provide fused video comparisons to more thoroughly demonstrate the effectiveness of our method.

2. Q2: A more intuitive explanation for MS2R
   R2:

   • MS2R evaluates how smoothly objects move over time by measuring acceleration consistency.
   • It compares the second-order motion (flow difference) of the fused video with those from the two reference input videos.
   • A lower MS2R implies more natural, consistent motion, which correlates with fewer flickering or jittering artifacts. We will include a brief explanation in the revised manuscript to make this clearer.

3. Q3: Analysis on the number of Restormer blocks
   R3: We included a sensitivity analysis varying the number of Restormer blocks (e.g., 1, 2, ..., 7) to show how depth affects fusion performance in the table below. 4 blocks offered a good trade-off between quality and inference efficiency. Increasing the depth gave diminishing returns and higher latency.

   Num	VIF	SSIM	MI	Qabf	BiSWE	MS2R
   1	0.35	0.58	2.01	0.55	4.39	0.39
   2	0.40	0.62	2.29	0.63	4.11	0.36
   3	0.44	0.64	2.46	0.67	3.87	0.37
   4 (Ours)	0.44	0.64	2.47	0.68	3.84	0.35
   5	0.44	0.64	2.47	0.68	3.85	0.35
   6	0.45	0.64	2.48	0.68	3.84	0.34
   7	0.45	0.65	2.49	0.69	3.85	0.35

4. Q4: Ablation: Warping features vs. warping images
   R4:

   • Theoretically, our design choice to warp features (rather than input images) allows alignment at a semantically rich representation level, helping the model better deal with illumination or modality shifts.

   • Nonetheless, we perform the suggested ablation: warping input frames directly and feeding them into the encoder vs. our default feature warping to clarify where alignment is most effective.

     	VIF	SSIM	MI	Qabf	BiSWE	MS2R
     Warping images	0.41	0.63	2.45	0.67	3.97	0.36
     Warping features (Ours)	0.44	0.64	2.47	0.68	3.84	0.35

     The ablation results validate the validity of our algorithm design.

5. Q5: inference time, model size, or real-time feasibility
   R5: Per request, here we compare UniVF with 17 SOTA methods in four video fusion tasks in terms of the number of parameters, inference time, and FLOPs, and present the results in the table R4 of Response for Reviewer QQ99. All experiments were conducted in a uniform environment. Note that although UniVF provides multi-frame learning for temporally coherent video fusion, UniVF is near the median in terms of time, complexity and parameter number among the competitors, achieving excellent fusion results without excessive computational resources.

6. Q6: RCVS [1] missing in comparisons
   R6: We present additional comparison results with RCVS in the table below. Our method continues to achieve state-of-the-art performance in video fusion quality.

   	VIF	SSIM	MI	Qabf	BiSWE	MS2R
   RCVS	0.26	0.51	1.28	0.33	4.71	0.37
   Ours	0.44	0.64	2.47	0.68	3.84	0.35

Thank you for acknowledging our paper's idea and contribution. We are very encouraged by the comment for a reasonably well-designed method with clear motivations, architecture is sound, benchmark is timely, and loss and evaluation metrics are novel and reasonable.

1. Q1: DCINN results in IVF task.
   R1: We present additional comparison results with DCINN in the table below. Our method continues to achieve state-of-the-art performance in video fusion quality.

   	VIF	SSIM	MI	Qabf	BiSWE	MS2R
   DCINN	0.38	0.63	1.95	0.58	4.23	0.37
   Ours	0.44	0.64	2.47	0.68	3.84	0.35

2. Q2: Reliance on SEA-RAFT and ablation with other optical flow methods
   R2:

   • We selected SEA-RAFT for its SOTA power in flow estimation and its strong balance of accuracy and efficiency. To further isolate the contribution of UniVF from the flow estimator, we added an ablation study replacing SEA-RAFT with Unimatch [1] in the table in R4 of Response for Reviewer Wqk9.
   • While higher-quality flow leads to slight performance gains, UniVF consistently outperforms other fusion baselines across different flow estimators. This indicates that our framework is robust to the quality of the flow input and does not rely heavily on the accuracy of any specific flow method, thus confirming the value of our multi-frame fusion architecture.
     [1] Unifying Flow, Stereo and Depth Estimation. IEEE TPAMI 2023.

3. Q3: Explanation for validity mask M in Eq. (7).
   R3: To improve the robustness of the temporal consistency loss and avoid penalizing unreliable regions, we employ occlusion masks based on forward-backward optical flow consistency. Specifically, given the forward flow $\mathcal{O}\_{t \rightarrow t+1}$ and the backward flow $\mathcal{O}\_{t+1 \rightarrow t}$, we first warp the backward flow to the coordinate system of the current frame $t$: $\widehat{\mathcal{O}}\_{t+1 \rightarrow t}(p) = \mathcal{W}\left( \mathcal{O}\_{t+1 \rightarrow t}, \mathcal{O}\_{t \rightarrow t+1}(p) \right).$ Here, the warp operator $\mathcal{W}$ uses the forward flow $\mathcal{O}\_{t+1 \rightarrow t}$ to look up the corresponding backward flow vector from frame $t+1$. In essence, for a pixel $p$ in frame $t$, we find where it lands in frame $t+1$ and retrieve the backward flow vector from that new location. The forward-backward consistency error is then defined as the magnitude of the “round-trip” error vector:

   $$\Delta(p) = \left\| \mathcal{O}\_{t \rightarrow t+1}(p) + \widehat{\mathcal{O}}\_{t+1 \rightarrow t}(p) \right\|\_2.$$

A large value of $\Delta(p)$ indicates a significant inconsistency, which typically occurs when the pixel $p$ is occluded in frame $t+1$ or the flow estimation is unreliable. Therefore, we can identify these unreliable points by thresholding this error. We define the binary occlusion mask $M(p) \in \\{0,1\\}$ as: $M(p) = 1 \text{ if } \Delta(p) < \epsilon, \text{ otherwise } 0$ where $\epsilon$ is a predefined threshold (1.0 in our paper). This mask is applied to both previous and next frames (i.e., $M_{\text{prev}}$, $M_{\text{next}}$) to restrict the temporal consistency loss (Eq. 7) to only well-aligned and non-occluded regions.

3. Q4: Inference speed and model size.
   R4: Per request, here we compare UniVF with 17 SOTA methods in four video fusion tasks in terms of the number of parameters, inference time, and FLOPs, and present the results in the table below. All experiments were conducted in a uniform environment. Note that although UniVF provides multi-frame learning for temporally coherent video fusion, it is near the median in terms of time, complexity and parameter number among the competitors, achieving excellent fusion results without excessive computational resources.

   	Params (M)	FLOPs (G)	Time (s)
   CDDFuse	1.19	547.74	0.18
   EMMA	1.52	41.54	0.02
   ReFusion	0.06	18.21	0.03
   BSAF	9.69	62.96	0.03
   FILM	0.49	180.00	0.08
   TDF	0.06	18.21	0.02
   CUN	0.64	195.88	0.08
   DDMEF	20.57	226.28	0.01
   HOLOCO	17.69	183.93	0.10
   IFCNN	0.08	40.05	0.01
   EPT	15.86	423.16	3.19
   DCEvo	2.00	1825.33	0.25
   TextDIF	119.46	17891.10	15.31
   CRMEF	1.88	471.92	0.04
   MRFS	134.97	139.19	0.05
   TCMOA	340.35	8388.47	0.68
   RFL	1.90	2330.47	0.08
   UniVF (Ours)	9.15	825.20	0.21

4. Q5: More frames are used.
   R5: We added experiments evaluating "5-frame input" configurations in the table in R3 of Response for Reviewer Wqk9. From the comparison results, we observed that while larger windows improve long-term temporal consistency, they also introduce alignment drift and increase inference latency. In other words, increasing the window size to 5 brought only marginal improvements in temporal metrics, but negatively affected spatial metrics and significantly increased the model’s inference time. Therefore, considering the trade-off between temporal consistency, spatial information, and inference efficiency, we decided to use a temporal window of 3 frames, i.e., the current frame as well as the previous and subsequent frames.

5. Q6: Notation mapping to equations
   R6: We appreciate this suggestion. To improve clarity, we will revise Fig. 2 to explicitly annotate components with corresponding symbols from the equations in the revision of our paper.

We appreciate the reviewer’s follow-up question regarding the validity mask $M_{\text{prev}}$ and $M_{\text{next}}$ used in our temporal consistency loss (Eq. (7)). Below, we provide a detailed explanation of their motivation, computation process, and intuition.

• The goal of the validity mask is to identify well-aligned, non-occluded regions between adjacent frames, ensuring that the temporal consistency loss only penalizes meaningful discrepancies in stable areas of the video. Otherwise, occluded regions or areas with large flow errors may introduce misleading gradients during training.
• To achieve this, we adopt a standard forward-backward consistency check. Specifically, we set:
  • $\mathcal{O}\_{t \rightarrow t+1}(p)$: the forward optical flow from frame $t$ to $t+1$, at pixel $p$;
  • $\mathcal{O}\_{t+1 \rightarrow t}(q)$: the backward optical flow from frame $t+1$ to $t$, defined over pixels $q$ in frame $t+1$.
• To compare the flows at the same spatial location, we must warp the backward flow to frame $t$. This is necessary because the backward flow is defined on the pixel grid of frame $t+1$, while the forward flow originates from frame $t$. We compute:

where $\mathcal{W}$ denotes the bilinear warping operation that samples the backward flow at the non-integer position $q = p + \mathcal{O}\_{t \rightarrow t+1}(p).$ This warping step is critical because it allows both forward and backward flows to be compared at the same spatial location—namely, the original position $p$ in frame $t$. We then define the forward-backward consistency error as $\Delta(p) = \left\| \mathcal{O}\_{t \rightarrow t+1}(p) + \widehat{\mathcal{O}}\_{t+1 \rightarrow t}(p) \right\|_2$ and if this round-trip motion error is small, we consider the flow at pixel $p$ to be reliable.

• Intuitively, this operation checks whether a pixel can “go forward and come back” along the flow paths with minimal drift. If the two flows disagree significantly, the pixel is likely located in an occluded area, near motion boundaries, or in a region with ambiguous flow. Warping is essential in this context because the forward and backward flows are defined in different coordinate spaces—thus, to conduct a valid consistency check, they must first be spatially aligned.

We thank the reviewer for the encouraging comments on both VF-Bench a valuable contribution and the UniVF framework well-motivated and coherent and effective solution for video fusion.

1. Q1: Explanation of "feature warping"
   R1: Feature warping in UniVF is implemented via bilinear sampling, following the common practice in optical flow-based temporal modeling. Specifically, for each input image, we estimate the bidirectional optical flows between adjacent frames, denoted as $\\{\mathcal{O}\_{t-1 \rightarrow t}^{k}, \mathcal{O}\_{t+1 \rightarrow t}^{k}\\}$. Each optical flow field represents the motion of pixels from one frame to another, where each flow vector indicates the displacement of a pixel to its corresponding location in the neighboring frame. These flow fields are then applied to the deep features $\\{\Phi\_{t-1}^{k},\Phi\_{t+1}^{k}\\}$ extracted by the encoder, enabling us to align them with the target frame $t$. The features are sampled at subpixel locations using differentiable bilinear interpolation to produce the warped representations $\\{\widetilde{\Phi}\_{t-1 \rightarrow t}^{k},\widetilde{\Phi}\_{t+1 \rightarrow t}^{k}\\}$, which are then fused for final prediction.

2. Q2: Are the encoder weights shared or independent?
   R2: The two encoders processing the input video streams are independent.

   • Theoretically, when we have inputs from different modalities, we need separate encoders to extract their respective private features.

   • In practice, we demonstrate the effectiveness of using two shared encoders through ablation studies. Following ablation experiments results in Table 4 from the original paper, the IVF task results are shown in the table below.

     	VIF	SSIM	MI	Qabf	BiSWE	MS2R
     Shared Encoder	0.43	0.64	2.43	0.65	3.94	0.36
     Independent Encoder (Ours)	0.44	0.64	2.47	0.68	3.84	0.35

     The ablation results validate the soundness of our algorithm design.

   • Notably, for video frames from the same input source, the same set of encoder parameters is used to encode each frame.

3. Q3: Window size chosen for Temporal loss
   R3: Our current temporal loss leverages the two adjacent frames $(t-1,t,t+1)$ to enforce local temporal consistency. Here we present a comparison of the IVF task results when the temporal window size is set to 5:

   	VIF	SSIM	MI	Qabf	BiSWE	MS2R	Inference Time
   5-frame	0.45	0.64	2.45	0.67	3.82	0.35	0.44
   3-frame (Ours)	0.44	0.64	2.47	0.68	3.84	0.35	0.21

   From the comparison results, we observed that while larger windows improve long-term temporal consistency, they also introduce alignment drift and increase inference latency. In other words, increasing the window size to 5 brought only marginal improvements in temporal metrics, but negatively affected spatial metrics and significantly increased the model’s inference time. Therefore, considering the trade-off between temporal consistency, spatial information, and inference efficiency, we decided to use a temporal window of 3 frames, i.e., the current frame as well as the previous and subsequent frames.

4. Q4: Selection of SEA-RAFT.
   R4: SEA-RAFT was chosen for its efficiency and reasonable accuracy in optical flow estimation. Per request, we conducted ablation with an alternative flow estimator, Unimatch [1], to better isolate the impact of flow quality on overall performance. The comparative results are reported in the table below. While higher-quality flow leads to slight performance gains, UniVF consistently outperforms other fusion baselines across different flow estimators. This indicates that our framework is robust to the quality of the flow input and does not rely heavily on the accuracy of any specific flow method, confirming the value of our multi-frame fusion architecture.

   	VIF	SSIM	MI	Qabf	BiSWE	MS2R
   w/ Unimatch	0.43	0.64	2.45	0.66	4.09	0.37
   Ours	0.44	0.64	2.47	0.68	3.84	0.35

   [1] Unifying Flow, Stereo and Depth Estimation. IEEE TPAMI 2023.

5. Q5: Flow network: frozen or end-to-end?
   R5: In our current implementation, the SEA-RAFT flow estimator is used as a frozen module (i.e., not updated during UniVF training). This decision is made to reduce training complexity and prevent degradation of flow quality due to fusion loss gradients, which are not directly aligned with flow supervision. We will also plan to experiment with fine-tuning the flow network jointly in future work.

We sincerely thank the reviewer for the encouraging comments regarding the practicality, ambition, and contributions of our work.

1. Q1: Time-slice visualizations for temporal consistency
   R1: We apologize for not being able to include images in the rebuttal, but we sincerely appreciate your suggestion. In the revised version, we will incorporate time-slice visualizations across all pixels of the fused video sequences, comparing UniVF with existing methods. These visualizations intuitively reveal differences in flickering and temporal smoothness. The results demonstrate that our method still achieves the best temporal consistency performance.
2. Q2: Blur strength parameter in multi-focus dataset
   R2:
   • The blur intensity factor $\sigma=0.025$ is empirically selected to match realistic defocus levels based on common imaging optics and perceptual blur thresholds. We determined this via visual tuning on a validation subset to balance realism and diversity.
   • To define the near-focus and far-focus planes, we first extract a video depth map using a monocular depth estimator. The far-focus plane is set to the 80th percentile of depth values, representing typical background regions. For the near-focus plane, we compute the average depth of each segmented object using DAVIS masks, then select the closest object and use its depth value.
3. Q3: Clarifying the HDR-based multi-exposure data pipeline
   R3:
   • Exposure adjustment in the linear light domain is critical because exposure is a multiplicative operation on scene radiance. Performing this in the gamma-compressed (non-linear) space would lead to physically incorrect results and color distortions. Linear light ensures radiometric fidelity.
   • Conversion from 10-bit BT.2020 to 8-bit BT.709 is necessary because: BT.2020 is used in HDR formats and supports a wider gamut and dynamic range. However, most consumer displays and fusion models are trained in 8-bit SDR color space with BT.709 format. The down-conversion ensures compatibility and standardization while retaining realistic visual variations.
   • We will revise the data generation section to clearly outline these motivations and justify the pipeline choices more rigorously.
4. Q4: Clarification of the feature warping operation
   R4:
   • The feature warping operation is deterministic and implemented using differentiable bilinear interpolation, not as a learnable module. Given optical flow from SEA-RAFT, we warp feature maps from adjacent frames to the current frame by sampling features at flow-displaced coordinates. This technique is widely used in temporal alignment and video restoration [1,2].
   • Specifically, for each input image, we estimate the bidirectional optical flows between adjacent frames, denoted as $\\{\mathcal{O}\_{t-1 \rightarrow t}^{k}, \mathcal{O}\_{t+1 \rightarrow t}^{k}\\}$. Each optical flow field represents the motion of pixels from one frame to another, where each flow vector indicates the displacement of a pixel to its corresponding location in the neighboring frame. These flow fields are then applied to the deep features $\\{\Phi\_{t-1}^{k},\Phi\_{t+1}^{k}\\}$ extracted by the encoder, enabling us to align them with the target frame $t$. The features are sampled at subpixel locations using differentiable bilinear interpolation to produce the warped representations $\\{\widetilde{\Phi}\_{t-1 \rightarrow t}^{k},\widetilde{\Phi}\_{t+1 \rightarrow t}^{k}\\}$, which are then fused for final prediction.
     [1] BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment. CVPR 2022.
     [2] VRT: A Video Restoration Transformer. TIP 2024.
5. Q5: C2RF results in IVF task.
   R5: We present additional IVF comparison results with C2RF in the table below. Our method continues to achieve state-of-the-art performance in video fusion.

   	VIF	SSIM	MI	Qabf	BiSWE	MS2R
   C2RF	0.27	0.57	1.67	0.47	5.34	0.70
   Ours	0.44	0.64	2.47	0.68	3.84	0.35

6. Q6: "image domain metrics" vs. "temporal domain metrics"
   R6: We sincerely appreciate your suggestion. We will revise the manuscript to refer to them as "spatial metrics" and "temporal consistency metrics" for clearer distinction.

Thank you for the follow-up question and your positive feedback on our previous response.

We believe that the multi-exposure and multi-focus data generation pipelines are both indispensable and, under current constraints, represent the most practical and scalable solutions for constructing a high-quality, diverse video fusion benchmark. Each pipeline is carefully designed to emulate the distinct physical and perceptual characteristics of its respective fusion task. Omitting either step in the pipeline would significantly reduce the benchmark’s scope and realism. Specifically:

• Multi-exposure pipeline: To simulate a wide dynamic range and preserve scene details under varying exposure conditions, we begin with 10-bit HDR video, perform exposure adjustment in the linear light domain, and apply realistic tone compression and gamut conversion. Simpler alternatives—such as capturing multi-exposure video sequentially with a single camera, or simultaneously with two cameras using different exposure settings—are either labor-intensive (requiring custom hardware and careful synchronization) or suffer from misalignment issues that are difficult to correct, particularly in dynamic scenes.
• Multi-focus pipeline: As discussed in Lines 159–164 of our paper, capturing multi-focus video using light field cameras or manually labeled focal planes is expensive and difficult to scale up. Naively applying foreground/background segmentation with artificial blur fails to reflect the physics of optical focus. Instead, we use depth-aware defocus simulation based on estimated depth maps, enabling realistic and scalable synthesis of focus transitions between foreground and background.

In summary, we agree that future work could explore a more unified or hybrid generation process that simultaneously models both exposure and focus variations. However, to establish clean task boundaries and ensure interpretability in our initial benchmark, we find that the current pipelines offer the best trade-off between realism, scalability, and clarity.