Single Image Reflection Separation via Dual-Stream Interactive Transformers

Thank you for recognizing the motivation, technical soundness, and state-of-the-art performance of our method. Below, we address the key concerns you raised:

Q1: Why the design of inter-patch/intra-patch attention is quite plain?

A1: Keeping the design simple is beneficial to validate its effectiveness. When the pipeline becomes complicated, it can be hard to determine which part of the design really works. Also, more powerful and complicated attention designs can be developed in future work, which can potentially enhance the performance further.

Through the derivation of LaCA (Layer-aware Cross-Attention) provided in the global Author Rebuttal, it is evident that even with a straightforward token concatenation operation and corresponding modifications to the relative position bias, we naturally introduce inter- and intra-layer attention for dual-stream information. This demonstrates the inherent relationship between dot-product self-attention mechanisms and dual-stream interactive modeling.

In the meantime, our strategy can also energize other multi-stream transformer designs. For instance, in high-resolution image generation scenarios where a single GPU cannot support the entire image generation process, multiple information streams of the model can be distributed across different GPUs. Each stream would handle a specific image patch, and inter-stream feature interactions would ensure the coherence of the generated patches, enabling them to be seamlessly stitched together into a complete image (like in [1]).

Q2: What about the contributions of gradient and feature losses?

A2: Both gradient and feature losses contribute positively to our method. The importance of gradient relationship has been emphasized by early works in SIRS [2],[3],[4]. Feature (or perceptual) loss, on the other hand, significantly aids in modeling natural images by ensuring that the reconstructed images are perceptually similar to the ground truth [5],[6]. As suggested by the reviewer, to confirm their contributions, we additionally conducted ablation experiments, as shown in the table below. We evaluated the impact of removing Gradient Loss (DSIT (w/o Gradient Loss)) and Feature Loss (DSIT (w/o Feature Loss)) on the performance of our model. The results indicate a noticeable decrease in performance when these losses are disabled, underscoring their necessity. Specifically, the gradient loss helps preserve edge details, while the feature loss enhances the perceptual quality of the outputs, both of which are beneficial for effective reflection separation.

Q3: Please provide the model complexity and inference time.

A3: We have provided a comparison in model parameters, GFLOPs (for $384 \times 384$ resolution inputs), inference time (for $384 \times 384$ resolution inputs, tested on an RTX 3090 GPU), and the average PSNR/SSIM across 4 datasets (Real20, Objects, Postcards, and Wild) in the table below. The number of parameters of our model is comparable to RAGNet and DSRNet. Due to our multi-scale and non-recursive network structure, we achieve superior GFLOPs efficiency compared to all other methods. Despite being based on a Transformer architecture, our method demonstrates a slightly faster inference time than DSRNet, highlighting the efficiency of our design. The above results show that the improvement of our model is non-trivial. As further illustrated in the additional cases provided in the global rebuttal PDF, our method exhibits impressive generalization performance in real-world scenarios compared to previous state-of-the-art approaches. This reflects the strong capability in effectively modeling reflection scenes of our method.

[1] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models. CVPR 2024.

[2] User Assisted Separation of Reflections from a Single Image Using a Sparsity Prior. ECCV 2004.

[3] Single Image Layer Separation Using Relative Smoothness. CVPR 2014.

[4] A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing. ICCV 2017.

[5] Image style transfer using convolutional neural networks. CVPR 2016.

[6] Perceptual losses for real-time style transfer and super-resolution. ECCV 2016.

Thank you for your detailed feedback. We appreciate your comments and try our best to address the issues as follows:

Q1: The writing of the method section.

A1: We apologize for any difficulties caused by the writing in the method section. In the next version, we will thoroughly proofread and clarify our paper. Specifically, we will ensure that the terms DS-SA and DS-CA are properly defined before their first use, as follows:

"Following the LayerNorm, we apply Dual-Stream Self-Attention (DS-SA) to $**X**^{\text{LN}}\_0$ and Dual-Stream Cross-Attention (DS-CA) to $**X**^{\text{LN}}\_1$, obtaining $**X**\_{\text{SA}}$ and $**X**\_{\text{CA}}$, respectively. The design of these two attention mechanisms will be detailed later in this section."

Q2: (1) The appliance of the pre-trained transformer. (2) The CNN-based network placement.

A2: (1) We utilize a pretrained Transformer architecture for the Global Prior Extractor, as shown in Figure 2 (a) in the main paper. The transformer blocks in GPE can load pretrained weights before training, which can leverage rich priors learned from large-scale datasets to aid in reflection separation. This practice is consistent with previous reflection separation methods, which have similarly used pretrained models to enhance feature extraction, such as the HyperColumn used in Zhang et al. [1] and YTMT [2], and the DSFNet in DSRNet [3].

(2) As depicted in Figure 2 (a), the Local Prior Extractor, referred to as the CNN-based Network, is implemented using a convolutional dual-stream interactive network structure. In our implementation, each DSLP Block specifically utilizes the MuGI Block from the DSRNet [3] to highlight our main contribution and avoid other possible factors influencing performance. Additionally, more advanced convolutional modules can be developed in future work.

We will highlight these points in our revision to avoid misunderstanding.

Q3: Where is the "cross" presented?

A3: We explain in detail how explicit correlation assessment works and where is the "cross" presented in the A1 of common issues in the global rebuttal. We will clarify this point in our revised version.

Q4: Elaboration on Figure 3.

A4: In the figure, red regions in the feature visualization indicate high attention values, while blue regions represent low attention values. We will add a color bar in the revised version to clarify this.

Please note that the pretrained model used in transfer learning is task-agnostic. We do not expect the priors alone to distinguish between the reflection and transmission layers clearly. Instead, the priors provide varying degrees of attention to different components of the image. For instance, the global prior $\mathbf{F}^{\text{GP}}$, extracted using a pretrained Swin Transformer model, shows higher attention towards reflection components. The shifted window-based self-attention contributes to the coherent attention across the image. Additionally, the Local Prior Extractor captures local prior information for both the transmission stream $\mathbf{F}^{\text{LP}}\_\mathbf{T}$ and the reflection stream $\mathbf{F}^{\text{LP}}\_\mathbf{R}$. These local priors exhibit a "granular" texture and may not accurately attend to the correct components of transmission and reflection. After the Cross-Architecture Interaction with the global prior $\mathbf{F}^{\text{GP}}$, the attention of $\mathbf{F}^{\text{LP}}\_\mathbf{T}$ and $\mathbf{F}^{\text{LP}}\_\mathbf{R}$ towards the transmission and reflection layers becomes more appropriately focused. Through further interaction and optimization of Dual-Attention Interactive Blocks, the dual-stream features $\mathbf{F}^{\text{DAIB}}\_\mathbf{T}$ and $\mathbf{F}^{\text{DAIB}}\_\mathbf{R}$ turn out to be more fine-grained and accurate. This improvement has also been recognized by Reviewer k1uS (Strengths 3).

Q5: Does the current setting align with real-world scenarios?

A5: Please note that all the testing data used in our paper are actually captured in the real world, including Real20, Nature20, Wild, Objects, and Postcard. These datasets encompass a variety of indoor and outdoor scenes, different glass thicknesses, and varying distances, thereby covering a wide range of real-world reflection scenarios. Again, we emphasize that our method does NOT focus on synthetic scenarios, as the reviewer criticized.

Meanwhile, during the rebuttal period, we captured several reflection scenes in the wild according to what the reviewer described as typical real-world scenarios. We compared the inference results of state-of-the-art methods and our approach in the global rebuttal PDF. In Figure 1 of the rebuttal PDF, the first 3 examples primarily contain localized reflection regions. The 4-th example showcases the reflection phenomenon on a water surface, which is not covered in the "minimal real-scene data" during training mentioned by the reviewer. Our results are visually impressive, demonstrating our model's generalization capability. Besides, Figure 2 of the rebuttal PDF presents the predicted reflection layers from our method on these real-world images, along with the corresponding binarized maps. From these visualizations, it is evident that our method can effectively identify localized reflection regions.

Inspired by the suggestion of the reviewer, in future work, we may explore the introduction of a lightweight "reflection localizer". This would allow us to bypass the processing of non-reflective regions (like the manner of early exiting [4]), thereby accelerating the reflection separation process.

[1] Single Image Reflection Separation with Perceptual Losses. CVPR 2018.

[2] Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation. NeurIPS 2021.

[3] Single Image Reflection Separation via Component Synergy. ICCV 2023.

[4] Adaptive Patch Exiting for Scalable Single Image Super-Resolution. ECCV 2022.

Thank you for your positive feedback on our work. Below, we address your main concerns:

Q1: (1) Clarify the implementation of CAI, (2) N_T, N_W, and (3) DSLP.

A1: (1) CAI is a general design introduced to enable interaction and fusion of features extracted from different network architectures during the feature extraction phase. A specific implementation of this design is the dual-stream feature interaction module. We tested 3 different implementations of CAI as in the ablation study shown in Table 3 of the main paper. It turned out that our current choice (DAIB) is significantly superior to other alternatives, which exhibits the effectiveness of DAIB for both cross-architecture and cross-layer interactions.

(2) In our experiments, the training image size is fixed at $384\times384$. The window size of attention mechanisms, $N_W$, is fixed to $12\times12$. The number of windows, $N_T$, varies depending on the spatial scale of the features. For instance, if the spatial scale is $384\times384$, then $N_T = (384/12)\times(384/12)=1024$.

(3) The DSLP can be any convolutional dual-stream network module, such as the YTMT Block [1] and MuGI Block [2]. To highlight our main contribution and avoid other possible factors influencing performance, we simply adopt the MuGI Block in our experiments. Also, more powerful convolutional modules can be developed in future work.

We will clarify these model settings in our revised version and thoroughly proofread our paper to eliminate other writing problems.

Q2: Simpler Implementation of DAIB.

A2: In Table 3 of the main paper, we provided an ablation study on CAI. The "Add" variant represents the removal of the DAIB, directly adding the global priors on the dual-stream local priors. Although this setting reduces the complexity of the model (from 517 GFLOPs to 471 GFLOPs), it decreases the average PSNR from 26.27 to 25.74. To ensure superior performance, we retained the DAIB as the interaction module for CAI. However, considering broader applications, exploring more lightweight model designs can be more attractive, which may use a more lightweight backbone (MobileViT for example), reduce the number of channels and blocks, and employ knowledge distillation techniques. Thank you for your valuable suggestion, which inspires us to optimize the model for more practical use in the future.

Q3: Provide a comparison of FLOPs and Params.

A3: As shown in the Table below, we present a comparison in terms of GFLOPs (obtained via fvcore package at $384\times384$ input resolution) and learnable parameters between previous state-of-the-arts and ours. It can be seen that the parameter amount of our model is comparable to that of RAGNet and DSRNet, but our model is of the lowest computational complexity among all the compared methods. This efficiency is achieved by our multi-scale and non-recurrent model design. Moreover, our method demonstrates the best average performance across all datasets, as evidenced by the average numerical metrics on all testing images from Real20 and SIR^2.

Q4: Some parts in Figure 2 overlap excessively.

A4: We will definitely polish the figures in our revised version, for better visual appeal.

Q5: How does the model perform without using a pre-trained model?

A5: As shown in the table below, DSIT (w/o Pretrain) refers to our model with the Transformer backbone randomly initialized, and trained from scratch alongside the main network. The results reflect a remarkable performance drop, which comes from the data-hungry nature of Transformer architectures; training from scratch on a relatively small dataset can negatively impact the performance of models [3]. This outcome also underscores the importance of high-semantic pretraining in addressing the ill-posed nature of the reflection separation task, which has been demonstrated in previous methods like the HyperColumn in YTMT [1] and DSFNet in DSRNet [2]. In future work, improvements of the pre-training technique, such as incorporating same-task pre-training strategies like HAT [4], could further enhance the model performance.

Q6: Why does DSLP not use pre-trained models?

A6: We have already employed the pretrained Global Prior Extractor to extract high-semantic prior, the information flow of which is task-agnostic and requires task-specific guidance to adapt to our current task. The proposed dual-stream Local Prior Extractor (LPE) serves this purpose, guiding the task-agnostic information to be more relevant to reflection separation. Therefore, there is no need to further introduce a pre-trained task-agnostic LPE.

To further validate this, we experimented as shown in the table below. In the DSIT with CNN Pretrain, we replaced the original LPE with a pre-trained RepVGG-B3 model. We used CAI to interact and fuse features extracted by the pre-trained models from both architectures. However, we observed that this approach not only increased the model's overall GFLOPs but also resulted in weaker performance, which confirms our opinion above that using a pre-trained CNN network in place of our LPE does not improve performance. Thank you for your insightful question.

[1] Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation. NeurIPS 2021.

[2] Single Image Reflection Separation via Component Synergy. ICCV 2023.

[3] Training data-efficient image transformers & distillation through attention. ICML 2021.

[4] Activating More Pixels in Image Super-Resolution Transformer. CVPR 2023.

First of all, we sincerely appreciate your comments, which can help us further improve the quality of our paper. We also apologize for any confusion or inconvenience caused by writing issues. Below, we address the main concerns.

Q1: What does the superscript symbol $k$ mean? It is not consistent with the superscript of $\mathbf{F}$ in Figure 1.

A1: $\mathbf{F}$ denotes the feature flow in our dual-stream architecture. Specifically, $\mathbf{F_T}$ and $\mathbf{F_R}$ represent the feature flows for the transmission and reflection layers, respectively. When the superscript of $\mathbf{F}$ is a lowercase letter, such as in $\mathbf{F}^k_\mathbf{T}$, it indicates the feature flow of the transmission layer after passing through $k$ DAIBs (Dual-Attention Interactive Blocks). The superscripts $l$ in Figure 1 and $k$ in the methodology section have the same meaning. Moreover, when the superscript is formed by uppercase letters, such as in $\mathbf{F}^{\text{LN}}_\mathbf{T}$, it designates the feature flow after passing through a specific layer within a DAIB (in this case, LayerNorm). We appreciate your attention to this detail. We will definitely unify the notations in the next version for clarity.

Q2: The DS-SA and DS-CA should be explained or at least come with citations.

A2: The DS-SA (Dual-Stream Self-Attention) and DS-CA (Dual-Stream Cross-Attention) mechanisms are introduced in Section 3.1 of our paper, which are explained in the subsequent paragraphs following their first mention. In the revision, we will ensure that a brief explanation is provided at their first mention to clarify their roles as follows:

"Following the LayerNorm, we apply Dual-Stream Self-Attention (DS-SA) to $\mathbf{X}^{\text{LN}}\_{0}$ and Dual-Stream Cross-Attention (DS-CA) to $\mathbf{X}^{\text{LN}}\_{1}$, obtaining $\mathbf{X}\_{\text{SA}}$ and $\mathbf{X}\_{\text{CA}}$, respectively. The design of these two attention mechanisms will be detailed later in this section."

Q3: (1) How does DSLP FFN extract local features? (2) How are local features defined? (3) What is the difference between the input of DSLP and the input of the Transformer block?

A3: (1) Extraction of local features by DSLP FFN: The DSLP FFN (Dual-Stream Locality-Preserving Feed-Forward Network) and DSLP Block can be any convolutional dual-stream network modules, such as the YTMT Block [1] and MuGI Block [2]. In our experiments, we employ the MuGI Block. Since reflection decomposition is a dense prediction task, one of the primary functions of the DSLP is to maintain local information. DSLP achieves this goal by using convolutional structures to extract local features, like other hybrid Transformer architectures [4],[5],[6].

(2) Definition of local features: Local features refer to the local correlation among adjacent pixels that often have similar values. CNNs capture the local structures through limited receptive fields, weight sharing, and/or spatial sub-sampling [3],[4].

(3) Differences between inputs to DSLP and Transformer blocks: Both the features for the Transformer block and the DSLP block originate from the input image $\mathbf{I}$. However, the features fed into the Transformer block first undergo a PatchEmbed process. Before being input into the DSLP FFN, these features are reshaped after the LayerNorm to fit the convolution operation, like other hybrid Transformer architectures [5],[6].

To avoid any confusion, we will provide a detailed explanation in the next version to address these mentioned points thoroughly.

Q4: How does the proposed method assess the information flow between the two streams?

A4: We explain in detail how explicit correlation assessment works in the Common Issue in the Global Rebuttal. Additionally, the ablation study on DAIB in Table 3 of the main body of our paper shows that removing the cross-attention mechanism ("w/o DS-CA") leads to significant performance degradation on real-world datasets. This also demonstrates that our proposed method is able to make the information positively effective for reflection separation.

Q5: Does the proposed method fail to deal with the regions with dominant reflections?

A5: In the situation mentioned by the reviewer (Figure 13), some regions in the image are dominant with strong reflections (in other words, information of the transmission layer has been largely suppressed), which poses challenges not only to our method but also to previous state-of-the-arts. While other methods struggle to discriminate the reflection component from the entangled layers in such scenarios, our method removes significant portions of the reflections, way more effectively, which corroborates our generalization capability.

[1] Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation. NeurIPS 2021.

[2] Single Image Reflection Separation via Component Synergy. ICCV 2023.

[3] Object recognition with gradient-based learning. CGCV 1999.

[4] CvT: Introducing Convolutions to Vision Transformers. ICCV 2021.

[5] Incorporating Convolution Designs into Visual Transformers. ICCV 2021.

[6] Uformer: A General U-Shaped Transformer for Image Restoration. CVPR 2022.

The authors' rebuttal works have addressed most of the weaknesses raised in the initial review. Generally, the main concern is addressed and I've raised the score to 5.

The markdown equation display issue is resolved. I can see the equations after refreshing my browser. Please ignore my last message.

-AC