ViT-UWA: Vision Transformer Underwater-Adapter for Dense Predictions Beneath the Water Surface

Regarding Question 4:

SAM2 relies on prompts to leverage its segmentation capabilities. Our ViT-UWA is an underwater dense prediction model based on Vision Transformer, which does not rely on any prompts and achieves outstanding performance in underwater image dense prediction tasks. We compared the performance of our method and SAM2 in underwater instance segmentation [2]. The experimental results are shown in the Table 2. ViT-UWA-B† achieves 11.1% mask AP and 16.2% mask AP gains compared to SAM2-H on UIIS and USIS10K.

Table 2: Comparison of SAM2 and ViT-UWA on UIIS and USIS10K datasets.

[2] Lian S, Li H. Evaluation of segment anything model 2: The role of sam2 in the underwater environment[J]. arXiv preprint arXiv:2408.02924, 2024.

Thank you for your insightful comments and valuable suggestions. We have addressed each of your questions as follows:

Regarding Weakness 1 and Question 1:

We propose an adapted ViT backbone to address underwater dense prediction tasks in a unified manner.

We designed HFCP for ViT architecture to recover high-frequency information lost in underwater images. The original ViT focuses more on global information through the self-attention mechanism. Compared to other adapted ViT backbones that do not alter the plain ViT architecture, we observed that using the high-frequency components of underwater images as a prior for ViT effectively helps it capture underwater local high-frequency information. Moreover, compared to directly enhancing the images (Appendix B.5), our HFCP is better suited for underwater dense prediction tasks.

Convolutions can enhance transformers' ability to capture local spatial information, while differential convolutions can enhance the representation and generalization capacity of convolutions. We proposed HFDConv by combining adaptive differential convolutions (adaptive DC) and standard convolutions to enhance the visibility of details by detecting changes in pixel intensity (for detailed technical information, please refer to Section 3.3). Through using CNNs and HFDConv in DAM, we obtained multi-scale features focused on fine details, which are highly beneficial for underwater dense prediction tasks.

Our VCIM acts as a bridge between ViT and CNN, enabling bidirectional feature fusion through efficient deformable attention. It integrates the high-frequency features restored by HFCP with the multi-scale features captured by DAM, further enhancing the model's performance in underwater dense prediction tasks.

Regarding Weakness 2 and Question 2:

We have provided the FLOPs results in Tables 1, 3, and 4 in the main text. We also tested the inference time of our method and other adapted ViT methods, and the results are shown in the following Table 1 (also shown in Table 15 of Appendix B.6). Our ViT-UWA achieved minimal computational overhead and relatively fast inference speed. Moreover, we conducted qualitative comparison in some challenging real-world underwater application scenarios. As shown in Figure 11 of Appendix B.6, in low-light and turbid underwater environments, other ViT-based methods often encounter issues such as errors in object count detection and detection failures. In contrast, our ViT-UWA can alleviate these issues to some extent.

Table 1: Comparison of FLOPs and inference time.

Regarding Weakness 3 and Question 3:

There are many factors that contribute to underwater image degradation, such as light absorption and scattering. Underwater image degradation involves the loss of high-frequency information, color shifts, low contrast, etc. Based on our early attempts and analysis of related literature [1], using image enhancement methods (e.g., color correction and contrast improvement) to mitigate degradation totally does not improve the performance of subsequent tasks; in some cases, it may even lead to performance degradation.

From another side, our ViT-UWA focuses on dense prediction tasks, including detection and segmentation. Therefore, we prioritize the recovery of high-frequency details that are beneficial for improving our model's performance. During experiments, we found that our HFCP module could leverage frequency-domain information to mitigate underwater degradation issues such as color shifts and low contrast, thereby improving the model's performance. For instance, color shifts (evident in Rows 7 and 9 of Figure 7 in the paper) blur the boundaries between water bodies and the seafloor. Compared to other methods, our model can clearly distinguish these boundaries. Similarly, low contrast (illustrated in Row 7 of Figure 8 in the paper) makes it challenging for ViT-based methods to differentiate between fish and the background. In contrast, our model is able to effectively detect fish camouflaged within the background.

[1] Xu S, Zhang M, Song W, et al. A systematic review and analysis of deep learning-based underwater object detection[J]. Neurocomputing, 2023, 527: 204-232.

Due to the character limit of a single comment, we will address Question 4 in the next comment.

Thank you for your comments. We have addressed each of your questions as follows:

Regarding Weakness 1:

We propose an adapted ViT backbone to address underwater dense prediction tasks in a unified manner.

We developed HFCP specifically for the ViT architecture to address the challenge of recovering high-frequency information lost in underwater images. While the original ViT architecture emphasizes global information via the self-attention mechanism, our HFCP leverages the high-frequency components of underwater images as a prior, effectively enabling ViT to capture local high-frequency details. Compared to other adapted ViT backbones that retain the plain ViT architecture, our approach significantly enhances performance in underwater scenarios. Additionally, as demonstrated in Appendix B.5, HFCP is more suitable for underwater dense prediction tasks than direct image enhancement methods.

To further enhance the model's capability, we introduced HFDConv, which combines adaptive differential convolutions (adaptive DC) with standard convolutions. This design strengthens the ability of the model to represent and generalize fine details by detecting pixel intensity changes (detailed in Section 3.3). Integrated into DAM, HFDConv enables the extraction of multi-scale features focused on fine-grained details, providing substantial benefits for underwater dense prediction tasks.

Lastly, our VCIM bridges ViT and CNN by facilitating bidirectional feature fusion through deformable attention. It effectively combines the high-frequency features restored by HFCP with the multi-scale features extracted by DAM, resulting in improved model performance for underwater dense prediction tasks.

Regarding Weakness 2:

Due to wavelength- and distance-dependent light attenuation and scattering, underwater images commonly suffer from significant loss of high-frequency information. This phenomenon is rare in terrestrial scenes and typically occurs only under specific conditions (e.g., foggy or low-light). Compared to terrestrial images, underwater images have a greater demand of high-frequency information. Therefore, we introduced high-frequency prior into the model to alleviate this issue.

On the COCO dataset, our ViT-UWA-T achieved the second-highest box AP and mask AP scores among state-of-the-art models with similar parameter sizes. In future work, we will try to adapt ViT-UWA to other complex scenarios to further validate the generalization capabilities of our model according to your advice.

Regarding Question 1:

Thank you very much for your careful review and valuable feedback. We sincerely apologize for the oversight in line 504. The correct phrasing should indeed be “decrease by 2.3 and 0.6 AP”. We have corrected the statement in the revised manuscript to clarify the performance change as you suggested.

Regarding Question 2:

We have conducted additional experiments to address your concerns. Specifically, we have removed HFCP entirely and replaced it with some supervised underwater image enhancement methods. The results of these experiments are presented in Table 14 of Appendix B.5 (also shown in the following Table 1), along with an analysis of their impact on performance. we observed that when training with enhanced images, the model's performance experienced a certain degree of decline. This may be due to underwater image enhancement methods altering the feature distribution of underwater images and introducing additional noise (e.g., halo effect), which negatively impacts dense prediction tasks. Additionally, we have included comparative visualizations (Figures 9 and 10 in Appendix B.5) to illustrate the differences in feature maps after removing HFCP. These additions highlight the effectiveness of HFCP.

Table 1: Different methods of underwater enhancement.

Regarding Question 3:

We have added an ablation study on the attention mechanism in Table 12 of Appendix B.3 (also shown in the following Table 2). Ordinary cross attention has quadratic complexity, which significantly increases the model's computational cost and training time. However, sparse attention like deformable attention has linear complexity, which reduces the model's computational cost and improves performance to some extent.

Table 2: Ablation of different attention mechanisms.

Thank you for your valuable comments and constructive feedback. We appreciate your efforts in reviewing our work. Below are our responses to your questions:

1. Regarding Weakness 1 and Question 1:

We have provided visualizations in the newly submitted manuscript under Appendix B.5 (Figures 9 and 10), where we demonstrate the ability of HFCP to capture high-frequency information. Additionally, we included experimental results in Table 14 (also shown in the following Table 1) to further validate the impact of removing HFCP.

Table 1: Different methods of underwater enhancement.

2. Regarding Weakness 2 and Question 2:

We have conducted ablation experiments to address this concern and included the results in the newly submitted manuscript under Table 13 of Appendix B.4 (also shown in the following Table 2). We adopt ViT-UWA-T as the basic model. These experiments compare DAM with simple CNN structures and similar-function modules of ViT-Adapter, ViT-CoMer, and InternImage to showcase its superiority.

Table 2: Ablation of Detail Aware Module. “DS” means downsampling layers.

As shown in the above Table 2, under a similar scale of parameters, our method achieves the lowest computational cost and the best performance, indicating that DAM can obtain multi-scale features with rich high-frequency details more efficiently.

Thank you for your recognition of our work, and the valuable advice. In the revised manuscript, we have added a discussion on physics-based light scattering correction methods (including work [1]) in the introduction section (highlighted in blue). Additionally, we conducted a preliminary experiment to attempt the incorporation with physical model correction. Referring to the method in [1], we employed a physical model to simulate light attenuation and backscatter, thereby performing a degree of signal correction on the input underwater images. As shown in Table 1, the box AP improved, likely due to the physical correction module reducing backscatter and light attenuation, which clarifies boundaries between targets and backgrounds in underwater images. However, the mask AP results showed a slight decrease, which may be due to the reduced consistency of internal features within the objects after physical correction. We will continue to explore improvements for this issue in future work, such as introducing certain constraints to minimize corrections within the internal regions of the objects. Moreover, depth information could be utilized to guide the model in learning varying light attenuation [1], enhancing its robustness in complex underwater scenarios.

The experimental results highlight the potential of physical model corrections, and we plan to continue refining these methods in future work. Thank you again for your valuable suggestions.

Table 1: Effect of Physical Correction Module. “PSC” means Physical Signal Correction.

[1] D. Akkaynak and T. Treibitz, "Sea-Thru: A Method for Removing Water From Underwater Images," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 1682-1691, doi: 10.1109/CVPR.2019.00178.