Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model

Thank you for your detailed review on our paper. We will address each of your comments thoroughly in the following sections.

Q1: Comparison with training segmentation model from scratch.

We conducted a performance comparison (Table 14 in Appendix F) between SAM training from scratch and Conv-LoRA on two datasets: ISIC 2017 Dataset and Leaf Segmentation Dataset. The substantial performance gaps observed underscore the considerable assistance provided by SAM's pretraining knowledge in enhancing downstream task performance. Moreover, the pretraining-finetuning paradigm has been proved useful in prior studies [1-3].

ISIC2017 Dataset

Leaf Segmentation Dataset

---

Q2: MoE vs. common multi-scale strategies.

A straightforward multi-scale strategy involves using multiple convolution branches and subsequently aggregating their features. In comparison, MoE proves to be more efficient due to its utilization of conditional computation, selectively activating only one branch during each forward pass. This conclusion is supported by the ablation study presented in Table 3, where MoE demonstrates faster training speed and reduced memory consumption. We also clarify the motivation of introducing MoE in Section 3.1.

An alternative multi-scale strategy entails integrating additional convolution networks alongside SAM to extract multi-scale features, akin to the ViT-adapter design [4]. However, this approach is no longer parameter-efficient, as it requires full fine-tuning the pre-trained backbone, which introduces significantly more parameters than LoRA, leading to increased training complexity. In contrast to these explicit multi-scale strategies, Conv-LoRA operates more implicitly within ViT, which inherently processes features predominantly at a single scale. This implicit integration distinguishes Conv-LoRA as a parameter-efficient alternative in addressing multi-scale considerations.

---

References:

[1] Dosovitskiy A, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929, 2020.

[2] Kaiming He, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000–16009, 2022.

[3] Liu Z, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF international conference on computer vision. 2021: 10012-10022.

[4] Chen Z, et al. "Vision transformer adapter for dense predictions." arXiv preprint arXiv:2205.08534, 2022.

Thank you for your thoughtful review on our paper. We will comprehensively respond to each of your comments in the following sections.

Q1: The reasons of performance improvements.

Our enhancements are rooted in two pivotal aspects. First, SAM relies on the plain ViT architecture, which is known for its absence of image-related local priors. Conv-LoRA tackles this limitation by introducing convolution operations that incorporate hard-coded local priors. The integration of local priors proves beneficial for segmentation tasks demanding pixel-level predictions.

Second, SAM's segmentation pretraining causes the ViT to emphasize low-level features while neglecting significant high-level semantic information essential for semantic segmentation. In contrast, Conv-LoRA, through finetuning, facilitates the ViT in reclaiming the capacity to capture nuanced high-level semantic information.

---

Q2: Motivation of MoE.

We leverage MoE to efficiently infuse local priors into ViT. As highlighted in Table 4 of our paper, the optimal feature scale for injecting local priors varies across different datasets, necessitating a multi-scale strategy. While a conventional approach involves employing multiple branches to simultaneously inject local priors at different scales and aggregating them, this method incurs higher computational costs compared to MoE. The efficiency of MoE lies in its ability to selectively activate sparse experts, minimizing computational overhead. Given our emphasis on efficient finetuning, we opt for MoE as a judicious choice for injecting local priors into ViT. We also clarify the motivation of introducing MoE in Section 3.1.

---

Q3: Marginal improvements.

Our approach showcases notable improvements on some datasets. For instance, on the ETIS dataset, as illustrated in Table 5, Conv-LoRA exhibits a performance boost of approximately 2% across three metrics when compared to LoRA. In certain cases, even a 0.2% improvement, as observed on the SBU dataset in Table 1, is deemed significant within the context of semantic segmentation literatures [1] [2].

It is essential to emphasize that our method adheres to a universal setting, validated across a diverse array of semantic segmentation tasks. This universality underscores the versatility and efficacy of Conv-LoRA, making it a robust choice for various applications in the realm of semantic segmentation.

---

Q4: What priors is MoE learning and how we grasp what it is doing?

MoE learns to dynamically select an appropriate scale for injecting local priors based on input features. To illuminate its functionality, we conducted an analysis by tracking the frequency of each expert's selection across different datasets during inference. The detailed results are presented in Figure 14 in Appendix H. Notably, distinct datasets exhibit preferences for different experts. For instance, in Leaf Segmentation, MoE tends to favor experts with upsampling ratios of 3 and 4, while on the ISIC 2017 dataset, it tends to select the expert with an upsampling ratio of 2. Connecting these insights with the optimal scale ablation results in Table 4, it become evident that MoE selects the expert adaptively for each sample, and consequently, the distributions of MoE selection across datasets reflect their different data distributions. This observation supports that MoE behaves adaptively and effectively with respect to each sample’s property. This adaptability reinforces the significance of MoE in tailoring its selection to the unique characteristics of diverse datasets, enhancing its effectiveness in local prior injection.

---

Q5: Reason of freezing the prompt encoder.

While the prompt encoder in SAM provides opportunities for seamless integration with other modules, such as object detectors, within larger systems, certain applications may necessitate a standalone end-to-end model for deployment. In this work, our focus is specifically on these application scenarios, aiming for a model that operates independently without reliance on external prompts. For simplicity, we have opted to remove the prompt encoder during both training and inference stages. The exploration of joint finetuning with the prompt encoder is left for future studies.

---

References:

[1] Zheng Q, et al. "Distraction-aware shadow detection" .Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 5167-5176.

[2] Zhu L, et al. "Mitigating intensity bias in shadow detection via feature decomposition and reweighting". Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4702-4711.

Q4: Computational cost comparison.

The comparison below evaluates the training and inference speed, along with the per-epoch training time, utilizing the ISIC2017 dataset on a single V100 GPU (Table 15 of Appendix F). Despite Conv-LoRA exhibiting a slower processing speed compared to other methods, it demonstrates consistent performance improvements across diverse semantic segmentation tasks, as illustrated in Table 1 of our paper. Our analysis indicates that the primary computational cost arises from the upscale and downscale operations. A potential future direction is investigating methods to inject local priors without the explicit scaling up and down of features. Additionally, we notice the presence of orthogonal works, such as token merging [2] [3], which may offer ways for accelerating Conv-LoRA.

Training:

Inference:

---

Q5: Extra tuning efforts for picking suitable experts? Again, will the way how different sizes are combined in the inception structure be a better option?

We employ MoE to automate the selection of appropriate scales, obviating the need for additional tuning efforts. In our experiments, we utilize 8 experts that cover a range of 8 scales, a configuration we have found to be effective for many datasets. As previously discussed, our rationale for favoring MoE over the inception structure stems from its ability to handle scales larger than the default, a capability that the inception structure lacks.

---

References:

[1] Szegedy C, et al. “Going deeper with convolutions”, Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 1-9.

[2] Bolya D, et al. "Token merging: Your vit but faster." arXiv preprint arXiv:2210.09461, 2022.

[3] Liang W, et al. "Expediting large-scale vision transformer for dense prediction without fine-tuning." Advances in Neural Information Processing Systems, 2022, 35: 35462-35477.

We appreciate your detailed and valuable review on our paper. We will address each of your comments thoroughly in the following sections.

Q1: Motivation of MoE + convolution as a couple.

Convolution is to inject the local prior into the plain ViT, and MoE is to help to do so efficiently. Table 4 in our paper reveals that different datasets exhibit varying optimal scales. While a multi-scale strategy is a straightforward approach, our results indicate that employing MoE proves to be a more efficient solution. As demonstrated in Table 3, the utilization of MoE demonstrates notable advantages in terms of training speed (0.71 vs. 0.46 iterations per second) and lower training memory consumption (21.7GB vs. 23.4GB) compared to the multi-scale strategy. We also clarify the motivation of introducing MoE in Section 3.1.

---

Q2 (1): What will be the difference between multiple down-scale+conv+up-scale blocks vs. convolutional blocks with various kernel sizes?

The key difference lies in how they operate on feature scales. Convolutions with different kernel sizes primarily function within the default feature scale. While a larger kernel size may approximate the injection of local priors into features of a smaller scale, it may not effectively introduce local priors into features of larger scales than the default. Given that the features in the Vision Transformer (ViT) are downscaled by a factor of 16 from the original image, the injection of priors into larger scale features proves to be generally more beneficial, as substantiated by the findings presented in Table 4, where the optimal scale is usually larger than the default scale.

Moreover, we conducted a performance comparison between various kernel sizes, following the Inception structure [1], and our proposed design. To ensure a fair evaluation, our design incorporates four experts, aligning with the branch number of the Inception structure. Experimental results (Table 13 of Appendix F) on ISIC 2017 Dataset and Leaf Dataset, each performed twice and averaged, unequivocally demonstrate the superior performance of our design over the Inception structure.

ISIC 2017 Dataset:

Leaf Dataset:

---

Q2 (2): Is MoE necessary? Will simple addition (or average work)?

In contrast to the approach of employing multiple branches followed by addition or averaging, where all branches are activated for every sample, the use of sparsely gated MoE proves to be a more efficient method without incurring a proportional increase in computational costs. As highlighted in Table 3 in our paper, MoE demonstrates superior efficiency in terms of both training speed and memory consumption. This efficiency holds significant importance for Conv-LoRA, particularly as our focus lies in the efficient fine-tuning of SAM. Additionally, MoE outperforms simple multi-scale addition as shown in Table 3.

---

Q3: Compare the training procedure with original SAM.

SAM’s image encoder was initialized from MAE pretrained ViT, and SAM was pretrained on billion masks and 11 million images. The training is a promptable segmentation task with a set of geometric prompts, including mask, point, and box. Given one image and a prompt, SAM predicts three nested masks (for the whole, part, and subpart), computes a linear combination of focal loss and dice loss for each mask, and backpropagates solely the minimal loss among the three predictions.

We aim to do parameter efficient finetuning (PEFT) of pretrained SAM on downstream tasks. During training, all the parameters in SAM’s image encoder are frozen except for the small amount of Conv-LoRA parameters. Additionally, we exclude the prompt encoder to transform SAM into an end-to-end model, eliminating the reliance on external prompts. In this finetuning process, the model directly predicts semantic segmentation masks from a given image, supervised by the ground truth masks specific to the downstream tasks.

Q2: Why SAM's foreground-background pre-training is insufficient?

SAM's foreground-background segmentation pretraining is instrumental in discerning object hierarchies. However, it is important to recognize that the segmentation within an object, though beneficial for hierarchy identification, differs significantly from the demands of multi-class segmentation. The latter necessitates a holistic understanding of global semantics and the ability to distinguish objects belonging to distinct classes.

SAM was trained on a dataset that exclusively comprises segmentation masks without explicit semantic information. In theory, to minimize loss, the fundamental objective for its encoder is to project pixels into a metric space where pixels from the same object are in close proximity, while those from distinct objects are distantly positioned. This projection requires an implicit understanding of "objectness", focusing on proximity within an image rather than preserving consistent representations of the same-type object across different images. This introduces a potential challenge in aligning representations with semantics across diverse images.

To substantiate this assumption, we conducted a linear probing experiment on the ImageNet-1K dataset, utilizing ViT base models from SAM and MAE. The results presented in Section 4.2 of our paper reveal a notable discrepancy, with MAE exhibiting significantly higher accuracy compared to SAM (67.7% vs. 54.2%). Further evidence is presented in Figure 12 of Appendix G, illustrating that many attention heads near the tail of the ViT exhibit relatively small attention distances after SAM pretraining.

linear probing experiment:

This tendency of SAM's pretraining to emphasize information retention for low-level segmentation comes at the expense of losing crucial information essential for high-level classification, a key requirement for multi-class semantic segmentation. These findings collectively underscore the possible limitations of SAM's foreground-background pretraining in adequately addressing the complexities of multi-class semantic segmentation tasks.

---

Q3: Misrepresented works.

Thank you for bringing attention to this aspect. We acknowledge your observation that both Zhang&Liu and Shaharabany et al. have indeed adapted SAM to function as an end-to-end model. In response to your feedback, we have revised our claims, removing the assertion of SAM being an "end-to-end solution" from our list of major contributions.

Regarding the notion of "capturing high-level semantic information," we wish to emphasize that Zhang&Liu did not explicitly delve into a detailed analysis of this particular limitation of SAM. We want to convey that, irrespective of whether these adaptations are made with an awareness of SAM's this limitation, our emphasis lies in conducting in-depth analyses (linear probing and attention distance statistics) and offering insights into SAM's constraints. We believe that such analyses are essential in shedding light on potential areas for improvement in SAM in the future.

---

Q4: Choice of Interpolation Method and Larger Upsampling Factors.

We utilize bilinear interpolation and have added a row with a scaling ratio of 8 in Table 4 of our paper. The findings indicate that a larger scaling factor does not consistently lead to improved performance, as the object scale exhibits variability across different datasets. Hence, we propose to let the model itself learn to select scale dynamically through MoE (Mixture of Experts).

---

Q5: Where is Conv-LoRA added in ViT?

We apply Conv-LoRA to the query, key and value matrices in self-attention layers, the same locations of LoRA.

---

Q6: Add more references. We have added the suggested references in the first paragraph of introduction.

---

Q7: Add related works. We have added one new paragraph in the related work section introducing other concurrent efforts in finetuning SAM. Following your suggestion, we have removed "end-to-end segmentation" from our major contributions.

We sincerely appreciate your thorough review of our paper and the valuable insights you provided. In the following sections, we will provide a detailed response to each of your comments.

Q1(1): What do the authors mean by "SAM's local prior assumption" mentioned in the abstract?

SAM’s local prior is grounded in its extensive segmentation pretraining. Through supervised training on a vast dataset encompassing 1 billion high-quality masks and 11 million images, SAM has honed a robust capability to discern and capture local features within images. Notably, SAM's encoder retains the ViT architecture, which inherently lacks a dedicated local prior. However, this deficiency is effectively compensated for by the significant local prior acquired through segmentation pretraining.

To substantiate SAM's local prior, we conduct an analysis using the mean attention distance as a metric [1] [2]. In Appendix G (Figure 12), our findings reveal that SAM exhibits many heads in the deep blocks with short mean attention distances. This observation indicates SAM's heightened focus on local information during the later stages of the encoder. In contrast, the MAE pretrained ViT[3], representing SAM's initialization, displays consistently long mean attention distances among attention heads in the later stages. Consequently, SAM's segmentation pretraining induces a transformative shift in ViT's attention heads, steering them from a global-oriented to a local-oriented configuration. This transformation underscores the efficacy of SAM's approach in imbuing the model with a distinctive local prior, enhancing its ability to capture fine-grained details within images.

---

Q1(2): How can "image-related inductive biases" be reinjected on top of features that do not have those biases?

It is crucial to clarify that the features do not entirely lose their "image-related inductive biases"; rather, they retain certain biases even after processing through the model. Extensive pre-training on large-scale datasets has been demonstrated to equip ViTs with these inductive biases [1] [2]. SAM's large-scale segmentation pretraining further amplifies these inductive biases within the features.

To evaluate the inductive biases of the features, we conduct comprehensive comparisons among randomly initialized ViT, MAE pretrained ViT, and SAM ViT. Two key metrics, mean attention distance, and relative log amplitudes of Fourier transformed features[4], are presented in Figure 13 in Appendix G. Notably, as the training progresses from randomly initialized ViT to MAE ViT and ultimately to SAM ViT, we observe a consistent trend: the mean attention distance decreases, while the high-frequency signals in the features increase, indicating a focus on local information.

It is important to underscore that Conv-LoRA is designed not to reintroduce inductive biases but rather to reinforce existing implicit biases within the features. Throughout the paper, we deliberately use the term "reinforce" to accurately convey the model's objective, avoiding any implication of reintroduction. This distinction reflects our nuanced approach in ensuring that the model builds upon and strengthens the inherent image-related inductive biases within the features.

---

Q1(3): "data-efficient method for finetuning"

Regarding your speculation about our method being a data-efficient approach, we consider this to be a complementary explanation for the observed performance improvement. Because convolutional operations naturally contain inductive biases.

We also conduct experiments (Table 16 in Appendix I) in the few-shot setting, wherein acquiring data for downstream tasks is challenging, and only a limited number of training samples are available. Our experiments are performed on the Trans10K-v2 dataset, which is used for twelve-class transparent object segmentation. We randomly select the training samples, to meet the settings of 1, 2, 4, 8, and 16 shots (i.e., the number of labeled training examples per class), and the experiments are run for 100 epochs:

The performance improvement of Conv-LoRA compared to LoRA, is particularly pronounced in an extremely low-data setting (e.g., 1-shot). The result demonstrates that the introduction of inductive bias in Conv-LoRA contributes to improved data efficiency.

Q8: Missing labels in Figures 8,9, and 10

Good catch. We have added labels in those figures.

---

Q9: Variable name consistency

We add the definition of the variable names in the caption of Figure 4.

In Equation 2, the dimensionality of $x$ is denoted as $B \times C_{in} \times H \times W$, where $B$ represents the batch size, $C_{in}$ denotes the number of input channels, and $H$ and $W$ correspond to the height and width, respectively. As encoder handles the channel dimension, the size of $W_ex$ being $B \times r \times H \times W$, where $W_e \in R^{r \times C_{in}}$. Previously, the batch size dimension was omitted; however, for clarity, we will now explicitly state the size of $x$ in Equation 2.

---

Q10: Add example images from the medical domain

We have provided some medical image examples in Figure 6 in Appendix E.

---

Q11: Spell out LoRA

We will update abstract with the full name of LoRA, i.e., low-rank adaptation. As ICLR2024 does not allow to change the abstract during discussion stage, we will modify "LoRA" to "Low Rank Adaptation" later.

---

Q12: Code Release

Good call. We are cleaning up our codebase and will release it later. We definitely want to contribute to the open-source community and facilitate future research.

---

References:

[1] Dosovitskiy A, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929, 2020.

[2] Raghu, Maithra, et al. "Do vision transformers see like convolutional neural networks?." Advances in Neural Information Processing Systems 34 (2021): 12116-12128. Zhang, Kaidong, and Dong Liu.

[3] Kaiming He, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000–16009, 2022.

[4] Park N, et al. "How do vision transformers work?." arXiv preprint arXiv:2202.06709, 2022.