GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation

We appreciate your insightful feedback and constructive suggestions, which help improve our paper substantially. We will address all the comments and include additional experiments in the revision. Please find our responses to the comments below. All ablation studies were conducted using the ViT-B backbone and evaluated on LVIS with point prompts.

Weakness 1. Statistical reporting

Thank you for the comment. We agree that reporting statistical variation is important for ensuring reliability. Accordingly, we repeated the main experiments three times with different random seeds using the ViT-B backbone and evaluated the models using point prompts. As shown in the table below, our method consistently achieves strong performance with low standard deviation across four datasets.

We will revise all tables to reflect the statistical results.

Question 1. Impact of dynamic (rank-1) selection and rank space gating

Thank you for the insightful suggestion. To isolate the contributions of dynamic (rank-1) component selection and rank space gating, we conducted the following ablation studies.

1. Isolating the effect of dynamic (rank-1) component selection

We tested variants using only dynamic (rank-1) component selection within a unified rank space. Enabling the dynamic selection improves performance when using a unified rank space with the small maximum rank (i.e., 16), highlighting the benefit of adaptively choosing components per input. However, using a unified rank space with the large maximum rank (i.e., 256) did not lead to better results. We attribute this to the limited diversity in active components when using a unified space, which can restrict the ability to adjust representational capacities according to each degraded input, thus hindering input-adaptive modulation. It motivated our design to split the rank space, which led to improved performance.

2. Isolating the effect of rank space gating

We also experimented with rank space gating alone with fixed rank settings (e.g., rank 16 or rank 128), disabling dynamic (rank-1) component selection. The results demonstrate that rank space gating alone significantly improves performance. We believe this is because it enables each rank space to specialize in handling different degraded inputs, allowing the gating to select the most suitable representational capacity for each. Furthermore, enabling (rank-1) component selection on top further boosts the performance, highlighting its complementary benefit.

These findings suggest that both dynamic (rank-1) component selection and rank space gating mechanisms are crucial and complement each other. We will include these analyses in our revised manuscript.

Question 2. Generalization beyond driving scenarios

We appreciate your important question regarding cross-domain applicability. While BDD consists of autonomous driving scenes, LIS contains diverse low-light scenes across both indoor and outdoor environments. We will clarify this in the revision.

To further evaluate the generalization capability of GaRA-SAM beyond driving scenarios, we additionally tested it on the ISIC-2016 medical segmentation dataset [1]. Using box prompts, GaRA-SAM showed notable improvements over SAM:

These results indicate that GaRA-SAM maintains strong performance even in domains with very different visual characteristics. We will include this result in the revision.

[1] Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC), arXiv 2016

We appreciate your insightful feedback and constructive suggestions, which help improve our paper substantially. We will address all the comments and include additional experiments in the revision. Please find our responses to the comments below. All ablation studies were conducted using the ViT-B backbone and evaluated on LVIS with point prompts.

Weakness 1. Methodological innovation

Thank you for the comment. We would like to clarify that GaRA introduces a key difference from existing methods such as NoRA [1], AdaLoRA [2], and ElaLoRA [3] by enabling input-adaptive rank modulation at inference time.

Specifically, NoRA employs a nested LoRA structure to improve parameter efficiency but uses a fixed rank throughout both training and inference. AdaLoRA and ElaLoRA adjust ranks during training to optimize parameter-budget allocation, but their rank configurations remain fixed at inference time and do not adapt according to the input.

In contrast, GaRA-SAM is motivated by the observation in Figure 2(b) of the paper that optimal LoRA ranks vary significantly across inputs. GaRA is specifically designed to leverage this variability by dynamically selecting and combining (rank-1) components per input. This enables effective input-adaptive rank modulation even at inference time.

We will revise the paper to more clearly highlight this distinction.

[1] Nora: Nested low-rank adaptation for efficient fine-tuning large models, arXiv 2024.

[2] Adaptive budget allocation for parameter-efficient fine-tuning, ICLR 2023.

[3] Elalora: Elastic & learnable low-rank adaptation for efficient model fine-tuning, arXiv 2025.

Weakness 2. Table order confusion

We apologize for the confusion. We will reorganize the tables in the revision to match the flow of sections, improving overall readability.

Weakness 3. Visibility of Figure 4

The third row in Figure 4 shows a real nighttime image under extreme low-light conditions. The RGB pixel intensities range from 0 to 34 with a mean of 4.49, indicating severely limited illumination in the scene. To enhance visibility, we will include a brightness-enhanced version and clarify this explanation in the revision.

Question 1-1. Fine-tuning query, key, and value projections

We conducted additional experiments directly fine-tuning query, key, and value projection layers. Although this fine-tuning strategy improved robustness compared to the pretrained SAM, it significantly underperformed compared to GaRA-SAM, as shown in the table below.

Question 1-2. Fine-tuning the traditional adapter

We fine-tuned the traditional adapter [1] as an alternative to LoRA. The results below showed limited robustness improvement compared to LoRA. We suspect this is due to the nonlinear bottleneck of the traditional adapter, which can distort features and lead to unstable adaptation under corruption. In contrast, LoRA applies linear updates directly to the weights, enabling more stable adaptation.

[1] Parameter-efficient transfer learning for NLP, PMLR 2019.

Question 2. Applying LoRA to the decoder

We applied LoRA to the decoder instead of the encoder, but this resulted in a significant performance drop. We attribute this to the encoder continuing to generate degraded representations under corrupted inputs, which causes a mismatch with the adapted decoder parameters. In contrast, applying LoRA to the encoder directly mitigates feature degradation, leading to substantially better performance:

Question 3. Mergeability of GaRA into the weights of SAM

GaRA cannot be merged statically into SAM since its gating decisions are dynamic and input-dependent, i.e., different (rank-1) components are activated for different inputs during inference. Nevertheless, GaRA remains efficient, as it introduces only minimal overhead while keeping the backbone frozen. We will clarify this point in the revision.

Question 4. Code availability

We fully agree that open-sourcing is essential for reproducibility and future research. We will release the full implementation, including model weights and training/evaluation scripts.

We appreciate your insightful feedback and constructive suggestions, which help improve our paper substantially. We will address all the comments and include additional experiments in the revision. Please find our responses to the comments below. All ablation studies were conducted using the ViT-B backbone and evaluated on LVIS with point prompts.

Weakness 1 & Question 1. Justification for rank space separation

We sincerely appreciate this insightful comment. The rank space separation is motivated by our empirical observations that, in a unified rank space, the model exhibits limited diversity in (rank-1) component activation across different degraded inputs. Specifically, the model tends to activate components within a narrow range (133-190; std: 7.7), limiting its ability to adjust representational capacity according to each degraded input. This is potentially suboptimal because different degraded inputs require different representational capacities (Figure 2(b) of the paper).

By separating the rank space into lower-rank and higher-rank regions, we allow the model to activate a more diverse range of components (5-181; std: 75.8), yielding significant performance improvement.

We guess this is because the rank space separation into two exclusive sets, each specialized for different degraded inputs demanding different representation capacities (Figure 2(b) of the paper), helps mitigate potential conflicts during (rank-1) component selection and promotes more diverse usage of components.

We will clarify this motivation and supporting evidence in the revised manuscript.

Weakness 2-1. Expanded discussion on limitations

We sincerely appreciate the thoughtful comment. We will expand our limitations section, clearly detailing the manual design choices and their potential constraints. We plan to revise the section as follows:

Limitations. While GaRA adaptively determines the appropriate rank for each rank space, several design choices, such as the split between lower- and higher-rank spaces, the predefined maximum ranks, and the fixed 2-level gating hierarchy, are set manually and thus limit flexibility. More generalized automated designs are conceptually desirable but practically challenging due to unstable optimization. Developing automated methods remains an important future direction.

Weakness 2-2. Addressing experimental limitations

Thank you for raising these important concerns. We conducted additional experiments to address the experimental limitations.

1. Training SAM from scratch: We trained SAM from scratch using corrupted data, which resulted in significantly lower performance compared to GaRA-SAM. This performance gap highlights the importance of leveraging the strong generalization ability of SAM, which originates from large-scale pretraining. We will include these results in the revised manuscript. | Model | IoU | |-------------------------|:------------:| | SAM trained from scratch | 57.3 | | GaRA-SAM | 77.0 |
2. Statistical significance of results: We repeated the main experiments three times with different random seeds using the ViT-B backbone and evaluated the models using point prompts. As shown in the table below, our method consistently achieves strong performance with low standard deviation across four datasets. We will revise all tables to reflect the statistical results. | Dataset | Degraded IoU | Degraded PA | Degraded Dice | Clear IoU | Clear PA | Clear Dice | |---------|:--------------:|:-------------:|:----------------:|:------------:|:-----------:|:-------------:| | LVIS | 76.7 ± 0.3 | 94.2 ± 0.2 | 85.5 ± 0.2 | 82.0 ± 0.8 | 95.4 ± 0.1| 89.1 ± 0.6 | | MSRA | 89.3 ± 0.2 | 97.4 ± 0.1 | 93.9 ± 0.2 | 91.3 ± 0.1 | 97.9 ± 0.1| 95.1 ± 0.1 | | COCO | 76.9 ± 0.5 | 94.4 ± 0.2 | 85.6 ± 0.3 | 80.5 ± 1.0 | 95.2 ± 0.3| 88.0 ± 0.7 | | SN | 77.3 ± 0.5 | 99.5 ± 0.0 | 86.2 ± 0.4 | 82.6 ± 0.4 | 99.7 ± 0.0| 89.9 ± 0.3 |

Weakness 3. Figure and table readability

Thank you for your helpful comment. We will improve readability in the revision by adjusting figure spacing, table layouts, and font sizes.

Weakness 4. Code availability

We fully agree that open-sourcing is essential for reproducibility and future research. We will release the full implementation, including model weights and training/evaluation scripts.

Question 2. Why $r_L = 16$ and $r_H = 256$?

These values were selected based on ablation studies across various rank configurations. The combination of {16, 256} consistently shows the best performance:

We will include these results in the revised version.

Question 3. Clarification on “seen datasets”

Thank you for pointing this out. The term “seen datasets“ refers to test splits from the same distributions as training data. We will clarify this explanation in the revision.

Question 4. Details on performance evaluation

All experiments were conducted using four NVIDIA RTX A6000 GPUs for training and one A6000 GPU for inference. The number of non-learnable parameters was consistent across methods and amounted to 1,250M parameters. Regarding training time, GaRA-SAM required 25 hours, while RobustSAM required 33 hours under identical settings.

Thank you for the clarifications and additional ablations. They address most of my concerns. The other reviewers seem to share my positive evaluation - I will keep my recommendation to accept.

We appreciate your insightful feedback and constructive suggestions, which help improve our paper substantially. We will address all the comments and include additional experiments in the revision. Please find our responses to the comments below. All ablation studies were conducted using the ViT-B backbone and evaluated on LVIS with point prompts.

Weakness 1 & Question 1-1. Memory overhead of GaRA-SAM

Thank you for highlighting this important point. Memory efficiency is indeed an important direction for future research, particularly for deployment on edge devices. Although GaRA-SAM introduces 2 GB of additional memory for inference, we note that its total memory usage (5.39 GB) is comparable to RobustSAM (5.41 GB). Additionally, this footprint remains within the capability of widely used modern edge devices such as Jetson Xavier NX and AGX Orin, making GaRA-SAM practical for real-world applications.

Question 1-2. Comparison with a larger SAM

Thank you for the constructive suggestion. To assess this, we designed a larger variant of SAM by inserting two additional 4-layer MLP blocks immediately after the image encoder and before the mask decoder, matching GaRA-SAM’s parameter count. Despite its increased capacity and full fine-tuning, this larger model showed inferior performance compared to GaRA-SAM. We attribute this to the loss of the strong generalization capability of the pretrained SAM due to the full fine-tuning. On the other hand, GaRA-SAM effectively preserves the capability by learning the lightweight, input-adaptive modules only while freezing the pretrained SAM.

This comparative analysis will be included in the revision.

Question 2. How is the gating path supervised?

The gating modules of GaRA-SAM are not directly supervised, but they are trained indirectly through the segmentation loss alone, following established practices in prior work [1,2]. In other words, no direct supervision is provided to explicitly guide which gating paths the modules should take. To empirically demonstrate that the gating modules are learned to provide desirable gating paths even with such indirect supervision, we compared GaRA-SAM and its variant with random gating. The results reported below indicate that GaRA-SAM significantly outperforms the random gating variant, supporting the effectiveness of our training strategy.

These findings will be elaborated on in the revised manuscript.

[1] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, ICLR 2017

[2] Categorical Reparameterization with Gumbel-Softmax, ICLR 2017

Question 3-1. Why are ranks 16 and 256 used?

We conducted comprehensive ablation studies on various rank configurations. As summarized below, the combination of {16, 256} consistently achieves the best performance.

Thank you for the comment. We will include these results in the revised manuscript.

Question 3-2. Isolating the effect of rank space selection

Thank you for the valuable suggestion. To isolate the effect of rank space selection, we experimented with rank space selection alone with fixed rank settings (i.e., rank 16 or rank 128), disabling (rank-1) component selection. The results demonstrate that rank space separation alone significantly improves performance. We believe this is because it enables each rank space to specialize in handling different degraded inputs, allowing the gating to select the most suitable representational capacity for each. Furthermore, enabling (rank-1) component selection on top further boosts the performance, highlighting its complementary benefit.

These results will be included in the revision.

Question 3-3. Impact of the depth of hierarchical gating

Thank you for the suggestion. We conducted additional experiments to assess deeper hierarchical gating structures (i.e., 3-level and 4-level). Our findings reveal that increasing the depth introduces optimization challenges due to more complex gating decisions, ultimately degrading performance. Hence, we conclude that the proposed 2-level hierarchy offers both strong performance and architectural simplicity.

We are glad that our responses have addressed the majority of your concerns! We sincerely appreciate your positive evaluation and will include the additional results and discussions in the revision. Thank you again for your constructive feedback and support of our work.