RouGE: Learning Gated Experts for Segment Anything in the Wild

1. Regarding Figures 2 and 3 in the motivation, does the qualitative and quantitative performance degradation of the SAM model occur only on synthesized degraded images? What would the results be if tested on degraded images from the real world? Can the SAM foundation model achieve good results on real-world degraded images?

2. Following up on the first question, would it be possible to add all synthesized degraded images to the SAM foundation model's pretraining dataset to directly improve its performance on degraded images?

3. During training, RouGE uses synthetic image pairs of degraded and clean images. Are the degraded images sourced from the real world or generated? Additionally, how does RouGE perform when real-world degraded images are used during testing?

4. The design of the probability gates lacks novelty. This type of gate, which relies on image features for implicit decision-making, can be applied to various tasks and is not specifically designed for handling different types of degraded images.

5. There is a lack of ablation studies, such as on the number of lazy experts, the number of trainable experts, the adapter design, and the choice of RouGE layer placement, among others.

1. Unclear Feature Definition: The image feature $F_t$ is not clearly defined, leaving ambiguity about its source and role. Please provide a detailed description of the generation process for $F_t$.

2. Computational Cost of CLIP: If CLIP is used for $F_t$ extraction, the computational load could be significant. A full breakdown of pipeline steps is needed to assess efficiency.

3. Unspecified Detection Model: The object detection model used for bounding boxes is not specified, leaving reproducibility issues and potential variability in results.

4. Ambiguity in Baseline Comparisons: The paper uses a sequential adapter integration, whereas Adaptformer has shown that parallel adapter configurations can yield better performance. Without testing RouGE with a parallel MoE adapter setup, the paper does not convincingly demonstrate that its adapter design is optimal.

5. Missing Adapter Details: Critical details like the adapter’s bottleneck dimension are omitted, limiting the method's replicability.

6. Incomplete Ablation Studies: The paper lacks comprehensive ablation studies for critical hyper-parameters, such as the number of experts and adapter bottleneck dimensions. Including these would strengthen the evaluation and provide insights into the effects of architectural choices on performance.

While the improved performance looks promising, the reviewer has several concerns:

1. There are many typos in the current manuscript and the consistency can be improved. For example, the reviewer think in the Figure 1 it should be "Mask" instead of "Musk"

2. Again in Figure 1, the clean image is the same as the rainy image, while the blurry and low-light ones have different content. This makes the figure hard to understand.

Some more major concerns:

3. It seems that the proposed RouGE has all experts activated, which loses the advantage of boosting inference during inference as in conventional MoE. Therefore, the current method looks more like a combination of different gating functions with a weighted function.

4. It is unclear what experts are activated for which kind of input and how the weighting function varies. In other words, the paper lacks in-depth analysis.

5. The proposed method only compares with one single baseline method. It is unclear if the proposed method can generalize well on the SAM-based works. In addition, the authors replace the proposed adapting/prompting method with other counterparts. While it is so-called a fair comparison, it would be better to at least include the variance of each method. It might be likely that the authors take the best performance from the proposed method and compare it with a lower performance from the other counterparts.

6. From Table 1, it can be seen that LN can already hugely improve the baseline performance with only 10% learnable param of the RouGE block. Therefore, the contribution of the the RouGE block becomes less convincing since we can believe adding more LN might achieve a similar performance. By the way, the proposed RouGE in fact looks very much like a concatenation of LN, with some dimension changes .... So the novelty is limited...

7. The reviewer is not sure if the claim on Real-World degradation is valid since most of the datasets contain only synthetic degradation.,,,

8. There are many more advanced works in the MoE for image restoration as shown in [1]. However, these works are not discussed. The reviewer thinks that the MoE design from this work looks too naive compared to these counterparts...

[1] A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends

1. Utilizing MoE to enhance robustness and performance is not a novel concept. The authors integrate MoE with low-rank Adapters in PEFT for the Segment Anything model, which may not provide more insights for the community.
2. Some ablations or results are absent, including Adapter dimensions and a comparison of training sources (e.g., GPU memory and additional computational overhead).
3. Reporting on the performance of structures using MoE with full fine-tuning is essential to evaluate the effectiveness of the PEFT strategy.