Adapting Informative Structures for Cross-Domain Few-Shot Segmentation

1. The proposed Informative Structure Identification (ISI) module, while innovative, appears to be tailored towards identifying informative structure parameters within CNN-based architectures. However, the manuscript does not sufficiently address how this approach could be extended to Transformer-based models, which have become increasingly prevalent in various computer vision tasks, including semantic segmentation.
2. The Informative Structure Identification (ISI) method seems to be close to the domain of Neural Architecture Search (NAS). However, the manuscript appears to lack a comprehensive discussion on NAS in both the related work and the experimental sections.
3. From Table 3, we can observe the computation cost is a significant drawback of the proposed method, as reporting a substantial running speed decrease from 16.5 FPS to 1 FPS. It is a major concern for applications requiring real-time processing capabilities. Additionally, the introduction of the EFI computation significantly increases the computational requirement, potentially making the method impractical for large-scale or time-sensitive tasks.

1. There is not a complete process diagram illustrating what each part does, and the relationships and roles between two parts require many readings of the paper to make the connection.
2. In the ISI phase, why can Fisher information be used as an indicator for tuning layers and parameters, and is there any empirical, theoretical, or experimental evidence for this?
3. Why can Fisher information be used as an indicator for ADJUSTMENT Layer and parameter optimization in the ISI phase? Is there any relevant experience, theoretical support or experimental results to prove this? Specifically, how can the relationship between Fisher information and model performance be derived from existing studies? How is this indicator compared with existing studies to show its superiority?
4. Specifically, the PSA stage needs to clarify the criteria and rationale for sample selection. Is the sample selection strategy based on the distribution of Fisher information, gradient changes, or through some heuristic rule? In addition, the triggering conditions for entering the sample-increasing inference phase need to be identified, e.g., is the expansion of samples started only after a certain degree of loss convergence is reached, or is it based on the dynamic judgement of the model's performance on the existing samples?
5. In the present experimental results, there is only a small increase in performance, and this marginal improvement is likely to be of limited value in practical applications. It is therefore difficult to fully justify the significant superiority of the methodology used on this basis alone. This requires a more in-depth analysis to explain the conditions under which such an improvement is reasonable and meaningful, and to ensure that such marginal improvements are not merely coincidental.
6. Currently there is a lack of sufficient experimental evidence to fully demonstrate that this metric based on Fisher information outperforms other methods, such as gradient-based methods, in all cases. More experimental comparisons are needed to determine whether the introduction of Fisher information actually improves model performance under different tasks or scenarios, and whether it is more generalizable and robust compared to gradient-based methods.
7. The PSA phase of the experiment has not been fully analyzed in comparison to existing sample conditioning strategies. This makes it difficult to judge whether there are unique advantages or improvements in strategies for sample selection in this phase.
8. The association between sample selection strategies and feature information needs to be explored further. Are all Fisher information equally important in sample selection? How does Fisher information guide the process of sample selection, and is there a difference in the role of Fisher information at different stages (e.g., pre-training vs. post-training)?

• Unclear training time consumption. As ISA includes another stage of identifying domain-aware model structures, while other CD-FSS approaches do not involve, authors are suggested to include more comparisons on FLOPs or test time consumption. If ISA sacrifices time for performance, authors are suggested to acknowledge this as a limitation. Also, can authors specify how many epochs ISA goes through when performing test-time training?

• Unclear comparison with previous approaches. In line 514-517, I notice that this paper mentions ISA adopts a different evaluation setup distinct from IFA [A], and when using IFA evaluations, ISA surpasses IFA (72.8 v.s. 71.4), which seems an average 5-shot result. Can authors clarify major difference between two evaluation setups, and also provide full comparisons (on 4 datasets, 1-shot and 5-shot), with IFA codebase?

[A] Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining. CVPR 2024.

This is an intuitive method, but the motivation seems weak, making the proposed method seems incremental.

W1&Q1: The authors claim that ``adjusting hand-selected model parameters, such as test-time training, typically neglects the distinct domain gaps and characteristics of target domains." Actually, online Test-Time Adaptation (OTTA) and Test-Time Instance Adaptation (TTIA) are two widely studied settings for Test-Time Training (TTT). These techniques can adapt a model to any single unlabeled test image, and goes far beyond adjusting hand-selected parameters [1]. As one of the most important motivations for this article, it is not convinsing enough, I think the author lacks broader literature research.

[1] A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts.

W2&Q2: Using Fisher information to find important parameters and adjust them seems to be a common idea in the field of machine learning, such as the author mentioned, it is also applied in continual learning. So where is the necessity of applying this technology in few-shot segmentation tasks? In other words, how does the author come up with the idea of ​​applying this technique to this task? The author lacks some analysis on this key point.

W3&Q3: The motivation of progressive structure adaptation (3.3-3.4) is not clear. Why progressively increasing support samples can deal with domain shift problems ?

Q4: The author uses FIM to find important parameters. I would like to know if there are any rules in the location and importance of these important parameters? (For example, certain parameters are always important in all situations)

1. Feature selection techniques are very common in machine learning and also represent a standard paradigm. While using the Fisher score to measure the importance of neural network parameters is beneficial, it remains a straightforward application and lacks innovation.
2. Although the authors' model achieves excellent results on multiple datasets, it is not state-of-the-art, and there is a lack of corresponding model comparisons in Table 1, such as IFANet (CVPR 2024).
3. Although the authors provide a comparison of the trainable layer information structure distribution in Figure 3, there is a lack of detailed explanation regarding what the input 'x' and output 'y' represent, along with the corresponding spatial dimensions, which is confusing. For example, does 'x' refer to the feature map of a single layer or a specific convolutional parameter? Does 'y' refer to the corresponding logit or something else? However, this lack of clarity detracts from the overall understanding of the results.
4. Table 4 presents informative structure identification (ISI) with different structure Fisher scores. It would be beneficial to supplement this with the selection of different informative structures using (TOP-K) or the Fisher scores that are discarded, to more intuitively measure the differences and effectiveness of this method.
5. The authors used support images as pseudo-query images to form support-query (pseudo-support) image pairs, and then they progressively increased the number of support samples to construct training pairs. However, the authors lack an explanation of why support images are used as pseudo-query images instead of directly using query images. Additionally, are the support and query (pseudo-support) images the same during training?

[1] IFANet: Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining

• ISA could be more clearly distinguished from existing approaches, particularly other CD-FSS methods like PATNet and RD. Comparisons with traditional Fisher Information-based parameter selection techniques would further clarify ISA's unique advantages for CD-FSS-specific challenges.
• The rationale for selecting specific layers for adaptation could be expanded, especially in low-support scenarios. It may also be beneficial to explore whether PSA (Progressive Structure Adaptation) could dynamically adjust layer selection based on domain properties, further enhancing the model's adaptability.