A Surprisingly Simple Approach to Generalized Few-Shot Semantic Segmentation

Thank you for your review. Please find our answers to your questions below.

The overall novelty is relatively limited, as the idea that "a novel class is classified as the background or a similar base class by the base-class model" has already been explored in continual semantic segmentation [1], where novel class "lake" is classified as base class "water" before learning the new class lake.

Thank you for pointing out the paper [1]. In [1], a novel class is classified as a base class, and then misclassification is corrected by post-processing, so-called logit manipulation. In contrast, we explicitly integrate the idea into the model architecture, which vastly differs from [1]. The architecture allows us to see which novel class is related to which base class through BNM, as illustrated in Figure 2(c). Moreover, we showed that our method prevents catastrophic forgetting in theory (Proposition 4.1) and improved performance, particularly in the 1-shot setting. Although the two settings (continual learning in panoptic segmentation [1] and GFSS) have some overlap, achieving better performance by [1] in GFSS would not be straightforward since the number of training samples is quite limited in GFSS, particularly in the 1-shot setting. We believe that our paper contains various quantities that contribute to the overall novelty.

When writing our paper, we did not realize [1] since it does not consider GFSS, and it was published very recently in CVPR2024 and appeared on arXiv roughly a month before the submission deadline. In the final version, we will cite [1].

The improvement across different settings is relatively marginal.

The improvement of the Novel score in the 5-shot PASCAL-$5^i$ is marginal, but except for that, our method improved the Novel score by at least 1% and a maximum of 6%. From the viewpoint of inference time, our method is much faster than DIaM, as shown in Figure 7. Considering these points, the results of experiments would be sufficient to show the effectiveness of the proposed method against the existing methods.

The computational cost is about 1.5 folds compared to CAPL as in Fig. 7. From my view, the base model is shared among different g_beta. The only computation increase should only be the added final linear layers, which is roughly neglectable. Therefore, I expect more explanations for the increase.

Our method has $|\mathcal{B}|$ final linear layers for novel classes, leading to a subtle slowdown when switching the layers, unlike the end-to-end computation of CAPL. Also, our current implementation uses CPUs for the final linear layers, as Scikit-learn is used, unlike CAPL on GPU. This device difference might be another reason for the slowdown. Besides, other miscellaneous things may cause about six milliseconds of slowdown. Nevertheless, the current implementation would show the usefulness of our method. In the final version, we will mention the above points and explain that a more sophisticated implementation will shorten the gap between CAPL and our method.

When s > 1, each novel class may have multiple mapped base classes. In this case, how to combine the final predictions for the novel classes is not elaborated.

Even when s > 1, the inference procedure works as it is since a single base class is mapped to multiple novel classes in BNM.

We show the case when s > 1 with a tiny example: the base classes are 0 and 1, and the novel class is 2. Suppose that we have the following BNM when s = 2:

This table shows when the novel class 2 is mapped to two base classes, 0 and 1. In this case, we have the two models: $g_{\beta=0}$ returns 0, or 2, and $g_{\beta=1}$ returns 1 or 2. For each pixel, the base-class model outputs either 0 or 1. We then compute the prediction of the corresponding model $g_\beta$ and overwrite it. Since our method does not need to overwrite the same pixel multiple times, we can straightforwardly combine predictions of $g_\beta$ for the final prediction. To improve the clarity of our paper, we will add the above explanations in the final version. We hope that this answer will resolve your concerns.

I would like to see the comparison with the few-shot fine-tuning techniques proposed in FSOD [1-2], i.e., finetuning the last linear layers. Adding these two simple methods to the baseline will make the experimental part more convincing.

Moreover, I suggest the authors briefly discuss the limitations of continual CSS [3-5] when it comes to GFSS in the related works.

I will finalize my rating based on the authors' responses.

[1] Wang, Xin, et al. "Frustratingly simple few-shot object detection." arXiv preprint arXiv:2003.06957 (2020).

[2] Yang, Ze, et al. "Efficient few-shot object detection via knowledge inheritance." IEEE Transactions on Image Processing 32 (2022): 321-334.

[3] Cermelli, Fabio, et al. "Modeling the background for incremental learning in semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[4] Yang, Ze, et al. "Label-guided knowledge distillation for continual semantic segmentation on 2d images and 3d point clouds." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Kim, Beomyoung, Joonsang Yu, and Sung Ju Hwang. "ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

I would like to see the comparison with the few-shot fine-tuning techniques proposed in FSOD [1-2], i.e., finetuning the last linear layers. Adding these two simple methods to the baseline will make the experimental part more convincing.

DIaM fine-tunes the last linear layers, regarded as the simple baseline. In Figure 1(a) in the DIaM paper, they compared the method that fine-tunes the final linear layer with the cross-entropy, where the procedure is close to the few-shot object detection method [1]. The results showed that DIaM outperformed the simple baseline, meaning that our method will outperform such a baseline.

In the final version, we will discuss [1-2] in the related work.

Moreover, I suggest the authors briefly discuss the limitations of continual CSS [3-5] when it comes to GFSS in the related works.

Thank you for your suggestion. We will discuss it in the related work.

Regarding s>1, only the top-1 prediction from the base model will be the index to select corresponding to output the prediction, which will be used to overwrite the base prediction. Is it correct?

Yes. The top-1 prediction of models is used. The symbol $s$ is for the top-$s$ strategy in our method. We will clarify this point.

Thank you for your feedback. Please find our answers to your questions below.

It should be clarified whether the co-occurrences matrix and BNM are calculated per-dataset or per-batch.

The current implementation computes BNM per dataset. We will clarify this point in the final version.

Since the proposed method uses feature pre-processing and model ensembling techniques, the comparison may be somewhat unfair.

We presented the ablation study, showing that our method without feature pre-processing and ensemble learning techniques achieved better novel-class segmentation performances in the 1-shot setting.

The existing methods often use meta-learning, information maximization principle, and transductive learning in addition to the simple supervised learning method, e.g., minimization of the cross-entropy loss. Each existing method uses various techniques to improve its performance further. Given that, our comparison is reasonable from the viewpoint of standard practice.

From another perspective, the advantage of our method is that we can use feature pre-processing and model ensemble learning techniques at a low cost, as shown in Figure 6 (training time). Pre-processing to feature vectors, rather than input images, would downgrade the performance of the existing methods. Ensemble learning with the existing methods causes computation issues. For instance, ensemble learning further slows down the speed of the existing methods based on transductive learning, such as DIaM.

The written language can be improved.

We will revise the written language.

Typo: line 43, "beneficial, especially."

Thank you very much. We fixed the typo.

Thank you for your feedback. It appears you have listed some of the strengths in the weaknesses section. If you forgot to copy and paste questions from your memo, e.g., questions about explanations in our paper, please let us know during the discussion. We would like to improve the presentation of our paper through your comments.