SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation

We sincerely thank the reviewer for their time and efforts in reviewing our work and providing valuable feedback that can further strengthen our manuscript. Below please find our detailed responses:

Results on high resolution dataset.

---

Due to page limits, we chose three widely used benchmark datasets for our experiments, i.e., ADE20K, COCO-Stuff-10K and PASCAL-Context. Following your comments, we have conducted several experiments on the CityScape dataset. FLOPs and Latency are calculated with the input resolution of 1024x2048 . We will add more baselines in the revised version.

As shown in the table, SSA also has a significant improvement in mIoU on the Cityscape dataset. For example, when SeaFormer-B is used as the baseline, SSA possesses an mIoU boost of 1.8 and does not affect the efficiency of the model. This is due to the fact that SSA mainly performs convolutional operations on the prototype and is less affected by the image resolution. Therefore, the efficiency and performance is more prominent when applied to high-resolution images.

memory usage comparison

---

Memory usage metrics have received little attention on segmentation tasks. For example, recent segmentation methods including SegViT (NeurIPS2022), CGRSeg ( ECCV2024), PEM (CVPR2024), SeaFormer (ICLR2023), and EfficientViT (ICCV2023) all do not give memory usage. Therefore, for ease of comparison with the previous method, we do not give the memory usage. In addition, to answer your question, we have done a simple test using SeaFormer-L as the baseline, and the required memory is 4479M and 4511M respectively. This further verifies the relevant content of lines 247-250 of the paper. We will subsequently count the memory usage of the model under more baselines and add them to the revised version.

In addition we have also added the comparison of parameters as shown in table 4 of rebuttal. In conclusion, the extra parameters, FLOPs, Memory and Latency brought by SSA are all negligible.

Input image resolution for training and inference

---

The resolution of the input images and the experimental settings are described in lines 523-535 of the Appendix. Specifically, when training, for ADE20K and COCO-Stuff-10K, we have a cropping size of 512 × 512, while for PASCAL-Context, we have a cropping size of 480 × 480. When inference is performed, we use whole-image inference mode. We'll add it to the main paper later.

Detail of teacher classifier

---

The architecture of the teacher classifier is shown in Fig. 2 and details can be found in the caption of Fig. 2. Specially, the architecture of the teacher classifier is identical to the main classifier, with the difference that it takes the truth mask $M_g$ as input, while the main classifier takes the coarse mask $M_c$ as input. We will add a more detailed description for lines 198-202 of the paper in the revised version.

Code of SSA. We download the supplementary material and confirm that the code is included.

Redundant expressions. We will streamline the expression in the revised version with your comments.

Comparison with mask-level classification models and parameters comparison. please refer to global rebuttal.

Experiments or references to support the statement. We provide more graphic illustrations in Fig.1 of rebuttal. In addation, a simple hypothesis can help you to solve this confusion. If pixel features of the same class do not exhibit large intra-class variance in the training and test sets, then the model should achieve very high accuracy on the test set. However, experimental results show that the baseline only has an mIoU of 42.4, and the IoU of door category is only 46.9. SSA can somewhat target this problem to achieve improved segmentation performance (mIou 45.4, IoU 49.7).

Explanation of the Fig.1. Your understanding is accurate. We provide a prediction result of the baseline as shown in Fig.2 (a) of rebuttal,which shows that many pixels belonging to door are misclassified. This is consistent with the visualization results of t-SNE. Your statement that the prediction should include many pixels of other categories classified as "doors" is not accurate. This is because they may be closer to the prototypes of other categories than to the prototypes of doors. In particular, due to the lack of constraints, the prototypes of some categories are more similar, as shown in Fig.3 of the paper.

t-SNE of the result. The schematic diagram facilitates us to describe the technical route of the whole approach, i.e., semantic prototype adaptation (Fig. 1(c)) and spatial prototype adaptation (Fig. 1(d)).We also provide t-SNE and prediction for SSA in Fig.2 of rebuttal, which show that SSA indeed facilitates the offset of the fixed prototype towards the semantic domain center and the spatial domain center of the test image.

Overfitting to the current dataset. I'm in disagreement with you. As far as I know, with the exception of multimodal segmentation work (e.g., SAM), current semantic segmentation methods (such as PEM (CVPR2024), and CGRSeg (ECCV2024)) are generally trained and tested on a single dataset. They also essentially fit the current dataset. Note that we significantly improve the performance of various state-of-the-art baseline models on all three commonly used semantic segmentation benchmarks, which validate the practical application value.

Apply method to cross-domain segmentation tasks. It is not the focus of this article, but we will apply it to cross-domain segmentation tasks in future work.

Related works is brief. We will add more descriptions of related work in the revised version.

where is the analysis. The analysis is in Sec 3.1, i.e., the inadequacy of the vanilla pixel-level classifier (feature deviation in the semantic domain and information loss in the spatial domain) .

Experiments to support line 150-151. We provide an example as shown in Figure 3 of rebuttal.

Content does not match the formula. We have verified that there is no mismatch.

Explanation of the prototype offset. I think you're misunderstanding the offset. In your terms, Sec 3.2.1 essentially just integrates features similar to class prototypes. however, the weights of the original fixed prototype S do change through this integration of features, and there is no problem with interpreting this as an offset of the prototype. In particular, we constrain this offset process through semantic domain distillation, which ensures that the prototypes are shifted towards the center of the semantic domain.

Premise of section 3.2.2. Contextual modules such as ppm and aspp utilize spatial information to enhance the feature representation of pixels through contextual modeling. And SSA models the spatial information of semantic objects through spatial prototype adaptation and in turn influences the similarity between pixels and prototypes to optimize the classification decision process. They are different technical routes and can gain from each other, as shown in Table 1 of the paper.

Ablation of position basis. Please refer to Table 2 of rebuttal.

Different instances can be far apart. We have considered this before, but three aspects ensure that SSA proposes the design of SPPA.1, in the semantic segmentation dataset, only one instance for a category occurs in most of the pictures (especially the less-sample categories, such as door and sidewalk in Fig. 5 of the paper).SSA is better able to bring in the structured information of these objects.2, considering the occurrence of the same category in the pictures of different instances, we implement SPPA based on CPVT. it models spatial prototypes as relative segments rather than absolute anchors. The detailed analysis can be found in lines 314-321 of the paper.3, Ablation experiments verify the effectiveness of SPPA.

Relation of the methods and problem. I'm confused about this. The problem we are posing is that vanilla pixel-level classifiers suffer from feature deviation in the semantic domain and information loss in the spatial domain. And we mitigated that problem with semantic prototype adaptation and spatial prototype adaptation. We believe that the relation between the problem and the method is highly relevant.

Performance with self-distillation. Please refer to Table 3 of rebuttal. We argue that the prediction mask is not accurate, and using it as an input to the teacher classifier will introduce incorrect category features and noise.

performance gains not align with motivation. Please refer to Table 5 of rebuttal, which shows that simply increasing the size of the model does not result in such a large performance gain. As for the loss function, it was proposed to serve the structure of the model, which is an inseparable part of SSA and one of the contributions of this paper.

We sincerely thank the reviewer for their time and efforts in reviewing our work and providing valuable feedback that can further strengthen our manuscript. Below please find our detailed responses:

Ablation of loss

---

We retain by default the cross-entropy losses $L_{ce}$ and $L_{dice}$, the former is an basic loss function for semantic segmentation and the latter commonly used in mask classification models for mask generation. We focus on ablation analysis of online multi-domain distillation loss in the paper, which is one of the important contributions of the paper. We have also done the ablation as shown below:

we observe that when removing $L_{dice}^c$ or $L_{dice}^g$ (the cross-entropy loss cannot be removed), the mIoU slightly decreases, i.e., from 45.36 to 44.91 and 45.06 respectively.

Adapt to mask-level methods

---

As pixel-level classifiers, SSA and mask classification methods belong to two different technical routes. However, SSA is lightweight enough, thus has the potential to be combined with mask classification methods. For example, SSA can be embedded behind a mask classifier to further refine the pre-classification results obtained by Mask2Former. We will explore better adaptation method in our future work to further improve the range of SSA applications.

Sec3.2.2(SPPA) assumes that objects of the same class should all be in a specific region of the image

---

Thank you for your response. We have responded to reviewer bEdU in response to your question. However, due to the character limit, it is possible that our answer was not as convincing as it could have been. Here we provide a more detailed answer.

• First, we do not assumes that objects of the same class should all be in a specific region of the image in our paper. This comment is unfounded and reviewer bEdU may not have fully understood our method. Although SPPA proposes the concept of spatial domain centers, our spatial domain centers are obtained based on the Conditional Positional Vocoding (CPVT) [1], which represents for relative segments rather than absolute anchor points. Therefore, our spatial domain center models the collection of information about neighboring segments belonging to the same object, rather than an absolute one point. We have a corresponding analysis in lines 314-321 of the paper. In other words, it enables the model to indirectly take into account the semantic features of pixels at neighboring locations when classifying them. More specifically, you can refer to CPVT [1], and we believe you can understand how we avoid modeling absolute spatial anchor points.

• Second, many images in ADE20k and COCO-stuff have the same category scattered in different areas. Nevertheless, our classifier SSA can still significantly improve the performance of the model on these datasets. This is because SPPA takes into account the spatial structure information. We have conducted ablation experiments on SPPA in the paper, as shown in Table 3 of the paper. When SPPA and distillation loss are applied, the model accuracy increased by 2.12% mIoU. This validates the effectiveness of spatial domain adaptation.

We hope you and AC are not misled by this erroneous comment.

Clarifying the motivation

---

As described in lines 3-8, 29-40 of the paper, our motivation is clear, i.e., the vanilla softmax classifier uses the inner product of pixel features and fixed prototypes to generate segmentation masks, leading to feature deviation in the semantic domain and information loss in the spatial domain problem. Therefore, we introduce semantic prototype adaptation and spatial prototype adaptation, adjust the fixed prototype to the center of the semantic and spatial domains in the test image, and then consider both semantic and spatial domains of the adaptive prototypes to complete the classification decision, which effectively mitigates the above problems.

Providing evidences of large intra-class variance with pixel features

---

We provide more graphic illustrations in Fig.1 of rebuttal. Specially, Fig.1 shows that the t-SNE of some example images, which are randomly selected from the ADE20K dataset. The first row represents the distribution of pixel features in the door class, and the second row represents table class. it can be observed that due to the complex scenarios and varying object distributions, pixel features of the same class tend to exhibit larger intra-class variance when the trained model on the training set is applied to the test set.We will add these graphic illustrations and a more detailed analysis to the revised version of the paper.

SSA+Mask2Former

---

We have further validated your confusion. Specifically, after we embedded SSA into Mask2Former's mask classifier, the mIoU of the model increased from 47.2 to 48.5. This shows that SSA is able to adapt to existing mask classification methods.

Conclusion

---

We have responded to each of the reviewer bEdU's comments. If you have additional questions or find one of our responses unconvincing, please do not hesitate to point it out. We look forward to further discussions with you.

[1] Conditional Positional Encodings for Vision Transformers (ICLR 2023)