Connectivity-Driven Pseudo-Labeling Makes Stronger Cross-Domain Segmenters

Thank you for recognizing our work and providing constructive feedback.

Q1: SCC vs DivideMix

Although our SCC is partly inspired by DivideMix, it is different from DivideMix from four aspects.

Address Different Tasks:

• SCC: Cross-domain semantic segmentation.
• DivideMix: Noisy labels learning for image classification.

Solve Different Problems:

• SCC: We focus on mitigating the pixel-level noises in pseudo-labels raised by domain shifts and SAM refinement.
• DivideMix: It focuses on mitigating the image-level label noises.

Different Motivations:

• SCC: We aim to convert pixel-level denoising to connectivity-level denoising, which takes the advantage of context information (relationship among neighbouring pixels) and thus is more robust to noises.
• DivideMix: It aims to effectively address the image-level denoising.

Different Implementations:

• SCC: 0. Use pooling to treat aggregated pixels as connectivity.
  1. Use early stop and GMM for filtering and correcting connectivity with noise.
  2. Use a fixed threshold to assign pseudo-label for selected connectivities.
  3. Use connectivities with pseudo-labels to train the model.
• DivideMix:
  1. Use co-divide-based GMM to divide data.
  2. Use co-guess for pseudo-label correction.
  3. Use mixmatch for semi-supervised learning.

To better verify the effectiveness of our SCC, we add two experiments: a) directly applying DivideMix to pixel-level denoising. b) using DivideMix to denoise the pixels aggregated by our PSA (using SAM). The results show that pixel-level denoising methods based on DivideMix are ​​inferior to SCC even with SAM, highlighting the advantage of denoising at the connectivity level.

To this end, we hope Reviewer pA2t can find that our SCC is different from DivideMix and that the proposed connectivity-level denoising is a key step in our full framework.

Q2: Related work on noisy label learning.

Good suggestion. We add a discussion on Noisy Label Learning(NLL). Currently, NLL focuses on classification tasks with techniques like robust loss design [1], regularization [2], label weighting [3], and correction [4]. These methods typically target image-level noise and may not be effective for pixel-level segmentation, which involves complex spatial and semantic dependencies among pixels. Maintaining spatial consistency across millions of pixels is a challenge for current image-level denoising methods. In segmentation, few methods focus on denoising the pseudo-label, such as ProDA [84] and RPL [90], which denoise each pixel independently and still face the challenges highlighted in our paper. Our SeCo effectively links image-level techniques with segmentation tasks, offering novel solutions for pseudo-label denoising in segmentation. In the future version, we will add more related work to this section.

Q3: DG experiment.

In this experiment, we followed CLOUDS (CVPR'24), which uses synthetic data from Stable Diffusion(SD) to assist DG. Our aims are: 1) to show that our method can handle challenging synthesized data with open-set noise, and 2) to compare our SAM usage with the competitive scheme in CLOUDS. We agree with your point that using synthetic data from SD does not constitute a strict DG setup. In the revised version, we will clarify that this experiment is to assist DG rather than adhering to a strict DG setting.

Q4: The details of CLIP+SAM in Line 248.

For CLIP+SAM, the steps are: 1) Use SAM to segment the input image. 2) Extract the largest bounding rectangle from each segment. 3) Create text descriptions for categories, e.g., "a photo of a road." 4) Use CLIP to match image patches with text descriptions, assigning text labels as categories. CLIP+SAM+UDA combines pseudo-labels from CLIP+SAM and UDA, using voting fusion to select consistent predictions for training. These details will be added to the main text.

Q5: More detailed ablation on SCC.

We conduct a detailed ablation study on SCC across multiple tasks in GTA $\rightarrow$ Cityscapes, evaluating it with two metrics: PL mIoU (pseudo-label quality on the training set) and Val. mIoU (model performance on the validation set). The results show that: 1). removing the filter function causes an average loss of 4.1% mIoU in pseudo-label quality and 1.8% mIoU in validation performance. 2). removing the correction function causes an average loss of 3.6% mIoU in pseudo-label quality and 1.2% mIoU in validation performance. This shows that both modules are crucial for reducing pseudo-label noise and improving model performance.

[1]. Generalized cross entropy loss for training deep neural networks with noisy labels, (NIPS'18)

[2]. Robust early-learning: Hindering the memorization of noisy labels, (ICLR'21)

[3]. Meta pseudo labels, (CVPR'21)

[4]. Dividemix: Learning with noisy labels as semi-supervised learning (ICLR'21)

Thank you for your feedback, it is very helpful for us to improve our paper.

First, we would like to emphasize that one of the main contributions of SCC is to provide the idea of ​​denoising at the connectivity level, which makes it possible to apply other image-level denoising methods such as Dividemix to segmentation tasks. Building on this, we further provide the implementation of DivideMix at the connectivity level across multiple tasks, including GTA → Cityscapes (G→C UDA), SYNTHIA → Cityscapes (S→C UDA), GTA5 → BDD-100k (G→B OC-DA), Endovis17→Endovis18 (E17→E18) and Potsdom → Vaihingen (P→V).

Implementation comparison: In DivideMix, two models are first trained with different warm-up iterations. Then, each model uses different GMMs to select samples and then exchanges the selected samples (co-divide) between the two models for further co-guessing-based training with more iterations. The implementation requires careful selection of several hyperparameters, including the warm-up iterations for both models, the number of further training iterations, and the threshold for co-guessing. In contrast, SCC requires only a single model with fixed 5k early-stopping training for all tasks and then uses GMM for sample selection.

Performance comparison: The performance and training time of DivideMix at the connectivity level are shown in the table below. We observe that DivideMix performs well in tasks with lower noise rates, such as GTA → Cityscapes (UDA), where it achieves comparable performance to SCC. However, in tasks with higher initial noise rates, such as GTA5 → BDD-100k (OC-DA), SCC outperforms DivideMix. This is because higher noise rates introduce more challenges in hyperparameter tuning, particularly in choosing the warm-up epochs for the two models, e.g., close warm-up iterations fail to create sufficiently different models, while significantly different iterations can lead to one model overfitting to noise. (The table below shows results where we chose models with 2.5K and 5K iterations for both models.) This complexity hinders DivideMix from maintaining stable denoising performance in more challenging scenarios.

Overall, compared to DivideMix, we think our SCC not only introduces a new idea of denoising at the connectivity level, but offers a simpler implementation, requires less hyperparameter tuning, has shorter denoising training times, and delivers more stable denoising performance.

Thank you for your recognition of our work and your constructive suggestions.

Q1: Quantitative Analysis on "Prompting Only" and "Semantic Alignment"

We conduct ablation studies on "Prompting Only" (PO) and "Semantic Alignment"(SA) across multiple tasks in GTA $\rightarrow$ Cityscape. We provide two metrics for these detailed ablations: PL mIoU (pseudo-label quality on the training set) and Val. mIoU (model performance on the validation set after training with those pseudo-labels). As shown in the table below, the "prompting only" method reduces the quality of pseudo-labels in the training set, leading to poor adaptation performance. This is because the unreliable interaction method introduces excessive noise into the pseudo-labels generated by SAM. "Semantic alignment" improves the quality of the training set pseudo-labels, but the improvement is limited, resulting in limited adaptation benefits. In contrast, our method enhances the quality of the training set pseudo-labels through better interaction, leading to superior performance gains.

Q2: The Effect of the Enlargement Factor on the Bounding Box Area

As shown in the table below, we validate the impact of bounding box size on adaptation results across multiple UDA settings. Overall, a too-small bounding box (e.g., 0.5) causes a performance drop because it fails to encompass the "things" category adequately, hindering the effectiveness of SAM prompts. Too-large bounding boxes (e.g., 5) result in a slight performance decline due to poor delineation of the "things" category, leading to semantic ambiguity in SAM. When the bounding box size is between 1.5 to 2 times the size of the largest external rectangle, the model achieves the best adaptation performance, with minimal performance variation within this range.

Q3: The Effect of the Number of Training Iterations on the Connectivity Classifier.

We supplement our study with the impact of early-stop iterations on adaptation performance across multiple tasks in GTA $\rightarrow$ Cityscapes. As shown in the table below, both too few iterations (1K) and too many iterations (20K) harm adaptation performance. Too few iterations result in the classifier's insufficient fit to the connectivity, while too many iterations cause overfitting to noise in the connectivity. Setting the iterations between 5K and 10K provides stable denoising benefits and performance improvements.

Thank you for your recognition and your useful suggestions.

Q1. The work may be limited to a relatively narrow field.

Good suggestion. We explore SeCo's performance in more segmentation scenarios, including indoor scenes, cross-domain medical images, and cross-domain remote sensing images, as shown in the Table below. Based on these positive experimental results, we believe SeCo has the potential to be integrated into more segmentation scenarios involving the use of unlabeled data. For a qualitative visual comparison, see the PDF in Rebuttal.

1. Indoor scenes: The commonly used segmentation dataset for indoor scenes is ADE20K, but no cross-domain segmentation benchmark exists. Thus, we conduct experiments on a semi-supervised segmentation task, which also involves utilizing pseudo-labels from unlabeled data (domain adaptation is seen as semi-supervised learning with domain shift). We perform PSA using the stuff and thing definitions in ADE20K and execute SCC with default parameters. We use the Unimatch model as the baseline and follow its settings. The results of incorporating SeCo are shown in the table below. In multiple labeled data splits (1/64 - 1/8) in ADE20K, SeCo shows significant performance improvement compared to directly using SAM.
   | Indoor Scenes: ADE20K | | | | | |:-:|:-:|:-:|:-:|:-:| | Labeled data | 1/64 | 1/32 | 1/16 | 1/8 | | UniMatch [1] (CVPR'23) | 21.1 | 28.8 | 30.9 | 35.0 | | Switch [2] (NIPS'23) | 22.6 | 27.9 | 30.1 | 33.8 | | +SAM (Semantic Alignment) | 20.6 (-0.5) | 28.9 (+0.1) | 31.3 (+0.4) | 35.5 (+0.5) | | +SeCo (w/o SCC) 21.8 (+0.7) | 28.0 (+0.9) | 31.9 (+1.1) | 36.0 (+1.0) | | +SeCo (Full) | 25.1 (+4.0) | 32.4 (+3.6) | 34.6 (+3.7) | 38.1 (+3.1) |

2. Medical images: We follow the medical image UDA setup from Sim-T[3], using the Endovis17 and Endovis18 abdominal surgery datasets collected from different devices containing 3 instrument type classes. We treat the segmentation objects as "things" and aggregate pixels using only boxes and points from the pseudo-label. The table below shows how SeCo greatly benefits SAM in this challenging task. | Medical Image: Endovis17→Endovis18 | | | | | | |-|- |- |-|-|---| | Performance | scissor | needle driver | forceps | mIoU | | | SimT [3] (TPAMI'23) | 76.2 | 39.8 | 58.9 | 58.3 | | | +SAM (Semantic Alignment) | 73.0 (-3.2) | 38.3 (-1.6) | 55.7 (-3.2) | 55.6 (-2.7) | | | +SeCo (Full) | 78.4 (+2.2) | 41.2 (+1.4) | 61.2 (+2.3) | 60.4 (+2.1) | |

3. Remote sensing: We follow the UDA setup in remote sensing from the CIA[4], using the Potsdam and Vaihingen datasets collected from different satellites. These datasets contain five common semantic categories: car, tree, impervious surface, building, and low vegetation. We treat cars and buildings as "things," and the rest as "stuff." The table below shows that SeCo still achieves significant performance improvement compared to directly using SAM. | Remote sensing: Potsdom → Vaihingen | | | | | | | |:-----------------------------------:|:------------:|:------------:|:-------------:|:------------:|:------------:|:------------:| | Performance | Imp.Sur | Build. | Vege. | Tree | Car | mIoU | | CIA-UDA [4] (TGARS'23) | 63.3 | 75.1 | 48.4 | 64.1 | 52.9 | 60.6 | | +SAM (Semantic Alignment) | 61.8 (-1.6) | 70.67 (-4.4) | 50.1 (+1.7) | 66.84 (+2.7) | 50.9 (-2.0) | 60.1 (-0.5) | | +SeCo (w/o SCC) | 64.4 (+1.0) | 76.31 (+1.2) | 50.45 (+2.1) | 66.41 (+2.3) | 54.67 (+1.8) | 62.4 (+1.8) | | +SeCo (Full) | 69.4 (+6.1) | 80.5 (+5.4) | 51.9 (+2.5) | 70.6 (+6.5) | 57.7 (+4.6) | 66.1 (+5.5) |

Q2: Fairness concerns

We discuss fairness comparisons in detail in lines 185-195 of the original paper. We want to emphasize the following two points here:

1. To ensure a fair comparison, we conducted experiments without SAM, as shown in Figure 5 of Section 3.2. In these experiments, we applied our SeCo to the outputs from the source model's connectivity, not using SAM. Across multiple tasks and frameworks, this still shows a clear advantage over pixel-level self-training. This experiment demonstrates that the idea of connectivity denoising is universal. Even with less structured connectivity, it can alleviate the noise issue in self-training for semantic segmentation tasks. Moreover, well-structured connectivity can further enhance the performance of connectivity denoising.
2. Our method aims to be integrated to enhance existing pseudo-labeling methods rather than to compete with them. Results show that our method can significantly improve previous state-of-the-art baselines by adding our method to each baseline individually, which could be regarded as a fair comparison and shows the complementary of our method to these baselines.

[1]. Revisiting weak-to-strong consistency in semi-supervised semantic segmentation CVPR'23

[2]. Switching temporary teachers for semi-supervised semantic segmentation NIPS'23

[3]. SimT: Handling Open-set Noise for Domain Adaptive Semantic Segmentation TPAMI'23

[4]. Category-Level Assignment for Cross-Domain Semantic Segmentation in Remote Sensing Images. TGARS'23