Semi-Supervised Semantic Segmentation via Marginal Contextual Information

Thank you for review and for finding our approach interesting. Let us address some of your concerns in this comment, and we will update the paper accordingly during the discussion period.

---

Q1: Most of the improvements are marginal (less than one percent) in the provided tables (and within the error range)

A1: We are adding the empirical variance of our method and the baseline over PASCAL VOC 12; for 366 labeled examples, the standard deviation is 0.2 mIoU, meaning the improvements are significant compared to the error range. It is worth reiterating that our method improves without additional parameters or loss components. Previous works often show similarly-sized improvements, partly due to the saturation of benchmarks.

---

Q2: The paper needs to showcase more segmentation results in the qualitative section because it is not quite clear how the performance is like from only two sets of samples (Fig3).

A2: A larger set of qualitative results are displayed in Figure H.1 in the appendix. In addition, we will add qualitative results from predictions on COCO in a low annotation regime.

---

Q3: The correspondence of the qualitative result to the experiment "partition" is not obvious

A3: All qualitative results shown are from Pascal VOC 2012 with 366 labeled samples. Thank you for pointing out that issue; we will add clarification in the paper.

---

Q4: The inclusion of neighboring pixels may affect the segmentation in the boundary of the object, and it may cause artifacts, as can be seen in Fig 3. The authors have not discussed how they would tackle/minimize this problem.

A4: As reported in reply to reviewer P9tM, we indeed observe degradation of Boundary IoU while using S4MC, from 19.3 to 18.2 mIoU on FixMatch using 183 annotated images from PASCAL VOC 12. Possible solutions can include edge detection to give less weight to pixels near boundaries, which is part of our future work.

---

Q5: Most given samples in the figures contain one object and the background. How does the performance of a complicated background exist with multiple-depth facades?

A5: Our method, with PLR in particular, aims to solve the ambiguity between all possible classes at the pixel level. Figure H.1 in the appendix contains interesting showcases, where objects from different classes are bunked together, such as the dog under the table. In addition, since PASCAL has only 20 object classes, we conducted experiments over the COCO dataset, which comprises 80 classes, meaning more finely annotated objects are densely placed. Besides the experiments, we will add more visual results over COCO to the appendix to qualitatively show the improvement using S4MC in dense scenes.

We will happily answer additional questions; if we address your concerns, we hope you will reflect it in your final rating.

Thank you for your review! We are happy that you appreciated the simplicity and effectiveness of the proposed method. Here, we will address the points you raised in your review, and we will update the paper accordingly later during the discussion period.

---

Q1: Experiments do not report variance across different subsets of labeled/unlabeled images.

A1: Semantic segmentation experiments are computationally heavy; segmentation SSL methods do not report the variance. We agree that the variance learning may indeed be high, so we run three random seeds for Pascal VOC 2012; for 366 labeled examples, the standard deviation is 0.2 mIoU. We will report it in the paper. Adding variance to every experiment is, unfortunately, beyond our computational resources, especially for large datasets.

---

Q2: Can you decouple improvement due to threshold \gamma_t being applied to union-level confidence from the improvement due to using union-level predictions in the loss (3)?

A2: The union-level predictions only affect pseudo-labels. The predictions used for the optimization process are not affected by the refinement mechanism. We will make sure to be clear in the paper about where we use the refinement module.

---

Q3: Comparison with UniMatch is difficult due to different experimental setups. The authors do not explain the reasons for reproducing UniMatch performance instead of copying the numbers from the original paper.

A3: We reproduced UniMatch results using the official implementation and the experimental configuration described in the paper; the results are close to the paper. In some experiments, e.g., PASCAL + SBD, not all data partitions are available, so for a fair comparison, we used the same random split for us and UniMatch. This may have caused a mismatch between our and reported results.

---

Q4: Why do the blue graph and the orange graph in Figure 4b converge at the end of training?

A4: Fig. 4 is generated using a single training run that uses S4MC, i.e., the training is performed using S4MC; yet, at each iteration, we calculate the metrics displayed in Fig. 4 using two methods. This allows us to evaluate marginal improvements at each point in time rather than cumulative ones. As a result, when all pixels pass the threshold, the accuracy is the same. We will add clarification to the paper.

---

Q5: Explain the difference between the best numbers in Tables 4a/b and Table 5.

A5: These tables use different setups for ablation. Table 4 refers to ablation on neighborhood size and $\alpha_0$ with CutMix-Seg+S4MC on PASCAL VOC 2012 with 366 labeled samples. Table 5 reports results on Fixmatch+S4MC on Pascal VOC 2012 with SBD and 2646 labeled examples. We added this information to the table captions.

---

Q6: Report minimal hardware requirements (GPU RAM) and computational budget (GPU days) for reproducing the experiments

A6: On 8 RTX A6000 GPUs, Pascal VOC 2012 with 366 labeled samples takes an average of 24:34 hours to train with DeepLabV3 with ResNet101; with ResNet50, it takes 12:11 hours to train on average. The minimum requirement is 48GB GPU memory for a batch size of 1.

---

Q7: Comparison with CutMix-Seg is difficult due to different backbones and different segmentation architecture.

A7: We are comparing to CutMix-Seg using DeepLabV3+ResNet101; the results of CutMix-Seg are from “Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels” (U2PL), which used identical architecture.

---

Q8: Can you provide a comparison with previous work under their original experimental setups?

A8: We have provided all compared methods using their original setup except for CutMix-Seg, for which we used results from U2PL to use the same architecture for the different baselines.

---

Q9: Can you confirm that the supervised baselines for all approaches in Tables 1-3 are equal?

A9: Yes, the supervised baseline for the different methods is the same.

---

Q10: Report experiments with ResNet-50 to reduce environmental impact and to allow reproduction on modest hardware.

A10: We will add results on Pascal VOC 2012 with ResNet50.

---

Q11: Consider correcting "coarse PASCAL" as "augmented PASCAL."

A11: Thank you for the suggestion. We will correct it.

---

Q12: Consider rephrasing the term "information gain" since information gain is often considered as a synonym to KL divergence

A12 Thank you for noting; we will replace the term “information gain” with information utilization.

---

Q13: Improve descriptions of the related work. For instance, the sentence with "unreliable prediction" fails to describe the gist of (Wang et al. 2022).

A13: We will rephrase the descriptions of the related work. In this example, we will use “unreliable pseudo-labels,” following Wang et al.

We will happily answer additional questions; if we address your concerns, we hope you will reflect it in your final rating.

Edit: The previous version of this message contained a mistake.

All our results involve the same version of the model. The star denotes that the results are from our local run of the reference code of UniMatch. The missing star from Table 4 was a typo, which we will fix. Hope this clarifies the issue and convinces you the comparisons are all fair.

Thank you for your review and your positive feedback! In this answer, we will address the points you raised in your review, and we will update the paper accordingly to all the reviews later during the discussion period.

Q1: The model may mispredict the majority of interesting regions (e.g., labeling a sofa as a chair or a car as a bus), rendering it unable to refine its predictions with neighboring pixels.

A1: Our method indeed relies on at least partial success of the refined semantic segmentation; it can’t and shouldn’t fix misclassifications but rather refines the probabilities to improve the pseudo-labeling. Thus, while the described failure mode is possible, we believe tackling it is outside of the scope of this work.

---

Q2: In the case of boundary regions, the neighboring pixels may exhibit similar confidence values (lack of confidence). In such instances, the reviewer considers that the proposed method may not perform effectively in these areas.

A2: The proposed method does not introduce improvements that tackle the problem of boundaries in particular; as such, the problem of low confidence in boundaries remains. Using the Boundary IoU [1] metric, we observe a degradation from 19.3 to 18.2 compared to FixMatch using 183 annotated images from PASCAL VOC 12. Importantly, our method capitalizes on the advantages of spatial continuity to enhance the performance as measured by the mIoU metric. We believe that the overall benefits our method brings to spatial coherence significantly outweigh the minor compromises in pseudo-label quality along boundaries.

---

Q3: Additionally, it would be beneficial to conduct another ablation study involving the propagation of pseudo-labels based on a k-NN (k-Nearest Neighbors) propagation algorithm with various pixel selection strategies, such as including all neighboring pixels (after filtering out those with low confidence) or other strategies.

A3: We appreciate your insightful observation. We are actively conducting experiments to explore the use of kNN with deep features or prediction probabilities to identify the optimal selection strategy for neighboring pixels. Since our current method chooses neighbors that enhance the confidence of each class, something we cannot do with kNN, the forthcoming results from our experiments may yield valuable insights, potentially uncovering new perspectives and avenues for improvement.

---

Q4: The proposed method is not working well for the evaluation set. What are the author's reasonable explanations for this?

A4: Could you clarify which evaluation set you have in mind and what you mean by “not working well”? S4MC consistently improves underlying baseline methods and shows state-of-the-art results in many settings. Indeed, for ’coarse’ PASCAL VOC 2012 val, S4MC results are worse than PCR, but this can be attributed to the significant advantage of PCR compared to methods used for S4MC and not the poor performance of S4MC itself.

---

Q5: The refinement process appears to be ineffective in the later training period, as indicated by Figure 4-b. What is the reason for this phenomenon?

A5: Since we utilize DPA during later training stages, the share of pixels used for pseudo-labeling increases until it achieves 100% at the end of training. This does not inherently improve classification accuracy; instead, it merely changes which examples make the cut for training. While there are fewer examples added by the refinement process in later stages, these examples, on average, carry more information since they were severely misclassified; it is hard to judge whether this is more or less efficient. Of course, when all pixels pass the threshold, refinement has no effect.

We will happily answer additional questions; if we address your concerns, we hope you will reflect it in your final rating.

---

[1] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C. Berg, Alexander Kirillov: Boundary IoU: Improving Object-Centric Image Segmentation Evaluation, CVPR 2021.

Thank you for your review! We are happy to know that you enjoyed reading the paper. We would like to answer some of the concerns and clarify the points raised in weaknesses. We will update the paper accordingly to all the reviews later during the discussion period.

Q1: I would suggest changing the main figure of the paper (the elements within the figure are way too small and hard to follow).

A1: We appreciate your feedback and have increased the size of the elements in the main figure for better clarity.

---

Q2: The contribution is not groundbreaking. The +1.29 mIoU gain on PASCAL VOC 2012 and +1.01 mIoU improvement on Cityscapes declared at the beginning of the paper are not backed up by the numbers in the table (Table 1 and Table 3). The numbers in the tables show that the method is not robust enough to offer a consistent improvement in all tested scenarios.

A2: We agree with this observation; however, we do not believe that being groundbreaking should be a criterion for acceptance. Our paper offers meaningful advancements by incorporating an auxiliary approach that enhances existing approaches to semi-supervised semantic segmentation. We appreciate you highlighting the typo regarding the mIoU gains on PASCAL VOC 2012; we will correct this and reaffirm that Table 3 reflects a +1.01 mIoU improvement on Cityscapes with our method.

The figure of +1.01 mIOU improvement on Cityscapes with 186 annotated images can be seen in Table 3 (75.99 mIoU for UniMatch and 77.0 for S4MC).

The figure of mIoU improvement on Pascal VOC 2012 with 366 annotated images, as shown in Table 1 (77.7 UmIoU for UniMatch and 79.09 for S4MC), is +1.39; there has been a typo in the abstract, and we thank you for pointing it out.

---

Q3: The biggest weakness of the paper is Section 4.3 (the ablation studies) and Tables 4 & 5. The text states that the experiments were conducted using the CutMix-Seg framework, but I could not find the numbers in the previous tables. Also, in Table 5, the caption states that the numbers are for FixMatch. The text, the numbers, and the tables do not correspond; this part needs further clarification (or another check) because it confuses me the most.

A3: We agree that this part may appear confusing, and we will improve the presentation in the paper. As reported in Section 4.3, ablations of Table 4 were performed on PASCAL VOC 2012 with 366 labeled examples (1/4) using CutMix-Seg+S4MC; the best-performing setting achieves 75.41 mIoU; the same number appears in Table 1. Table 5 ablations were performed on Pascal VOC 2012 with SBD examples and 5291 labeled examples; our result in Table 5 matches the corresponding result in Table 2. The vanilla FixMatch result (first row of Table 5) is indeed wrong; the correct number is 77.46 (as in Table 2), and we will update it. We will also update the captions of the referred table to avoid confusion.

---

Q4: Low range of datasets, more experiments that include more varied and challenging scenarios to better understand the method's limitations.

A4: We have extended our experiments to include the COCO dataset, reinforcing the robustness of our method with improvements in state-of-the-art comparisons across 4 out of 5 dataset partitions. Additional details regarding the performance variance will be provided in the revised manuscript. We would like to emphasize again that our method enhances existing SSL approaches. COCO is not an established benchmark for semi-supervised semantic segmentation due to its complexity: it has 80 classes over 118k images, and our approach achieves a gain of 1.5 mIoU using only 1/256 annotated data.

---

Q5: There are no insights as to why the best window for the used contextual information is in a 3x3 range - this actually suggests that the context is not used properly, or what is actually causing this degradation in performance when more neighboring pixels are used?

A5: Aggregating long-distance relations in a visual task is usually embedded into the architecture, such as dilated convolution and attention mechanism. Our approach tries to compensate mostly for the lack of spatial continuity in pseudo-labels. Since we resize PASCAL images to 321x321, looking at close neighbors is sufficient. It is likely that for higher resolutions, a larger neighborhood is required.

We will be happy to answer additional questions; if we managed to address your concerns, we hope you will reflect it in your final rating.