Beyond Confidence: Exploiting Homogeneous Pattern for Semi-Supervised Semantic Segmentation

Thanks for taking the time to share your comments in the review assessment. We provide a detailed point-by-point response to your comments.

Note that the following link refers to https://anonymous.4open.science/r/AgScore/Rebuttal.pdf

---

Q1: Advantages of AgScore.

A1: Semi-supervised semantic segmentation is essentially a learning strategy-centric task, and its core lies in effectively filtering pseudo-labels to explore unlabeled data.

• Previous confidence-based methods tends not to be preferred ascribed to the trade-off between TPR and FPR when handling pseudo-labels. A high confidence threshold ensures pseudo-label quality (low FPR) but discards many correct pseudo-labels with lower confidence (unfavorably low TPR), and vice versa.
• We focus on the homogeneous pattern in the embedding space. Intuitively, the high-dimensional, sophisticated embedding space has greater potential for assessing pseudo-label reliability than the relatively simple one-dimensional confidence metric from the prediction space. This provides hints for assessing pseudo-label reliability in the embedding space.
• In implementation, we absorb the merits of confidence to construct clean positive and negative agents. For pixel predictions difficult for confidence to handle, we employ the agent score function to score pseudo-labels in collaboration with clean positive/negative agents, leveraging the embedding space's capability without sacrificing confidence's advantages, yielding higher TPR and lower FPR.

Experimentally:

• Fig. 1 (d) shows that AgScore achieves a better TPR-FPR balance over confidence.
• Fig. 3 depicts the better TPR and FPR dynamics of AgScore than confidence-based methods during training.

These experimental results, along with the superior segmentation performance (Tab. 1-4), provide strong evidence that AgScore reaches new heights in reliable pseudo label filtering beyond confidence.

Furthermore, we attempt to analyze from the perspective of information theory. We prove that the mutual information between the pseudo labels selected by AgScore and the ground truth is greater than that of regular confidence-based methods (Theorem G.3 in link). Therefore, AgScore theoretically improves pseudo label selection compared to the confidence metric.

In summary, from the perspectives of intuitive motivation, experimental analysis, and theoretical justification, we demonstrate the advantages of AgScore over confidence-based methods.

---

Q2: About SAM.

A2: We respectfully disagree with your viewpoint.

• Semi-supervised learning focuses on better exploring vast amounts of unlabeled data under extremely limited labeled data.
• In fact, even for powerful visual foundation models like SAM, they struggle to generalize to domain-specific scenarios, such as medical images, requiring fine-tuning to adapt to downstream tasks. In these tasks, obtaining sufficient annotations is challenging and time-consuming, making it highly valuable to explore how SAM can leverage easily accessible unlabeled data.

To further validate the value of SSL, we evaluate AgScore with SAM under the extreme 1-labeled case. As shown in link Tab. 8, we find: (1) The original SAM model struggles under the extreme 1-shot setting, underscoring the challenge of domain transfer for large vision models. (2) Fine-tuning SAM with a confidence-based pseudo-labeling strategy significantly boosts performance, demonstrating the value of SSL for adapting foundation models to specialized tasks. (3) Integrating AgScore into the SAM fine-tuning pipeline further improves results, validating the effectiveness of AgScore even with powerful vision backbones.

---

Q3: About RankMatch.

A3: Thanks for pointing this out. RankMatch proposes a sample selection strategy via a confidence voting scheme in the embedding space to increase sample selection quantity for learning with noisy labels.

However, we argue that a essential distinction is that RankMatch focuses solely on positive samples , neglecting the utilization of negative samples. Unlike RankMatch, AgScore combines knowledge from both positive and negative samples by examining the difference in embedding similarity between the candidate pseudo-label's corresponding pixel and the positive/negative agents to measure reliability, thus better leveraging the capabilities of the embedding space. Moreover, we provide a theoretical justification for AgScore's working mechanism.

Furthermore, we construct an ablation study in link Tab. 9 to verify the impact of considering only positive agents (1-st entry), demonstrating that the introduction of negative agents improves performance.

---

We hope our response can resolve your concern. Please do not hesitate to let us know if you have further questions :)

Thanks for your time and consideration. Have a wonderful day!

[1] How to Efficiently Adapt Large Segmentation Model (SAM) to Medical Images.

Thanks for taking the time to share your comments in the review assessment, as well as for acknowledging the new perspective, well-supported motivation and clear idea. We provide a detailed point-by-point response to your comments.

Note that the following link refers to https://anonymous.4open.science/r/AgScore/Rebuttal.pdf

---

Q1: Concept of Agent.

A1: Sorry for not providing you with an intuitive understanding of the term "agent". In our work, "agents" refer to the sets of pixels that are considered to have correct or incorrect pseudo-labels, serving as a bridge for evaluating the correctness of the pseudo-labels for the remaining pixels. This is why "agents" are named.

---

Q2: Independence Assumption of X and Y.

A2: We assume that X and Y are independent for the convenience of subsequent theoretical derivations. In practice, we first select the top-1% and bottom-1% confidence pixels and then randomly/orthogonally sample positive/negative agents from them. This is done to ensure that the independence assumption holds to the greatest extent. To make the theoretical derivation more general, we even consider the case where the sampled agents are not identically distributed, i.e., the Poisson binomial distribution (Sec. C in the Appendix). Overall, our assumption is mild and supported by implementation.

---

Q3: Value of our Theoretical Analysis.

A3: Our theoretical analysis is closely intertwined with our method and experiments. As mentioned above, the agent selection strategy employed in our method is designed to ensure the independence assumption in the theoretical analysis. Moreover, the conclusion drawn from our theoretical analysis, "increasing M enhances the separability between correct and incorrect pseudo-labeled pixels," is also validated by Tab. 5 in our experiments. At the same time, the experiments also demonstrate that when M becomes excessively large, the performance degrade, since the independence assumption breaks down.

---

Q4: Training and Evaluation Efficiency.

A4: Thanks for your valuable suggestion. Notably, our AgScore is only involved in the selection of pseudo-labels during training and does not introduce any additional cost during evaluation. To further quantify the efficiency impact of AgScore, we report the GPU memory and training time of AgScore and baseline UniMatch under the same setting (Pascal VOC classic, 92 partition, cropsize=513, batchsize=8, ResNet50).

We observe that AgScore brings a significant performance improvement at the cost of a slightly increased memory consumption and an acceptable increase in training time.

---

Q5: Assumptions in Theoretical Analysis.

A5: As shown in Fig. 1(c), we compute the average similarity between pixels corresponding to correct and incorrect pseudo-labels. It is evident that the similarity between pixels with correct pseudo-labels is greater than that between pixels with correct and incorrect pseudo-labels, i.e., $p_1 > p_2$ holds statistically. Similarly, $q_1 < q_2$ also holds statistically.

---

Q6: Semantic Overlap in Negative Agents.

A6: As you mentioned, when M becomes sufficiently large, there will be semantic overlap among negative agents. In this case, the most harmful negative agents are those that have semantic overlap with positive agents, as this violates the assumption that $p_1 > p_2$. Consequently, $\mu_1 - \mu_2$ increases, leading to a larger $\text{FPR}_\lambda$ (as shown in Lemma 4.3), which in turn degrades the separability between pixels with correct and incorrect pseudo-labels.

---

Q7: More Visualization of Results.

A7: As shown in link Fig. 4 and 5 , we supplement additional t-SNE visualizations for more classes on the Pascal and Cityscapes datasets. It can be observed that all classes clearly exhibit the Homogeneous Pattern. Additionally, as shown in link Fig. 7, the precision of the positive and negative agents obtained using simple top/bottom confidence is satisfactory. Therefore, leveraging relatively clean positive and negative agents, we are able to distinguish between correct and incorrect pseudo-labels.

---

Q8: Impact of Negative Agent Selection.

A8: In Tab. 6 of Appendix, we explore various strategies for negative agent selection strategy, including "Uniform" represents uniformly sampling negative agents from the candidate set, "Bottom" represents sampling negative agents in ascending order of confidence. We observe that our strategy of "Orthogonal" achieves the best results with a light computational cost.

---

We hope our response can resolve your concern. Please do not hesitate to let us know if you have further questions :)

Thanks for taking the time to share your comments in the review assessment. We provide a detailed point-by-point response to your comments.

link refers to https://anonymous.4open.science/r/AgScore/Rebuttal.pdf

---

Q1: More Evidence.

A1:

• Claim 1: To demonstrate the generalization of the homogeneous pattern phenomenon, we conduct experiments and visualizations on both Pascal VOC and Cityscapes (1/16 partition) using ResNet-50.

  (1) Single-category visualizations (Fig. 5 & 6 in link) highlight the top-4 most frequently predicted classes in each dataset. These results show that pixels with similar patterns within a class tend to share homogeneous semantics, while those from different classes exhibit distinct patterns. This supports our claim that “pixels with similar patterns tend to share homogeneous semantics compared to different patterns.”

  (2) Multi-category visualizations offer a global perspective: correctly predicted pixels (darker) and incorrectly predicted ones (lighter) form well-separated clusters in the embedding space, further validating the link between visual patterns and semantic consistency.

• Claim 2: To clarify how Fig. 1(a) is generated, we provide a step-by-step explanation in Fig. 7 (link).

  Step (1) shows the confidence distributions of correct and incorrect predictions on Pascal VOC (1/16 split).

  Step (2) defines true/false positive rates (TPR/FPR), while Steps (3) and (4) illustrate the trade-off between them under different confidence thresholds. Ground truth for unlabeled data is used here only for analysis.

• Claim 3: In Fig. 1(c) (or Tab. 7 in link), we quantitatively measure the embedding similarity between pixels with correct and incorrect pseudo labels. For each class, we compute the similarity between each predicted pixel and the sets of correct/incorrect pixels, average the results, and normalize them class-wise.

The results show that correctly predicted pixels have much higher average similarity to other correct pixels (0.925) than to incorrect ones (0.075), and vice versa. These findings provide numerical evidence that aligns with the qualitative observations in Fig. 1(b).

---

Q2: More Experiments.

A2:

• Exp 1: Agent Selection Effectiveness & Class Imbalance

  (1) We evaluate the top-1% confidence-based selection strategy on Pascal VOC (1/16 split, ResNet-50). As shown in Fig. 8 (link), the high precision of selected pixels for both positive and negative agents confirms the effectiveness of our confidence-based strategy.

  (2) Pascal VOC itself exhibits severe class imbalance (e.g., background has >200× pixels than bicycle). Cityscapes and COCO show even larger head-to-tail ratios (>400 and >10,000). Despite this, our agent selection strategy (Algorithm 1), which samples clean examples in a class-balanced manner, remains robust—consistently improving performance across datasets (Tables 1–4).

  (3) As for using labeled GT to construct agents, we’ve tested this on Pascal VOC (92 labeled images) with UniMatch as baseline (Tab. 10 in link). Due to the known distribution gap between labeled and unlabeled data (as noted in DAW, SoftMatch), agents built from GT-labeled pixels underperform compared to our confidence-based method.

• Exp 2: Design Choices for AgScore

  (1) Incorporating Negative Agents: Using only positive agents (Row 1) omits key contrastive signals, leading to inferior performance. Adding negative agents (Rows 2/3) in a proportional form introduces nonlinearity that normalizes and stabilizes similarity scores, enhancing label separability.

  (2) Exponentiation for Score Scaling: Applying the exponential function to cosine similarity (Row 3) amplifies score differences, further improving performance—aligning with our theoretical motivation.

• Exp 3: Orthogonal Selection Strategy

  (1) Our orthogonal selection strategy is tailored for negative agent construction, selecting diverse, low-confidence samples to ensure agent purity (Fig. 8).

  (2) This enhances semantic coverage and improves the separation between correct/incorrect predictions, supporting our theoretical design.

  (3) In practice, AgScore serves as a general plug-in compatible with various methods (e.g., with different loss functions), making our orthogonal selection both reasonable and flexible.

• Exp 4: Sensitivity to Selection Ratio Experiments in Fig. 9 (link) show that raising the pixel selection ratio to 2% reduces accuracy. Our 1% threshold is thus a simple yet effective default. While not heavily tuned, this setting offers room for further optimization based on dataset characteristics.

We will include this analysis in the revised version.

---

Q3: Theory.

A3: Please refer to the answer A3 of reviewer bo7S.

---

Q4: Others.

A4: We will improve the typos and add more discussions due to space limitations.

---

We hope our response can resolve your concern. Have a nice day!

Thanks for taking the time to share your comments in the review assessment, as well as for acknowledging the well-supported core claim, simple but effective idea, systematic experimental validation, and insightful theoretical analysis. We provide a detailed point-by-point response to your comments.

Note that the following link refers to https://anonymous.4open.science/r/AgScore/Rebuttal.pdf

---

Q1: Robustness of AgScore in handling edge cases.

A1: We appreciate your insightful question. Indeed, edge cases like large domain shift, strong multi-class overlap or severely noisy unlabeled data pose significant challenges to semi-supervised learning. To mitigate these limitations, our orthogonal selection strategy for constructing the negative agent set aims to select representative and diverse negative samples, covering a broader semantic space of incorrect pseudo-labels. This helps to some extent in handling domain shift, class overlap and noisy data. However, we acknowledge that in extremely severe edge cases, the baseline methods may struggle to produce effective predictions, which could hinder the selection of reliable agents and the evaluation of pseudo-labels.

In this paper, we only focus on the conventional semi-supervised setting, where the baseline methods generally yield effective predictions that support our approach. Your valuable suggestion inspires us to explore extending our method to extreme cases in future work, which is crucial for enhancing the robustness of semi-supervised segmentation.

---

Q2: Robustness of AgScore when the teacher model is underfitted.

A2: Indeed, the key to the success of our method lies in selecting reliable positive and negative agents. As shown in Fig. 9 in link, we record the precision of the selected positive and negative agents after each training epoch. The agents are selected based on the top-1% and bottom-1% confidence scores, respectively. Notably, even after just the 0-th epoch, the precision of both positive and negative agents is already satisfactorily high. This suggests that the teacher model can provide reliable agents for AgScore almost right from the start of training. Consequently, to maintain the simplicity of AgScore, we do not employ any specific warm-up strategies, as the experiments demonstrate its effectiveness without such strategies. We concur that carefully integrating warm-up strategies could potentially lead to better performance, and we leave this as a direction for future investigation.

---

Q3: Typo and "AugSeg" misuse.

A3: Thanks for your meticulous review and valuable feedback. We apologize for the oversight in mistakenly referring to "AgScore" as "AugSeg" in the Experiments section. We will carefully review the manuscript, correct these errors, and improve the clarity of our expressions.

---

We hope our response can resolve your concern. Please do not hesitate to let us know if you have further questions :)