VQ-Seg: Vector-Quantized Token Perturbation for Semi-Supervised Medical Image Segmentation

We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.

---

[W1] The choice of codebook size.

[A1] Thank you for the helpful comment regarding codebook size. We conducted an ablation study to investigate how the codebook size $K$ influences segmentation performance under 5% and 10% labeled settings:

The results show that segmentation performance consistently improves as the codebook size increases, peaking at $K = 16384$, where Dice scores are highest and utilization remains high. Beyond this point, performance degrades and utilization drops, suggesting that excessively large codebooks may lead to underutilization. These findings indicate that a codebook size of 16384 strikes the best balance between representational capacity and generalization performance.

[W2] Dual-branch design incurs unnecessary overhead.

[A2] We thank the reviewer for raising the concern regarding the computational overhead introduced by the dual-branch (DB) architecture. While the DB module adds one additional image decoder, it reuses the same post-VQ latent space. More importantly, it plays a crucial role in improving both segmentation performance and codebook utilization, especially under low-label regimes. We justify this design choice from two perspectives:

• Segmentation Performance Improvement: As shown in Table 2 of the main paper, adding the DB module to a model already equipped with QPM yields a clear performance gain: Dice increases from 0.7701 to 0.7784. This suggests that the reconstruction branch helps preserve more structural details in the post-VQ representation, directly benefiting segmentation accuracy.

• Improved Codebook Utilization: We further conduct an ablation study to evaluate the impact of each module on codebook utilization (with codebook size set to 16,384). Introducing the DB module increases utilization from 87.2% to 90.1%, indicating that the auxiliary reconstruction task provides an additional learning signal that encourages broader and more stable codeword activation.

Although the DB module introduces a slight computational overhead, it significantly enhances codebook utilization and helps maintain a semantically meaningful perturbation space. This results in clear improvements in both segmentation performance and representation quality, which is especially valuable in semi-supervised settings.

[W3] The reliability of pseudo-labels is not discussed.

[A3] Thank you for the comment. We would like to clarify that our pseudo-labeling strategy is consistent with those adopted in prior teacher-student-based semi-supervised segmentation methods, including the baselines [1] used for comparison. Specifically, we employ an Exponential Moving Average (EMA) strategy to update the teacher network—a widely accepted practice that improves the stability and reliability of pseudo labels by smoothing parameter updates over time.

While NeurIPS does not allow modifications to the original submission during the rebuttal period, we will include visualizations of the generated pseudo labels in a future revision to better illustrate their quality and consistency.

[W4] The choice and design of encoder and decoder networks.

[A4] To ensure a fair comparison, our VQ-Seg model adopts the same encoder and decoder architecture as Unimatch [2], which achieves the best average performance among all comparison methods. Detailed architectural specifications can be found in the original Unimatch paper.

[Q1] Signs of codebook collapse.

[A5] Thank you for raising this important point. To clarify, our VQ-Seg framework achieves high codebook utilization, reaching 98% with a codebook size of $K = 16384$, as shown in the ablation study below:

This table demonstrates that each proposed component—QPM, DB, and PFA—contributes substantially to alleviating codebook collapse and enhancing utilization. In contrast, the base setting suffers from severe underutilization (only 11.6%), which clearly indicates codebook collapse.

We further examined the impact of varying codebook sizes on both segmentation performance and utilization:

These results confirm that codebook utilization remains consistently high (>90%) across different codebook sizes. Notably, performance peaks at $K = 16384$, suggesting it provides the best trade-off between representational capacity and generalization. Beyond this point, increasing the size yields diminishing returns and slightly reduced utilization.

Taken together, these findings strongly suggest that our framework effectively avoids codebook collapse during training.

[Q2] Superior results in upper rows of Table 1.

[A6] Thank you for pointing this out. The upper portion of Table 1 reports results from fully supervised models (denoted as “F”), which are trained using 100% of the labeled data. In contrast, our method and others listed below the horizontal line are trained using only 5% or 10% of the labeled data under a semi-supervised setting.

The apparent performance gap stems from this difference in supervision level. However, one of the core strengths of VQ-Seg lies in its ability to deliver competitive performance with minimal annotations. As shown in Table 1, VQ-Seg achieves Dice scores of 0.6643 (5%) and 0.7852 (10%), already comparable to those of fully supervised methods.

To further assess scalability, we gradually increased the proportion of labeled data and report the results below:

Remarkably, VQ-Seg outperforms the fully supervised UNet (trained with 100% labels) even when using only 50% of the labeled data. This demonstrates that VQ-Seg performs robustly under limited supervision and continues to improve with more annotations, ultimately surpassing fully supervised baselines.

These findings underscore the effectiveness and scalability of our approach, clarifying that the superior results in the upper rows of Table 1 are due to supervision level differences rather than performance inconsistencies.

---

References:

[1] Yu L, Wang S, Li X, et al. Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation[C]//International conference on medical image computing and computer-assisted intervention. Cham: Springer International Publishing, 2019: 605-613.

[2] Yang L, Qi L, Feng L, et al. Revisiting weak-to-strong consistency in semi-supervised semantic segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 7236-7246.

---

Additional results/analyses on your concerns are in the revised manuscript. We hope these address your questions. Let us know if you need further discussion, experiments, or clarifications—happy to provide more details.

We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.

---

[W1] The experiments are limited to small-scale datasets.

[A1] To further demonstrate the scalability of our approach, we conducted additional experiments on the large-scale FLARE22 benchmark. We compared our method, VQ-Seg, with a supervised UNet baseline (UNet-S) and the state-of-the-art semi-supervised method Unimatch, using average Dice score as the evaluation metric.

Our method achieves the highest average Dice score, demonstrating strong generalization and effectiveness on large-scale datasets. We will include these results and cite the FLARE22 dataset and its related publication in the final version of the paper.

[W2] More labeled data ratios should be evaluated.

[A2] Thank you for the suggestion. We fully agree that evaluating broader label ratios offers a more comprehensive view of the model’s scalability. Accordingly, we have extended our experiments on the LC dataset to include 20%, 50%, and 100% labeled settings. The updated Dice scores are presented below:

VQ-Seg consistently outperforms all semi-supervised baselines across all settings, and even surpasses the fully supervised UNet using only 50% of the labeled data. These results demonstrate the strong scalability and efficiency of our method.

[W3] No external dataset is used to evaluate domain generalization.

[A3] To evaluate external generalization, we conducted cross-domain experiments using the publicly available LUNA16 dataset [1], which also focuses on lung cancer segmentation. We compare our method and baselines trained on LC (5% labeled) and tested on LUNA16, along with a UNet trained directly on LUNA16 as an in-domain upper bound:

Our method achieves the best generalization performance among all semi-supervised models and closely approaches the in-domain upper bound, demonstrating strong robustness to domain shift.

[W4] Concern over using DINOv2.

[A4] We appreciate the reviewer’s insightful concern regarding the potential domain mismatch between DINOv2 and medical imaging tasks. While it is true that DINOv2 is pretrained on natural images, we selected it as our semantic teacher for several reasons:

(1) Strong Generalization and Empirical Success in Medical Imaging: DINOv2 has demonstrated remarkable generalization capabilities across diverse downstream tasks. Despite being trained on natural images, it has shown strong transfer performance in medical domains. For example, [2] reports that DINOv2 achieves the best average segmentation accuracy among a range of pretraining methods. Additionally, [3] shows that directly using frozen DINOv2 features yields state-of-the-art results in medical image registration. These findings suggest that DINOv2 encodes rich and transferable semantic representations, even under domain shift.

(2) Separation of Roles: DINOv2 for Quantization Alignment, DB for Domain Knowledge: DINOv2 is specifically used to align semantic representations before and after quantization, addressing the semantic degradation and information loss introduced by vector quantization. It serves as a stable, high-level anchor that preserves representation consistency across the quantization boundary. Crucially, medical domain knowledge is not derived from DINOv2; instead, it is injected via our Dual-Branch (DB) module, which incorporates task-specific features and medical semantics. This separation ensures that our model benefits from both robust general-purpose semantics (via DINOv2) and precise domain-specific guidance (via DB).

(3) Comparison with Medical Foundation Models: To further validate our design choice, we conducted an ablation study replacing DINOv2 with several alternative foundation models, including ones pretrained on medical data. The Dice scores under 5% and 10% labeled data are shown below:

Here, CLIP [4] and BiomedCLIP [5] are vision-language models trained on large-scale image–text datasets (natural and medical, respectively). MAE [6] is trained via masked autoencoding, and rad-DINO [7] follows a DINO-style architecture trained on radiology-specific images. As shown, DINOv2 consistently outperforms all alternatives across both labeling regimes, including models specialized for medical domains. These results empirically validate DINOv2 as a robust and effective semantic prior for semi-supervised medical image segmentation.

References:

[1] Setio A A A, Traverso A, De Bel T, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge[J]. Medical image analysis, 2017, 42: 1-13.

[2] Baharoon M, Qureshi W, Ouyang J, et al. Evaluating general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks[J]. arXiv preprint arXiv:2312.02366, 2023.

[3] Song X, Xu X, Yan P. Dino-reg: General purpose image encoder for training-free multi-modal deformable medical image registration[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2024: 608-617.

[4] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PmLR, 2021: 8748-8763.

[5] Zhang S, Xu Y, Usuyama N, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs[J]. arXiv preprint arXiv:2303.00915, 2023.

[6] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.

[7] Perez-Garcia F, Sharma H, Bond-Taylor S, et al. Exploring scalable medical image encoders beyond text supervision[J]. Nature Machine Intelligence, 2025, 7(1): 119-130.

---

Additional results/analyses on your concerns are in the revised manuscript. We hope these address your questions. Let us know if you need further discussion, experiments, or clarifications—happy to provide more details.

We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.

---

[W1] Limited novelty in using foundation models; lack of comparison.

[A1] Rather than simply using a foundation model, we introduce a novel strategy that leverages it as a semantic anchor to calibrate quantized representations for semi-supervised medical image segmentation. The Post-VQ Feature Adapter (PFA) aligns discrete VQ features with high-level semantics, effectively mitigating quantization-induced degradation. As shown in the ablation study (Table 2), removing the PFA results in a performance drop: Dice decreases from 0.7852 to 0.7784, and Jaccard from 0.6731 to 0.6620.

To evaluate the impact of different foundation models, we replace DINOv2 in the PFA module with several representative alternatives and report Dice scores under 5% and 10% labeled settings:

CLIP [1] and BiomedCLIP [2] are vision-language models pre-trained on large-scale natural and medical image–text pairs, respectively. MAE [3] is a vision model trained via masked image modeling. rad-DINO [4] adopts a DINO-like architecture trained on radiology-specific datasets. As shown above, DINOv2 achieves the best performance across both settings, confirming its strong semantic capacity as the backbone for our PFA module.

[W2] No error bars or significance testing in experiments.

[A2] Thank you for the constructive suggestion. To assess statistical robustness, we ran both Unimatch and VQ-SEG 10 times under the 5% labeled setting using different random seeds. The results are summarized below:

A paired t-test (p = 0.0013) and a Wilcoxon signed-rank test (p = 0.0021) confirm that the performance gain of VQ-SEG over Unimatch is statistically significant.

[W3] Codebook size not studied.

[A3] Thank you for the helpful comment regarding codebook size. We conducted an ablation study to examine how the codebook size $K$ affects segmentation performance under 5% and 10% labeled settings:

The results show that segmentation performance improves steadily as the codebook size increases, peaking at $K = 16384$, where Dice scores are highest and utilization remains high. Beyond this point, performance degrades and codebook utilization drops. This confirms that a codebook size of 16384 offers the best trade-off between representational capacity and generalization.

[W4] QPM lacks empirical justification vs. other strategies.

[A4] Thank you for the insightful comment. Our Quantized Perturbation Module (QPM) adopts a probabilistic replacement strategy based on pairwise codeword distances (Eq. 4), rather than simple spatial shuffling.

To justify this design, we compare QPM with two deterministic baselines under both 5% and 10% labeled settings:

• Nearest Neighbor Replacement: each codeword is replaced with its closest neighbor.
• Farthest Neighbor Replacement: each codeword is replaced with the most distant one.

QPM consistently achieves better performance than both baselines across different labeled settings.

[W5] No analysis of codebook quality or its impact on perturbation.

[A5] Thank you for your comment. The quality of the learned codebook is analyzed in [A3], where we examine how codebook size affects both segmentation performance and codeword utilization. We find that performance peaks when the codebook is sufficiently expressive and efficiently utilized (98% utilization at $K = 16384$). In contrast, excessively large codebooks suffer from underutilization and slightly reduced performance, while overly small codebooks, though fully utilized, lack diversity and representational capacity, leading to suboptimal results.

This level of utilization is critical for QPM: a well-utilized codebook provides dense coverage of the latent space, ensuring that each replacement remains within a semantically meaningful neighborhood. Conversely, a sparse or underutilized codebook results in unstable and potentially harmful perturbations.

[W6] Missing ablation beyond incremental contributions.

[A6] Thank you for your attention to the ablation analysis. We would like to clarify that an ablation study on the individual components of our method is already included in the main paper (Table 2, Section 4.5). This table presents a step-by-step evaluation of the Quantized Perturbation Module (QPM), the dual-branch architecture (DB), and the Post-VQ Feature Adapter (PFA).

The results show that each component contributes clear performance gains over the baseline, and their combination achieves the best overall performance. These findings confirm the effectiveness and complementarity of the proposed modules beyond mere incremental contributions. We hope this addresses the reviewer’s concern.

[Q1] Prevention of codebook collapse.

[A7] To clarify how our method prevents codebook collapse, we include an additional ablation study demonstrating that each component contributes to improved codebook utilization. The codebook size is set to 16,384.

Each component plays a distinct role in enhancing utilization. QPM introduces structured perturbations that activate a broader set of codewords. DB promotes representational diversity through joint reconstruction supervision. PFA encourages consistent and semantically meaningful codeword activation by aligning with high-level features. Together, these modules effectively mitigate codebook collapse.

[Q2] Utilization rate of 16384 codewords and its impact.

[A8] Please refer to [A3] for detailed analysis. A codebook size of 16,384 provides the best trade-off between performance and utilization (98%).

[Q3] Sensitivity to foundation model choice.

[A9] Please refer to [A1] for detailed analysis on the impact of different foundation models. DINOv2 consistently outperforms other alternatives, confirming its effectiveness.

[Q4] Validity of spatial shuffling in medical images.

[A10] We appreciate the reviewer’s concern regarding the validity of spatial shuffling in medical images. Recent studies [5] have shown that distances between codewords in the VQ codebook reflect semantic similarity. We observed a similar property in our experiments. Perturbations introduced by the QPM, which rely on controlled sampling of nearby codewords, help preserve anatomical semantics in the predicted segmentation. For instance, in lung cancer segmentation, QPM-perturbed predictions remain focused on cancer-relevant regions, maintaining their location and shape. In contrast, dropout often produces noisy outputs, including false positives in the background or mis-segmentation of surrounding structures such as the bronchi or chest wall. Although visualizations cannot be included at this stage due to NeurIPS policy, we will provide detailed qualitative and quantitative comparisons in the final version to substantiate this claim.

---

References:

[1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PmLR, 2021: 8748-8763.

[2] Zhang S, Xu Y, Usuyama N, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs[J]. arXiv preprint arXiv:2303.00915, 2023.

[3] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.

[4] Perez-Garcia F, Sharma H, Bond-Taylor S, et al. Exploring scalable medical image encoders beyond text supervision[J]. Nature Machine Intelligence, 2025, 7(1): 119-130.

[5] Hu T, Zhang J, Yi R, et al. Improving autoregressive visual generation with cluster-oriented token prediction[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 9351-9360.

---

Additional results/analyses on your concerns are in the revised manuscript. We hope these address your questions. Let us know if you need further discussion, experiments, or clarifications—happy to provide more details.

Dear Reviewer snt1,

Thank you very much for your valuable feedback and insights. We greatly appreciate the time and effort you have devoted to reviewing our work. We have carefully addressed the points you raised and would be grateful if you could review our responses to ensure that your questions and concerns have been fully addressed. We are more than willing to engage in further discussions.

Warm regards,

Authors of VQ-Seg

We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.

---

[W1] Limited labeled ratio settings.

[A1] We agree that incorporating additional labeled ratio settings can better illustrate the performance trend under varying levels of supervision. In response, we have extended our experiments on the LC dataset to include additional labeled ratios of 20%, 50%, and 100%. The updated results, evaluated using the Dice coefficient, are summarized below:

As shown above, VQ-Seg consistently outperforms all baselines across different supervision levels. Notably, with only 50% labeled data, our method already surpasses the fully supervised UNet baseline (0.8345). The consistent improvements as more annotations are available further highlight the scalability and effectiveness of our approach.

[W2] Lack of discussion on the impact of different foundation models.

[A2] Thank you for the helpful comment. We provide further clarification below.

We assess the impact of different foundation models by replacing DINOv2 in VQ-Seg and reporting Dice scores under the 5% and 10% labeled settings:

CLIP [1] and BiomedCLIP [2] are vision-language models pre-trained on large-scale natural and medical image–text pairs, respectively. MAE [3] is a vision model pre-trained via masked image modeling, while rad-DINO [4] employs a DINO-style architecture trained on radiology-specific datasets.

As shown above, DINOv2 consistently achieves the best performance across both supervision levels. It surpasses CLIP and MAE by more than 2% in absolute Dice score and outperforms BiomedCLIP and rad-DINO by a non-trivial margin.

These results reveal that foundation models pretrained with self-supervised, vision-only objectives (e.g., DINOv2, rad-DINO) produce more structure-aware representations and outperform vision-language models (e.g., CLIP, BiomedCLIP) on segmentation tasks. Among all candidates, DINOv2, which is used in our method, achieves the highest overall performance.

[W3] Lack of evidence supporting the benefits of QPM over dropout.

[A3] Thank you for the insightful comment. Due to NeurIPS rebuttal policy, we are unable to modify the main paper or include additional figures at this stage. However, we would like to clarify that the perturbations introduced by QPM are anatomically meaningful. For example, in lung cancer segmentation, QPM-perturbed predictions remain focused on cancer-related regions. In contrast, dropout often introduces noisy outputs, including false positives in the background or mis-segmentation of adjacent structures such as bronchi or the chest wall. This behavior reflects the controlled nature of QPM, which samples from semantically similar codewords and thereby preserves anatomical semantics in the predictions. Although visualizations cannot be included at this stage, we will provide detailed qualitative and quantitative comparisons in the final version to support this claim.

---

References:

[1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PmLR, 2021: 8748-8763.

[2] Zhang S, Xu Y, Usuyama N, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs[J]. arXiv preprint arXiv:2303.00915, 2023.

[3] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.

[4] Perez-Garcia F, Sharma H, Bond-Taylor S, et al. Exploring scalable medical image encoders beyond text supervision[J]. Nature Machine Intelligence, 2025, 7(1): 119-130.

---

We have incorporated additional results and detailed analyses addressing your concerns into the revised version of the manuscript. We hope these supplements adequately address your questions and improve the clarity and robustness of our work. Please do not hesitate to let us know if you require further discussion, additional experiments, or any clarifications—we are happy to provide more details to address your concerns comprehensively.