Revisiting Semi-Supervised Learning in the Era of Foundation Models

Thank you for your support of our work and your constructive feedback.

Concern 1: Selection of Evaluation Benchmark

We appreciate this observation and thank you for pointing out the limitation regarding task diversity in our evaluation benchmark. (1) Regarding the dataset in our benchmark: ​​While our study centers on classification, we deliberately ensured diversity by following the VTAB sub-categories (NATURAL, SPECIALIZED, and STRUCTURED), selecting at least two representative datasets from each. This design balances task diversity, domain coverage, and practical feasibility. To our knowledge, such a structured evaluation within the context of SSL remains underexplored. We hope our benchmark helps fill this gap and serves as a solid foundation to inspire future research on semi-supervised learning with VFMs across a broader range of tasks beyond classification.

(2) Regarding the task type of our benchmark: We agree that extending to tasks beyond classification is valuable. However, such tasks often involve task-specific architectures and training paradigms. To maintain focus on core machine learning challenges, we align with prior SSL works (e.g., FixMatch) by prioritizing image classification. We believe the core SSL principles and insights from our study are transferable. For instance, UniMatch [A], a popular SSL segmentation method, was directly inspired by FixMatch, and many SSL segmentation approaches [B, C] adopt self-training and pseudo-labeling. We appreciate the suggestion and will discuss this limitation in our revised manuscript.

Concern 2: Comparing to FineSSL

Thank you for your thoughtful observation. We agree that our work shares a similar high-level methodology with FineSSL [D] – both adopt pseudo-labeling in conjunction with PEFT to improve sample efficiency in semi-supervised learning with VFMs. FineSSL presents a simple and well-structured approach, and our work also provides valuable extension along two key axes: (1) building a more systematic and diverse benchmark across datasets, backbones, and PEFT methods, and (2) exploring model diversity through ensembling to further enhance pseudo-label quality.

We see FineSSL as a timely and valuable contribution to this emerging research direction. Our aim is to build upon such efforts to establish a strong and extensible baseline for future work. We will revise the related work and discussion sections to better clarify the connection and distinction between our method and FineSSL.

Concern 3: Scope of the Work

Thank you for your positive evaluation regarding the importance of the problem, technical soundness, and clarity of presentation. We also fully respect your assessment regarding the level of novelty. While the current results may not appear ground-breaking in isolation, we believe the proposed framework and benchmark introduce a new perspective in the SSL community that would likely lead to broader implications. We see this as a step toward a new paradigm that could catalyze future research and lead to impactful applications within the community.

Concern 4: Section for Limitations

Thank you for the constructive suggestion. We agree that, while we briefly acknowledge certain limitations in the main text (e.g., the focus on classification tasks [line 75] and the selection of six datasets [line 135]), these points would be clearer and more transparent if presented in a dedicated section. In the revised version, we will add an explicit "Limitations" section to clearly outline the scope of our current study and discuss directions for broader task coverage in future work.

Concern 5: Formatting issues

Thank you for pointing out our flaws in the formatting. We’ll revise them in the future manuscript and ensure a clean and professional final version.

References

[A] Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. CVPR’23.

[B] St++: Make self-training work better for semi-supervised semantic segmentation. CVPR’22.

[C] Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation. ICCV21.

[D] Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning. ICML’24.

Thank you for your interest in our work and for your constructive feedback. Meanwhile, we respectfully think some of the concerns may result from misunderstandings, and we apologize if these were caused by the organization of our paper. Before addressing your specific concerns, we would like to provide a brief overview of our paper and algorithm.

In the conventional setting of few-shot semi-supervised learning, our method follows a 4-stage pipeline:

1. Supervised Parameter-Efficient Fine-Tuning: We select a set of either (a) Different PEFT methods (PET) or (b) Different VFMs and PEFT methods (V-PET), and fine-tune them using only a few labeled samples.
2. Pseudo-Label Generation: The fine-tuned model is then used to generate pseudo-labels for the unlabeled data through single forward pass.
3. Pseudo-Label Ensemble: Multiple sets of pseudo-labels are ensembled through our mean-label strategy.
4. Ensemble Training: The final VFM is trained on the originally unlabeled data, using the ensemble pseudo-labels as supervision.

Based on the workflow. We’d like to address your concerns one by one as follows:

Concern 1: Computational Complexity

We acknowledge that, like other ensemble approaches, our method introduces some additional computational cost overhead. However, thanks to the efficiency of stage 1 and stage 2 listed above, this extra cost remains negligible. To better illustrate this trade-off, we summarize the end-to-end computation time in the table below:

As shown, PET incurs only ~3.3 additional minutes compared to baselines (1.07×), and V-PET ~6.6 minutes (1.14×). We believe this is a reasonable overhead given the performance gains achieved.

Concern 2: Self-Training vs. Ensemble

We appreciate your question. Before we discuss ensemble and self-training, we want to first clarify that our contributions extend beyond simply employing these techniques, but also lie in the insight that PEFT and multiple VFMs can generate high-quality, complementary pseudo-labels that make ensemble learning effective.

Regarding the performance gain, both ensemble and self-training are essential components. Self-training, as a SSL method, provides a principled way to leverage unlabeled data by generating pseudo-labels and fine-tuning the model in a supervised manner. Ensemble, on the other hand, serves to enhance the quality of these pseudo-labels. It is important to note that self-training is indispensable in our framework — without it, ensemble alone cannot fine-tune the pre-trained VFM. In our main results (Table 3), ST, PET, and V-PET all incorporate self-training. The distinction lies in how pseudo-labels are generated. We observe that ST alone does not consistently outperform the labeled-only baseline, indicating that the unlabeled data have not been effectively leveraged—likely due to the low quality of the pseudo-labels. In contrast, our approaches—PET and V-PET—use pseudo-label ensembles from multiple VFMs and PEFTs, enabling self-training to yield significant improvements over the labeled-only setting. This highlights the critical role of pseudo-label quality and underscores the effectiveness of our approach.

Concern 3: Limited Novelty and Relevance to Foundation Models

We respectfully disagree with the view that our methodology does not leverage the unique properties of VFMs. As mentioned in the paper [lines 51 - 56], our approach is built upon two core properties of VFMs. First, their ability to adapt quickly with minimal supervision — i.e., few-shot generalization. Second, the difference in their pretraining distribution or objectives. Our method explicitly exploits these capabilities by using VFMs to generate high-quality and complementary pseudo-labels from a few labeled examples, which enables us to further learn from unlabeled data.

While the use of ensemble and self-training may appear conventional, our key contribution lies in showing how these techniques can be adapted to the VFM regime — not as arbitrary choices, but as mechanisms that align with VFMs’ unique strengths (e.g., stable zero-shot predictions, diverse representations). Moreover, by establishing a controlled benchmark and introducing a simple yet effective baseline, we aim to lay the groundwork for a principled exploration of SSL in the context of VFMs. We believe this is both timely and impactful, given the growing interest in adapting large-scale models under limited supervision.

Concern 4: Incorporating Other SSL Techniques

Thank you for your question. We acknowledge that there are many existing SSL techniques. While thoroughly revisiting these tricks would be valuable to the community, we humbly believe this is orthogonal and beyond our focus. Indeed, as highlighted in paper [lines 34, 67], one of our aims is to establish a simple SSL pipeline—ideally with as few tricks as possible — by appropriately leveraging the unique properties of VFMs and PEFT.

That said, we do think that the existing SSL tricks and techniques (e.g., data augmentation, self-consistency) have the potential to further improve our method.

To explore this direction, we conducted additional experiments to evaluate the compatibility of our method with both weak and strong augmentations and self-consistency techniques. In the experiments, we fixed the hyperparameter settings under the DTD-3SHOT, DINOv2 configuration, using LoRA for ensemble training. We evaluated the performance of our method in combination with existing SSL techniques under two main settings:

1. Self-Consistency: Pseudo-labels are generated from weakly augmented (random horizontal flip) unlabeled data and averaged over 10 stochastic forward passes.

2. Strong Data Augmentation: During ensemble training, we add a cross-entropy loss term computed on RandAugment(m=3, n=5)-augmented samples.

We observe that strong augmentation yields modest performance gains, whereas self-consistency offers no additional benefit. This suggests that the effectiveness of such techniques may be less pronounced in our method or requires further tuning. We consider a more comprehensive exploration of such combinations as a promising direction for future work.

Question 1: Foundation Models’ Scaling Law for Labeled Data

Thank you for your constructive suggestion. Compared to training standard SSL classifiers from scratch, foundation models indeed require less labeled data – that is precisely why existing works leveraging foundation models for SSL [B][C], including us, primarily focus on low-shot settings. That said, we agree that systematically studying this question is an interesting direction. In response, we have extended our supervised fine-tuning part of Table 3 on the DTD dataset from 3-SHOT to 48-SHOT setting. The updated results are shown below,

As we can see, performance on low-shot settings (N3, N6) is already strong, thanks to the rich priors encoded in foundation models. As the number of labeled examples increases, performance continues to improve but with diminishing returns – indicating a flatter scaling curve compared to standard SSL classifiers trained from scratch. We appreciate the reviewer for highlighting this important difference, and we’ll revise the final version to have this study.

Question 2: Foundation Models’ Vulnerability to Imbalanceness

Thank you for the question. While we do not explicitly explore this question in the main paper, we respectfully believe that using foundation models as the backbone would make the overall SSL training more robust to class imbalance, compared to training from scratch (in conventional SSL methods). This is because foundation models often enable few-shot adaptation, which prevents minor categories from being simply dominated by major categories. Notably, our V-PET approach is a simple pseudo-labeling baseline without any rebalancing techniques, yet it still performs competitively. This suggests a certain degree of inherent robustness to imbalance in foundation models. We consider this a promising direction for future exploration.

References

[A] Lessons and insights from a unifying study of parameter-efficient fine-tuning (PEFT) in visual recognition. CVPR’25.

[B] Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning. ICML’24.

[C] USB: A Unified Semi-supervised Learning Benchmark for Classification. NeurIPS’22.

Re-Response: About Computation Efficiency

Thank you for your comment. After carefully reading your question again, we realize that our prior response did not fully answer it. We apologize for it, and we provide a more detailed response as follows.

To more easily decompose and analyze the computational time, please allow us to review our overall pipeline:

Supervised Parameter-Efficient Fine-Tuning: We select a set of (a) Different PEFT methods (PET) or (b) Different VFMs and PEFT methods (V-PET), and fine-tune each of them using only a few labeled samples per class. Pseudo-Label Generation: Multiple fine-tuned models are then used to generate pseudo-labels for each unlabeled data sample. Pseudo-Label Ensemble: Multiple pseudo-labels for each unlabeled data sample are ensembled through our mean-label strategy. Ensemble Training: The final VFM (one backbone) is trained (using one PEFT method) on the abundant unlabeled data, using the already pre-computed ensembled pseudo-labels in stage 3 as supervision.

Within this pipeline, the multiple VFMs (and multiple PEFT methods) are involved only in the first three stages. In stage 4, only one VFM and PEFT method is considered and trained, such as CLIP-LoRA or DINOv2-AdaptFormer, as reported in Table 3 of the main paper.

At first glance, it seems that the first three stages are where the computational bottleneck is, since they involve multiple VFMs. However, in reality, stage 4 — which involves one VFM and PEFT — indeed contributes to the major portion of computational time. This is because stage 4 involves training over all the unlabeled data (e.g. SUN397-3SHOT are 68,410 samples). In contrast, stage 1 only involves training over a few labeled data samples (same dataset, only 397*3=1,191 samples). As such, the computational time is relatively short compared to stage 4, even if we train multiple VFM-PEFT pairs. In stages 2 and 3, where we apply multiple pre-trained VFM-PEFT pairs on all the unlabeled data for pseudo-label ensembling, we emphasize that each VFM-PEFT pair only needs to perform one single forward pass for each unlabeled sample.

In our original rebuttal, ~Sec/Epoch (SFT) refers to stage 1 for each VFM-PEFT pair; ~Sec/Epoch (Ensemble) refers to stage 4; ~Total Time (Mins) refers to the total time of the four stages. We apologize for confusing as we did not provide further explanation for the naming. Note that, since in stage 4, we only train one VFM-PEFT pair — the ensemble pseudo-labels have been pre-computed in stage 3 and remain fixed — the ~Sec/Epoch will be approximately the same as other baseline methods.

Response: Why is there no computational overhead when you perform forward passing over multiple pre-trained VFMs?

We appreciate your question. There is indeed computational overhead when we train and infer with multiple pre-trained models. This was partially reflected in the ~Total Time (Mins) column (in the original rebuttal): V-PET takes a longer time than PET. In the original table, we neglect the computational overhead in stages 2 and 3 since it is faster than stages 1 and 4. However, we realize that for a fair comparison to the baselines, this should have been included. In the following, we refine the table, including computational time from all the stages.

We hope this detailed table addresses your concern. We note that this makes our computation time 1.08x for PET and 1.16x for V-PET over the baselines, respectively. That said, regarding your original concern in the review, “the computational cost of the proposed method seems significant,” we respectfully think a 1.16x computational time does not introduce a significant overhead. This is because the computational overhead of multiple VFMs only appears in stages 1-3, while stage 4 remains the major computational bottleneck. This also implies that our total computational time does not simply scale linearly when more VFMs and PEFT are involved. Specifically, considering Figure 5 in the main paper, where we study six VFMs, the overall computational time would go to ~58 minutes, roughly 1.24x over baselines instead of 6x.

Once again, we apologize for any potential confusion.

Thank you so much for your interest in our work and your valuable feedback. We would like to address all your concerns, point by point, as follows:

Concern 1: Dataset Selection

Thank you for your valuable insight regarding our dataset selection, particularly the observation that our selections are relatively "out-of-distribution (OOD)". We would like to respectively clarify our rationale. We first emphasize that we do not deliberately choose OOD datasets. As our focus is on semi-supervised learning (SSL) in the downstream tasks, we focus on tasks where frozen VFMs underperform, indicating where SSL could offer a beneficial solution [lines 37 - 38]. In retrospect, these datasets turn out to be those that exhibit a larger distribution shift between the pertaining and fine-tuning. That said, with PEFL techniques, the performance gap due to domain shifts can be effectively reduced, suggesting that the pre-training is still beneficial for downstream tasks even under distribution shifts.

Nonetheless, we fully agree with your point of view that incorporating relatively "in-distribution" datasets would create a more representative benchmark. Due to the large scale of INaturalist-2021 and TreeOfLife-200M, and limited time during the rebuttal period, we opted to evaluate on CUB-200 — a smaller-scale, but good quality, and also widely adopted dataset for fine-grained natural image classification. We conducted experiments on it following the same setting as Table 3 on 4-SHOT task. The results are attached in the table below, with the best performance in each column bolded:

Preliminary results on CUB-200 demonstrate that our proposed method also performs well in the fine-grained classification setting. And we’ll leave further exploration to future works.

Concern 2: Experiment

We appreciate your thoughtful suggestion regarding the interaction between VFMs and PEFT methods for ensemble methods. We agree that evaluating the interplay between these components is highly valuable and would strengthen our paper.

In response, we conducted the proposed ablation study by varying the number of VFMs (horizontally) and PEFT methods (vertically) from 1 to 6, following the same experimental setup as in Figure 5 of the main paper using DINOv2 and LoRA. The results are presented in an orthogonal grid of configurations, where for each setting, we report the average performance and standard deviation over three randomly sampled combinations.

Throughout the table, we can compute the Pearson correlation coefficient between the performance and the number of PEFT (or VFMs), as shown below:

This indicates that the number of VFMs is more related to the overall ensemble performance gain. However, the ensembling over PEFTs method is also useful and can serve as a supplement to VFMs, which is in accordance with the observations made by previous work [A].

Concern 3: Take-away message for VFMs vs. PEFT

Thank you for the valuable suggestion. As shown in Figure 5 and our reply to Concern 2, the marginal gains from ensembling VFMs consistently exceed those from applying PEFT. Based on this observation, we recommend prioritizing VFMs over PEFT when resources are limited. However, when only a few (or one) VFMs are accessible, PEFT offers an effective alternative to improve performance. We will include this key takeaway message in the revised version of the paper.

Concern 4: Computation Time

Thank you for your concern. We believe that the computational cost should not be a concern in our work. Like other ensembling-based methods, our approach introduces additional computational overhead. However, thanks to the efficiency of few-shot fine-tuning and pseudo-label generation, the additional cost is actually not severe. Thus, it keeps the overall cost relatively low compared to other ensemble methods.

To quantify this, we report the end-to-end runtime of PET and V-PET compared to standard baselines for DTD 3-shot setting. Based on our average measurements on experiments on an A100 GPU, we estimate the per-epoch time to be approximately 93 seconds.

As shown, PET incurs only ~3.3 additional minutes compared to baselines (1.07×), and V-PET ~6.6 minutes (1.14×). We believe this is a reasonable overhead given the performance gains achieved.

Concern 5: Formatting issues

Thank you for pointing out the formatting issues, both about the typos and the table 3 formatting. We will carefully revise the manuscript to correct these problems and ensure a clean, professional presentation in the final version.

References

[A] Lessons and insights from a unifying study of parameter-efficient fine-tuning (PEFT) in visual recognition. CVPR’25.

Thank you for the response. I have some clarifying follow-up questions.

Concern 1: Dataset Selection

Looking at the datasets in the paper again, I realize that it includes one which is near-OOD: SUN397. This mitigates some of my previous concern that all the datasets were far-OOD from ImageNet.

I thank the authors for running these additional experiments, demonstrating similar trends for CUB-200 as seen for the other datasets. (The TreeOfLife dataset was a bad suggestion, not just because of its size but also because it includes data from both lab and field settings, meaning the distribution is inconsistent.) I have a question for the authors: do they intend to add the CUB-200 results to the paper?

Concern 2: Experiment

Thank you for sharing these additional results. Can you clarify - does the number of fine-tuned models used for the ensemble equal the product of N_PEFT and N_VFM? In that case, looking at the iso-compute groups, it does indeed seem that adding more VFMs dominates over adding more PEFTs, e.g. comparing N_PEFT * N_VFM = 6:

• N_PEFT=1, N_VFM=6 => 63.40 ± 0.19
• N_PEFT=2, N_VFM=3 => 63.00 ± 0.58
• N_PEFT=3, N_VFM=2 => 63.10 ± 0.70
• N_PEFT=6, N_VFM=1 => 62.48 ± 0.52

Note however, these still leaves open the possibility of using a mixture of different PEFT methods for each VFM (as I suggested in my review). Using one PEFT method for each of the 6 VFMs, but choosing a different PEFT method for each, could be more beneficial than the N_PEFT=1, N_VFM=6 setting, without imposing additional computational costs, and may be worth considering.

Concern 3: Take-away message for VFMs vs. PEFT

I am happy with this conclusion, and support its addition to strengthen the paper.

Concern 4: Computation Time

Thank you for the clarification regarding the computational requirements. I am surprised that the costs of the PEFT stage is so light per model. Since three of the reviewers highlighted this as a concern, I think it would be prudent to update the paper to specify the computational requirements of each of the methods considered in Table 3. This would improve the impact of the paper, since people are more likely to use the method if they correctly understand the additional computational overhead is light.

Reply for Concern 1

Thank you for your recognition of our dataset selection, the additional experiments, and your continued interest.

Regarding the CUB-200 results, we are currently not planning to include them in the main table. The main reason is that our benchmark strictly follows a subset of VTAB [A], where we select exactly two datasets from each of VTAB’s official categories:

(1) Natural – DTD, SUN397
(2) Specialized – RESISC45, DIABETIC-RETINOPATHY
(3) Structured – CLEVR-COUNT, KITTI

Introducing a new dataset from outside would disrupt this structure. We will ensure this methodology is clearly reiterated in the revised version. That said, we are happy to include the CUB-200 results in the additional experimental sections.

Reply for Concern 2

Thank you for bringing up your further concerns. As for your question about the number of fine-tuned models — yes, the number of ensembled pseudo-labels equals N_PEFT × N_VFM. We also appreciate your sharp observation that, under equal computation, N_VFM scales better than N_PEFT. This provides additional empirical support beyond the Pearson correlation coefficient.

We will include a discussion in the revised version summarizing this exchange — including your suggestion of using a mixture of different PEFT methods across VFMs (e.g., assigning a different PEFT method to each of the 6 VFMs in the N_PEFT=1, N_VFM=6 setting), as well as the additional experiments we conducted during the rebuttal process.

Reply for Concerns 3 & 4

Thank you for your positive feedback regarding our additional results. We also appreciate your suggestion to clarify the computational requirements. As you pointed out, this is a common concern raised by several reviewers. We will include the corresponding compute estimates and conclusions in the revised version.

We hope the clarifications and additional results have addressed your concerns, and would appreciate your consideration in reflecting this in your final score.

---

References

[A] A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark, 2019

Response: Regarding the Additional Forward Pass Cost

We appreciate the reviewer for cross-checking other reviewers’ questions and our responses. We acknowledge that we neglected the forward pass of multiple VFMs in generating pseudo-labels, and we apologize for it. In our new response to reviewer gEWg (on August 8th, 2025), we bring back such a computational overhead.

We note that the forward pass in which the unlabelled data is passed through each of the 4 PEFT fine-tuned models costs less than 4/3*93 seconds = 2.1 minutes. This is because during pure forward passes (without backward passes), a larger batch size can be used compared to the one used in training. Based on the logs, the time for each VFM to forward pass through all unlabeled data is ~15 seconds, which becomes ~60 seconds when considering four models. In our original rebuttal, we found this time smaller than a single epoch and ignored it, but we agree with you that it would be better to report it for a more accurate report of our computational time.

Response: Full Computation Wall-Clock Time

We appreciate your suggestion to report the full, wall-clock computation time. The reason why we did not report it in the original rebuttal because we found some implementation details in the baselines (e.g., FixMatch, FlexMatch, and SoftMatch) may inadvertently introduce additional computation. For example, each algorithm extracts logits for multiple augmentations (e.g., no, weak, or strong), requiring more than a single forward pass through each training sample (in other words, a single training sample is seen more than once per epoch). Indeed, if we report the end-to-end wall-clock time without modifying the baselines, our method runs slightly faster.

NOTE: We rerun the experiments on a supercomputer center, obtaining absolute times slightly faster than those reported earlier in the rebuttal, which were based on the logs from the paper submission period. We hypothesize that this may be caused by an infrastructure upgrade, which should not affect the relative time across methods.

That said, these results certainly do not mean our method is more computationally efficient. We report them here mainly to justify why we did not report the real wall-clock time in the first place, but rather report an estimated time — we aim to report a time that reflects the difference introduced by the use of VFMs, rather than training details. We hope this clarification addresses your concern.

Response: Regarding Computation Time and the Number of Epochs

We appreciate your calculation regarding the additional time our methods need and whether it could be traded for more training epochs for the baselines. In our experience, 30 epochs are sufficient for the baselines and our methods to converge; reducing 10%~20% of training epochs introduces a negligible drop to our approach. In our humble opinion, considering the number of epochs would bring in an extra factor—the convergence speed of different methods—into consideration, which may defocus the comparison in this paper.

If we understand your and reviewer gEWg’s original review correctly, the concern was whether our method would introduce a significant linear scaling in terms of computation time, as we bring in multiple VFMs. In this regard, we respectfully believe that a 1.16x increase demonstrates that our method does not result in a significant overhead. This is because the computational overhead of multiple VFMs only appears in stages 1-3, while stage 4 remains the major computational bottleneck (based on our new response to reviewer gEWg on August 8th). Indeed, if we consider the experiments in earlier rebuttal post, where we studied 6 VFMs*6 PEFT methods, the overall computational time would go to ~115.5 minutes, roughly 2.48x over baselines instead of 36x.

We believe that a 1.16x difference in computational time is acceptable, not to mention that the implementation details may indeed introduce additional overheads for the baselines.

We hope our further clarifications above address your concerns, and we will be happy to provide further discussions.

Thank you for your interest in our work, and we greatly appreciate your valuable feedback. We would like to address all your concerns, point by point, as follows:

Concern 1: Why Ensembling Works Better

Thank you for raising this insightful question. We would like to reiterate and further elaborate on why ensemble pseudo-labeling is particularly beneficial in the VFM+PEFT setting, building on the evidence already discussed in Section 5.2.

Unlike traditional SSL methods that ensemble models trained via random initialization, our method leverages the inherent diversity across different PEFT strategies and VFM pretraining objectives. As shown in Section 5.2, even when different PEFT methods achieve similar overall accuracy, they tend to produce divergent predictions on individual samples (Figure 1) – this is in accordance with the observations in previous work [A]. Similarly, VFMs pretrained with varying data and objectives demonstrate complementary strengths, with no single model dominating across all examples.

This natural diversity leads to a situation where individual PEFT-VFM pseudo-labels may be biased or inconsistent. By ensembling these diverse predictions, we reduce model-specific noise and extract consensus signals, thereby producing pseudo-labels that are more robust and informative.

Furthermore, we address the practical challenge of inconsistent output scales across models (as discussed in How to Ensemble? in section 5.2 and shown in Figure 4) by proposing the Mean Labels strategy, which converts predictions into one-hot vectors before averaging. This avoids scale mismatch and ensures a more balanced ensemble.

Together, these designs allow our method to effectively leverage the diversity inherent in VFMs and PEFTs to construct high-quality pseudo-labels. We will take your advice and revise the corresponding section in the paper to make this motivation clearer.

Concern 2: Clarification on Table 3

We appreciate your feedback and apologize for any confusion. As indicated in the header and the first column of Table 3, all baselines and our proposed methods are trained using the same backbone and PEFT method within each part of the table. We will make this clearer in the final revision of the paper.

Regarding your concern about training time, we acknowledge that our ensemble-based methods may introduce additional computational overhead compared to other baselines. However, due to the efficiency of few-shot fine-tuning and pseudo-label extraction, we find the overhead to be relatively modest considering the performance improvements achieved. We provide detailed runtime comparisons in the following tables for DTD 3-shot setting, where we estimate the per-epoch wall-clock time as 93 seconds based on our average observation of experiments on one single A100 GPU:

As shown above, PET incurs only ~3.3 additional minutes compared to baselines (1.07×), and V-PET ~6.6 minutes (1.14×). We believe this is a reasonable and practical trade-off considering the performance gains obtained. We will also take your advice and incorporate this table in the revised version of the paper.

Concern 3: Lower Performance on Some Datasets

Thank you for your careful observation. We acknowledge that although our method consistently outperforms other approaches across the benchmark, its performance gain is less pronounced on certain datasets, namely Resisc45. Diving deep into it, we believe this is due to the nature of the dataset itself. Compared to the rest of the benchmark, Resisc45 is relatively less challenging — as evidenced by the higher performance across other methods in Table 3. In such cases, the outputs of different PEFT models and VFMs tend to be more aligned, leaving limited room for ensembling to further reduce bias or extract complementary information. As a result, the benefit of our ensemble approach is naturally diminished.

However, we would like to respectfully clarify that on Retinopathy, our method actually performs competitively — in fact, we outperform most baselines across almost all of the settings, as shown in Table 3. While there may be marginal variations in specific configurations, the overall trend supports the effectiveness of our approach, even on this challenging medical dataset.

Concern 4: Limitations on Scope

Thank you for your valuable feedback. We acknowledge that extending our work to tasks beyond classification is valuable, but these tasks often require task-specific architecture and training paradigms. To remain focused on core machine learning problems, we align with representative SSL works (e.g., FixMatch) by prioritizing image classification. We humbly believe the fundamental SSL principles and insights from our work are transferable across tasks. For example, UniMatch [B], a popular SSL segmentation method, was inspired by FixMatch. Many SSL segmentation methods [C, D] use self-training & pseudo-labelling. We’ll also take the suggestions and discuss the limitations of our work in the revised version of the paper.

References

[A] Lessons and insights from a unifying study of parameter-efficient fine-tuning (PEFT) in visual recognition. CVPR’25.

[B] Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. CVPR’23.

[C] St++: Make self-training work better for semi-supervised semantic segmentation. CVPR’22.

[D] Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation. ICCV21.