Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning

Weakness2

• First, we would like to emphasize that our work is less targeted at single task learning, e.g., vision-language learning. Instead, our key motivation lies in the inherent representation biases of VFMs, which result in advantages and disadvantages across different vision tasks. Our SAK synergizes these complementary biases to enhance multi-task learning. Additionally, we note that another related baseline, Theia, which focuses on robot learning, does not provide comparisons with RADIO on these tasks.

• Here we supplement the experiments of VLM. Following the setup of RADIO, we integrate SAK into LLaVA-1.5 [6,7], freezing the Teacher-Agnostic Stem and Teacher-Specific Adapter Path, and fine-tuning only the MoR Router. We evaluate three tasks – GQA, POPE, and VQAv2 – at resolutions of 432 and 512, consistent with RADIO. We also include DINOv2-L, CLIP-L, and SAM-H for comparison.

• The following table shows that while SAK is trained on significantly less data, and with smaller VFM teachers, it achieves comparable performance to the teachers and RADIO on GQA and POPE. The performance gap on VQAv2 is largely attributable to the simplicity of our distillation setup, which prioritizes training efficiency. Regarding dataset scale, as listed in Table 1 (page 2) and Table 20 (Appendix D.3, page 28), all VFM teachers and RADIO use large-scale datasets, e.g., DINOv2 uses 142M images, CLIP uses 400M, RADIO uses 1.4B. In contrast, our SAK distillation relies on ImageNet-1k, which has only 1.2M images. This limited dataset diversity could significantly impact performance on VQA tasks, which require extensive training data.

  Meanwhile, RADIO employs larger VFM teachers, including DFN CLIP-H, SigLIP-SO400M, DINOv2-g, and SAM-H, which result in 2.5 times the parameters and forwarding costs of SAK with teachers based on ViT-L. RADIO also uses a larger resolution of 1,024 during its distillation (In LLaVA experiments, the image resolution is 432 or 512), enhancing image information extraction but at a substantial increase in forwarding and backwarding costs. Considering these factors, it is reasonable that SAK cannot outperform RADIO in VLM applications.

• We believe that with access to larger-scale datasets and more powerful teachers during distillation, SAK could achieve competitive or superior results. However, such resources are currently unavailable due to time and computational constraints. We appreciate the reviewer for underscoring the importance of applications in multimodal LLM and will add this discussion and the corresponding results to our manuscript.

[1] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR, 2017.

[2] Yuqi Yang, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts. In CVPR, 2024d.

[3] Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G LearnedMiller, and Chuang Gan. Mod-Squad: Designing mixtures of experts as modular multi-task learners. In CVPR, 2023.

[4] Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, and Zhangyang Wang. M3ViT: Mixture-of-experts vision transformer for efficient multitask learning with model-accelerator co-design. In NeurIPS, 2022.

[5] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.

[6] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2024.

[7] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.

We appreciate your time and efforts in reviewing our paper, providing valuable comments and constructive suggestions. We are encouraged by your acknowledgement of our work as “a novel attempt to analyze the failures of previous distillation methods,” proposing “lightweight and easy to implement” modules, and demonstrating “significant and solid experiment results.” We provide the following responses to address your concerns.

Weakness1

Firstly, our proposed MoR Router is based on a simple architecture. As elaborated in Section 3.3 (page 6, line 313), it employs a two-layer MLP design. The noisy gating mechanism, described in Eq. 4, is a standard technique commonly used in MoE approaches [1].

It is important to highlight that, unlike MoE used in previous multi-task learning [2,3,4] or general-purpose models [1,5], where the experts are typically homogeneous during optimization, the experts in our framework consist of representations distilled from various VFM teachers as well as the Teacher-Agnostic Stem. These experts inherently possess different knowledge, making them heterogeneous by design. As a result, the MoR Router does not require a balanced routing because task-specific routers dynamically weigh and combine the most relevant representations for each task. Empirically, we did not encounter any difficulties in training the MoR Routers. This approach ensures that the router effectively synergizes the complementary strengths of the diverse representations to optimize performance for different tasks.

We appreciate the reviewer’s detailed feedback and will include this clarification in our manuscript.

The authors address my concerns therefore I raise the score to 8.

We appreciate your time and efforts in reviewing our paper, providing valuable comments and constructive suggestions. We are encouraged by your acknowledgement of our work in conducting “extensive experiments, ablation and analysis,” ensuring “small inference cost,” and recognizing it as “clear, well-organized” and “easy to follow.” We provide the following responses to address your concerns.

Weakness1

We appreciate the reviewer for the insightful question. It is important to note that performance loss is inevitable during distillation due to the limited scale of dataset used and the lightweight architecture of SAK. Therefore, we also include a student distilled from a single teacher to investigate the help of other teachers during distillation.

• Here we provide an evaluation of the monocular depth estimation task on the NYUd dataset [1] (note it is different from the NYUDv2 dataset used in our primary experiments). We follow the evaluation protocol [2] used by DINOv2 and use the identical experimental setups and decode heads (lin. 1 and lin. 4). Since DINOv2 does not provide a complete codebase for this task, we reproduce the pipeline. We also include ViT, SAM, Theia, and baselines reported in the DINOv2 paper, namely OpenCLIP [3], MAE [4], and DINO [5].

  The effectiveness of synergizing multiple VFM teachers is clearly evidenced by the results, as distillation from three teachers improves SAK’s performance on depth estimation, bringing it closer to the upper bound of the teacher. Notably, it also surpasses other larger foundation models.

• As mentioned, SAM is specifically designed for promptable segmentation rather than semantic segmentation, which aligns with its semantic-free training objective. Prior studies [6,7] have also demonstrated SAM's limitations in semantic segmentation. In our evaluation, we have assessed SAM’s performance on the benchmarks in Figure 1 (page 1) and Table 12 (Appendix C, page 24), with and without fine-tuning. Similarly, we provide additional results of a student distilled from solely a SAM-B teacher. The results indicate that SAM struggles to achieve good results with pre-trained knowledge; while fine-tuning improves its performance, it still falls short of the performance achieved by SAK.

Weakness3

We agree with the reviewer’s point that iteratively forwarding VFM teachers increases training costs, and these costs cannot be overlooked. However, we would like to emphasize that all multi-teacher distillation methods require this process, and when compared to other baselines, such as RADIO, SAK demonstrates remarkable efficiency.

• In the following table, we calculate the number of parameters and computational costs (measured by MACs, Multiply–accumulate operation) for teachers and students in RADIO and SAK, both using a ViT-L backbone for the student. The results show that RADIO incurs 2.5 times the parameters and forwarding costs of SAK. Additionally, RADIO’s larger input size of 1,024 introduces more tokens in the representations, further reducing the backward efficiency. Hence, SAK shows significantly lower demands in RAM, GPU memory, training time, and storage.

• Moreover, as listed in Table 20 (Appendix D.3, page 28), RADIO is trained on the large-scale DataComp-1B dataset (1.4B images) with 614M total samples seen. In contrast, SAK is trained on ImageNet-1k dataset (1.2M images) with only 36M samples seen, further highlighting the efficiency of SAK in training and memory costs.

We appreciate the reviewer for highlighting the importance of training costs. We will include this analysis and discussion in our manuscript.

[1] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, 2012.

[2] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. IEEE Transactions on Image Processing, 2024.

[3] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.

[4] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.

[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In CVPR, 2021.

[6] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. AM-RADIO: Agglomerative vision foundation model reduce all domains into one. In CVPR, 2024.

[7] Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. SAM-CLIP: Merging vision foundation models towards semantic and spatial understanding. In CVPR, 2024a.

[8] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. TPAMI, 44(7): 3614–3633, 2021.

[9] Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. MTI-Net: Multi-scale task interaction networks for multi-task learning. In ECCV, 2020.

Weakness2

Why cannot SAK outperform teachers at every task?

As noted by the reviewer, performance loss is inevitable during distillation, since our distillation process is lightweight compared to the extensive pretraining of VFM teachers.

As detailed in Table 1 (page 3), our VFM teachers are pretrained on large-scale datasets containing hundreds to thousands of times more image samples than the ImageNet-1k dataset used for distillation. Moreover, the Teacher-Specific Adapter Path, designed to preserve teachers’ biases, utilizes a standard and simple architecture with less than 5% of the backbone’s parameters and computations. Consequently, it is reasonable that the distilled SAK cannot fully inherit teachers’ knowledge when compressing three teacher models into a single student of similar model size. Another reason is that SAM operates at a higher input resolution of 1,024, rather than 512 for other models, which bolsters its strength in tasks like boundary detection that rely heavily on fine image details.

Why can SAK outperform teachers?

• The observed superiority of SAK over teachers stems from our motivation to synergize the complementary strengths of teachers. Complementarity exists not only across tasks but also within tasks. Take semantic segmentation as an example, as illustrated in Figure 1 and qualitative analysis (Section 2.2, page 4), DINOv2 excels at capturing localized features but CLIP offers strong object-level understanding with its rich semantic knowledge from the language domain. SAM produces exceptional fine-grained pixel-level masks due to its higher resolution. Additional analysis of representations in Appendix A (page 20) further supports these observations. By preserving the intra-task representation biases during distillation and amalgamating them using proposed Mixture-of-Representations, SAK achieves improved performance.

• Another contributing factor is the balance between common knowledge and task-specific information. Regarding saliency estimation and surface normal estimation, three VFM teachers perform suboptimally when frozen, as they lack downstream task-specific information. Though fine-tuning can alleviate this limitation, it results in degradation of accuracy in segmentation tasks for DINOv2 and CLIP, as shown in the table below (Table 12 in Appendix C, page 24). This trade-off, known as the negative transfer problem in multi-task learning [8,9], highlights the challenge of balancing task-specific and pretrained knowledge.

  In contrast, SAK preserves the representation biases of the frozen teachers within the TSAP modules while leveraging the Teacher-Agnostic Stem to learn shared knowledge and downstream information. This disentanglement ensures knowledge diversity without mutual interference. Moreover, MoR Routers dynamically weigh and combine the most relevant representations for each task, bridging the gap between general-purpose knowledge and task-specific characteristics.

We appreciate the reviewer for the insightful question and will include this discussion in our manuscript.

Question1

We appreciate the reviewer’s detailed feedback. We have already illustrated the teacher/student mismatch issue in Appendix B.1 (page 21, line 1127). When the student and teacher have different architectures, discrepancies could arise in the spatial resolutions due to varying input sizes or patch sizes and in the number of channels due to diverse embedding dimensions. In the Teacher-Specific Adapter Paths, all representations maintain the same shape as those in Teacher-Agnostic Stem. To align the student’s representations with those of the teachers, we introduce an upsampling layer to match the spatial resolution, and a linear layer for channel mapping when necessary, following established practices [9,10]. Note that the linear layer is only required during distillation and does not add any overhead during inference.

[1] Hanrong Ye and Dan Xu. TaskPrompter: Spatial-channel multi-task prompting for dense scene understanding. In ICLR, 2023b.

[2] Hanrong Ye and Dan Xu. TaskExpert: Dynamically assembling multi-task representations with memorial mixture-of-experts. In ICCV, 2023a.

[3] Yuqi Yang, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts. In CVPR, 2024d.

[4] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, 2012.

[5] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. IEEE Transactions on Image Processing, 2024.

[6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.

[7] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.

[8] Caron, Mathilde, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In CVPR, 2021.

[9] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. AM-RADIO: Agglomerative vision foundation model reduce all domains into one. In CVPR, 2024.

[10] Shengcao Cao, Mengtian Li, James Hays, Deva Ramanan, Yu-Xiong Wang, and Liang-Yan Gui. Learning lightweight object detectors via multi-teacher progressive distillation. In ICML, 2023.

We appreciate your time and efforts in reviewing our paper, providing valuable comments and constructive suggestions. We are encouraged by your acknowledgement of our work as a “novel method” that achieves “state-of-the-art performance” based on “thorough comparisons,” and offers “insights.” We provide the following responses to address your concerns.

Weakness1

We would like to clarify a potential misunderstanding of the inference procedure. Our framework does not require teachers during inference. Instead, the representations distilled from the teachers are generated directly by the lightweight TSAP modules, which introduce minimal additional parameters and computational overhead.

To provide a clearer perspective, we present the number of parameters and computational cost (measured by MACs, Multiply–accumulate operations) introduced by our modules when integrated into the ViT-B or ViT-L backbone. The calculation is based on a student model corresponding to three teachers on the five-task PASCAL-Context dataset. The table indicates that each TSAP branch accounts for less than 5% of the backbone’s parameters and computations (4M vs. 86M parameters, 4G vs. 88G MACs with ViT-B). Similarly, each MoR Router is as lightweight as 2M parameters and 2G MACs, even with the larger ViT-L backbone.

We appreciate the reviewer for highlighting this concern and will add related discussion on inference cost to our manuscript.

Weakness2

• We would like to highlight that our work targets multi-task learning, and the tasks we evaluated are defined by the dataset, since each image sample must be labeled for every task included. PASCAL-Context and NYUDv2 are multi-task benchmarks used by most recent works [1,2,3], allowing direct comparisons with state-of-the-art methods. We would also like to note that depth estimation is already included in the NYUDv2 dataset.

• Due to time constraints, we currently supplement an evaluation of the monocular depth estimation task on the NYUd dataset [4] (note it is different from the NYUDv2 dataset). We follow the evaluation protocol [5] used by DINOv2 and use the identical experimental setups and decode heads (lin. 1 and lin. 4). Since DINOv2 does not provide a complete codebase for this task, we reproduce the pipeline. We also include ViT, SAM, Theia, and baselines reported in the DINOv2 paper, namely OpenCLIP [6], MAE [7], and DINO [8].

  While SAK does not fully match the teacher’s performance due to inevitable loss during distillation and different patch size, it surpasses other foundation models and baselines, even those with larger backbones. As noted in Table 1 (page 3), DINOv2 is trained on the large-scale LVD-142M dataset, which contains over 100 times more image samples than the ImageNet-1k dataset we use for distillation (CLIP and SAM are trained on even huger datasets). Hence, it is reasonable that our approach cannot outperform the teachers on their proficient tasks. We believe that leveraging larger datasets could further enhance knowledge transfer during distillation and benefit downstream tasks. We will include this analysis in our manuscript.

Dear Reviewer yJkf,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. Thank you once again for your valuable and constructive comments, and we have incorporated the results of our discussion into the revised manuscript.

Given that the last day that reviewers can post responses is December 2, we would like to know if you have any additional comments. We would be more than happy to include them before the discussion phase ends.

Best regards,

Authors

Question2

Yes, SAK is a highly flexible framework that can be easily adjusted to add new teachers or new downstream tasks.

Add new VFM teachers

When adding a new teacher, we freeze the existing Teacher-Agnostic Stem and Teacher-Specific Adapter Paths to preserve the knowledge transferred from the previous teachers A new TSAP module is then introduced and distilled for the new teacher, facilitating efficient adaptation. In the second stage, we train on the downstream dataset with all VFM teachers and tune the entire student model.

We conduct an experiment by incorporating a new SigLIP teacher into a student already distilled from DINOv2, CLIP, and SAM. As shown in the following table, including SigLIP teacher boosts performance on Semseg and Normal tasks and yields competitive results on Saliency and Boundary tasks. However, since SigLIP encodes knowledge similar to CLIP during pretraining and does not surpass CLIP on the benchmark, it is reasonable that it could not lead to further improvements. Moreover, the additional representations may increase the difficulty for the MoR Routers in learning optimal weights, potentially accounting for the degradation in human parsing.

Add different tasks

Additionally, when applying to different downstream tasks, it is only necessary to add task-specific decoders and perform the second-stage training (with distillation) or fine-tuning (without distillation), as the first-stage distillation is agnostic to the downstream tasks. This process is significantly more efficient than the first stage. Meanwhile, we can readily remove a branch of TSAP if it is no longer necessary**,** without affecting the functionality of the TAS or other TSAP branches, ensuring the modularity and adaptability of our design. We have also shown in Figure 5 (page 10) and its accompanying analysis (page 9, line 470) that SAK exhibits strong robustness and generalization capability in downstream tasks.

We appreciate the reviewer for the insightful question and will add this discussion and experimental results to our manuscript.

[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2024.

[2] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.

[3] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024b.

Question1

Principles of VFM teacher selection

• It is well-established that DINOv2, CLIP, and SAM are among the most widely used Vision Foundation Models. Prior baselines RADIO and Theia also employ them for experiments. Besides, RADIO uses DFN CLIP and SigLIP, both of which are improved models derived from CLIP. However, as shown in the following table from our early attempts, DFN CLIP and SigLIP fail to attain comparable outcomes to CLIP. For this reason, we choose to adhere to CLIP when selecting teachers.

• Theia utilizes ViT (pretrained on ImageNet) and Depth Anything [3] as additional teachers. Since the Teacher-Agnostic Stem in SAK is initialized with an ImageNet-pretrained ViT backbone, we do not include ViT as an additional teacher to reduce training costs. While Depth Anything is a high-impact VFM, we leave its exploration for future work due to limited time and resources.

• Additionally, we avoid using very large models (e.g. ViT-H or ViT-g based) in our experiments due to constraints in computational resources (RADIO uses 64 GPUs, while we can only use 8 GPUs). We would also like to mention that SAK is a highly flexible framework, capable of integrating and benefiting further from more powerful VFM teachers, making it adaptable to various settings.

Sensitivity of SAK to VFM teacher selection

• We have already provided a detailed analysis of teacher selection in Section 4.3 (page 9, line 459). Table 7 (page 9) shows the performance experimented using different combinations of teachers. The results indicate that integrating knowledge from all three teachers leads to the strongest overall performance. Table 8 (page 10) and its analysis (page 9, line 476) further show that scaling up the teacher models proves beneficial.

• Here we supplement another analysis with a SAK-B student distilled from DINOv2-B and SigLIP-B teachers. Compared with the student distilled from DINOv2-B and CLIP-B teachers, it exhibits better performance on three out of five tasks, while achieving comparable overall accuracy, further underscoring the robustness and adaptability of our method. We will include the above discussions in our manuscript.

Weakness2

• We build an upper-bound model using the frozen encoders of three VFM teachers – DINOv2, CLIP, and SAM – along with a learnable ViT encoder as a surrogate for the Teacher-Agnostic Stem (TAS). All components employ a ViT-B backbone. For fusing the representations, we consider element-wise addition, channel concatenation, and our proposed Mixture-of-Representations.

• The following table shows that a naive addition of representations fails to fully leverage the teachers’ knowledge, as mentioned in the introduction (page 2, line 84) and further supported by our ablation study in Section 4.3 (page 8, line 424) and Table 3 (page 8). While concatenation addresses this limitation and leads to upper-bound performance, it significantly increases the parameter count and computational overhead since it increases the dimension of the fused representations. In contrast, our MoR approach surpasses addition by 0.5% while maintaining comparable parameters and MACs, demonstrating its effectiveness even in this alternative setup.

• Compared to these upper bounds, our SAK outperforms the naive addition with around 1/3 of the parameters and half of the computational cost. This further validates the effectiveness and efficiency of our method. We appreciate the reviewer’s valuable suggestion and will add the upper bound results into our manuscript.

We appreciate your time and efforts in reviewing our paper, providing valuable comments and constructive suggestions. We are encouraged by your acknowledgement of our work in achieving “reduced computational cost” while “effectively improving multi-task performance,” as demonstrated by the “sufficient ablation experiments.” We provide the following responses to address your concerns.

Weakness1

• First, we would like to emphasize that our work is less targeted at single task learning, e.g., image level reasoning or vision-language learning. Instead, our key motivation lies in the inherent representation biases of VFMs, which result in advantages and disadvantages across different vision tasks. Our SAK synergizes these complementary biases to enhance multi-task learning. Additionally, we note that another related baseline, Theia, which focuses on robot learning, does not provide comparisons with RADIO on these tasks.

• Here we supplement the experiments of VLM. Following the setup of RADIO, we integrate SAK into LLaVA-1.5 [1,2], freezing the Teacher-Agnostic Stem and Teacher-Specific Adapter Path, and fine-tuning only the MoR Router. We evaluate three tasks – GQA, POPE, and VQAv2 – at resolutions of 432 and 512, consistent with RADIO. We also include DINOv2-L, CLIP-L, and SAM-H for comparison.

• The following table shows that while SAK is trained on significantly less data, and with smaller VFM teachers, it achieves comparable performance to the teachers and RADIO on GQA and POPE. The performance gap on VQAv2 is largely attributable to the simplicity of our distillation setup, which prioritizes training efficiency. Regarding dataset scale, as listed in Table 1 (page 2) and Table 20 (Appendix D.3, page 28), all VFM teachers and RADIO use large-scale datasets, e.g., DINOv2 uses 142M images, CLIP uses 400M, RADIO uses 1.4B. In contrast, our SAK distillation relies on ImageNet-1k, which has only 1.2M images. This limited dataset diversity could significantly impact performance on VQA tasks, which require extensive training data.

  Meanwhile, RADIO employs larger VFM teachers, including DFN CLIP-H, SigLIP-SO400M, DINOv2-g, and SAM-H, which result in 2.5 times the parameters and forwarding costs of SAK with teachers based on ViT-L. RADIO also uses a larger resolution of 1,024 during its distillation (In LLaVA experiments, the image resolution is 432 or 512), enhancing image information extraction but at a substantial increase in forwarding and backwarding costs. Considering these factors, it is reasonable that SAK cannot outperform RADIO in VLM applications.

• We believe that with access to larger-scale datasets and more powerful teachers during distillation, SAK could achieve competitive or superior results. However, such resources are currently unavailable due to time and computational constraints. We will add this discussion and the corresponding results to our manuscript.

We would like to provide updates based on additional experiments on image level reasoning. Following the setup of DINOv2, we evaluate linear probing on the ImageNet-1k dataset. We freeze the backbones and train a linear classifier and the MoR Routers of SAK using the same hyperparameters as DINOv2. For comparison, we also evaluate the VFM teachers and the state-of-the-art baseline RADIO, while reporting Theia’s result from its original paper.

The following table shows that SAK outperforms the two baselines, despite a performance gap compared to the DINOv2 and CLIP teachers due to inevitable loss during distillation. It is worth noting that RADIO uses more powerful teachers, including DINOv2-g/14-reg (accuracy 87.1), while Theia uses DINOv2-L/14 teacher (accuracy 86.3).

Please feel free to let us know if you have any additional comments, we would be happy to discuss them.

Dear Reviewer gxTF,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. Thank you once again for your valuable and constructive comments, and we have incorporated the results of our discussion into the revised manuscript.

Given that the last day that reviewers can post responses is December 2, we would like to know if you have any additional comments. We would be more than happy to include them before the discussion phase ends.

Best regards,

Authors

Weakness2

We appreciate the reviewer’s detailed feedback regarding hyperparameter tuning. We would like to clarify that our experimental setups basically follow prior work and do not involve hyperparameter tuning across different experiments.

• Specifically, for the first stage, we follow the setup of the state-of-the-art baseline RADIO for optimizer, learning rate, LR scheduler, and distillation loss (in Eq. 5, page 7, line 332), while adopting another baseline Theia’s batch size configuration since we use the same dataset. To enhance training efficiency, we reduce the number of epochs. For the second stage, we use the identical hyperparameters (including data augmentation and task-specific loss functions) as those in prior multi-task learning works [1,2,3]. We offered detailed specifications of implementation and experiments in Appendix B (page 21-23).

• The only hyperparameter we have tried tuning is the balancing factor $\gamma$ in Eq. 6 (page 7, line 340), which balances the distillation loss and task losses during the second stage. Its impact was analyzed in Table 19 (Appendix C, page 26) and corresponding text (page 25, line 1347). For all other experiments, we use a default value of 1.0 for simplicity.

In summary, we would like to underscore that fixed hyperparameters are used across all experiments (except Table 19 for experiments on varying $\gamma$​), confirming the contributions of our proposed methodology without extensive hyperparameter search.

[1] Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding. In ECCV, 2022.

[2] Hanrong Ye and Dan Xu. TaskExpert: Dynamically assembling multi-task representations with memorial mixture-of-experts. In ICCV, 2023a.

[3] Yuqi Yang, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts. In CVPR, 2024d.

[4] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML, 2019.

[5] Minghao Fu, Ke Zhu, and Jianxin Wu. Dtl: Disentangled transfer learning for visual recognition. In AAAI, 2024.

[6] Otniel-Bogdan Mercea, Alexey Gritsenko, Cordelia Schmid, and Anurag Arnab. Time-Memory-and Parameter-Efficient Visual Adaptation. In CVPR, 2024.

[7] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision Transformer Adapter for Dense Predictions. In ICLR, 2023.

[8] Dongshuo Yin, Xueting Han, Bin Li, Hao Feng, and Jing Bai. Parameter-efficient is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions. In ACM MM, 2024.

[9] Fengze Jiang, Shuling Wang, and Xiaojin Gong. Task-Conditional Adapter for Multi-Task Dense Prediction. In ACM MM, 2024.

[10] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR, 2017.

We appreciate your time and efforts in reviewing our paper, providing valuable comments and constructive suggestions. We are encouraged by your acknowledgement of our work as “a good solution” in multi-task learning, noting its “substantial performance gains” and recognizing it as a “well organized and well written” paper. We provide the following responses to address your concerns.

Weakness1

We would like to emphasize that our module architectures are simple, introducing minimal additional parameters and computational overhead while achieving significant improvements over the previous state of the art, which also requires multiple VFMs. Furthermore, compared to prior multi-task approaches that rely on complicated module designs [1,2,3], our architecture is much simpler, with fewer parameters, yet delivers substantial performance gains, as shown in Tables 5 (page 8) and 6 (page 9).

• As described in Section 3.2 (page 6, line 283), the adapter follows a standard structure consisting of a two-layer MLP [4], which is widely adopted in existing approaches. The “layered structure” mentioned by the reviewer is also a common practice in parameter-efficient fine-tuning [5,6]. Notably, some methods employ more than one adapter module to each ViT block or utilize more complicated adapter structures [7,8,9]. Section 3.3 (page 6, line 313) elaborates on the MoR Router, which also employs a two-layer MLP design. The noisy gating mechanism, described in Eq. 4, is a standard technique in MoE approaches [10].

• We have validated the individual contributions of the components with these simple and lightweight designs in the ablation study in Table 3 (page 8) of the original submission. Here we supplement this by presenting the number of parameters and computational cost (measured by MACs, Multiply–accumulate operations) introduced by our modules, as shown in the table below. The calculations are based on a student model without decode heads, as heads are identical across configurations. The table indicates that each TSAP branch accounts for less than 5% of the backbone’s parameters and computations (4M vs. 86M parameters, 4G vs. 88G MACs). Similarly, each MoR Router is as lightweight as 1.2M parameters and 1.2G MACs.

  The results confirm that TSAP effectively preserves the representation biases from the teachers, as evidenced by the higher similarity between the student and teachers (“Rep Sim”). Additionally, we validate that simply adding diverse representations (“N MoR”) does not lead to an overall improvement and may even fall behind the student without biases. With both components, SAK not only preserves the representation biases but also maximizes multi-task performance across all five tasks, with an overall improvement of 1.18%.

• Additionally, we would like to underscore that our framework does not rely on the specific structure of the TSAP and MoR Routers. It is compatible with various adapter and router designs, allowing for potential enhancements in model complexity and performance with more dedicated designs.