Balancing Domain-Invariant and Domain-Specific Knowledge for Domain Generalization with Online Knowledge Distillation

Dear Reviewer 5xaq, Thank you for your thoughtful feedback.

1. While the CLIP-Adapter serves as a parameter-efficient fine-tuning method to enhance CLIP’s performance on various downstream tasks, our methods use adapters to enable the teacher model to incorporate domain-specific knowledge from different domains. Our main focus is not on proposing new adapter structures, such as Tip-Adapter [1], but on addressing the challenge of distilling both domain-invariant and domain-specific knowledge into a single student model, which is a challenge in domain generalization. Our key contribution lies in the use of an online knowledge distillation strategy, which distinguishes it from existing approaches that primarily rely on offline distillation methods. By employing KL divergence, our method allows the teacher model (the domain expert) to dynamically update based on feedback from the student model, facilitating more effective knowledge transfer.
2. We will provide additional clarification in the revised version. Rather than relying solely on a similarity-based metric, we employ cross-entropy loss because it inherently includes calculating similarity metrics [2]. In CLIP, similarity is computed as a prerequisite for the cross-entropy calculation. For each domain d_i, we generate m prompts, where m is the number of class labels. Using cross-entropy loss allows us to not only maximize the similarity between an image and its ground truth prompt but also to minimize the similarity between the image and its unmatched class prompts. This dual objective aligns with the training strategy used in the original CLIP paper [2].
3. The results presented in Table 1 for NKD, RISE, and BOLD are based on knowledge distillation from ViT-B/32 to ResNet-50 for a consistent and fair comparison. This setup ensures the comparison is not limited to BOLD but applies uniformly across all methods. Table 2 provides additional results for NKD, RISE, and BOLD under different distillation settings, also ensuring fairness. Regarding the discrepancy in RISE’s performance, the original RISE implementation only includes four benchmarks (PACS, OfficeHome, VLCS, and Terra). To ensure fair comparison across all baselines and benchmarks, we replicated RISE in our library and conducted all experiments under the same conditions, including consistent data augmentation, prompts, and other settings. The code for our replication, along with the other baselines, is available in the GitHub repository linked in the paper.

[1] Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., ... & Li, H. (2022, October). Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision (pp. 493-510). Cham: Springer Nature Switzerland.

[2] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

Dear Reviewer,

As the discussion period draws to a close, we would greatly appreciate your feedback on whether our responses have sufficiently addressed your concerns. If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you for your time and thoughtful consideration.

Best regards

Dear Reviewer 5xaq

We hope this message finds you well. We sincerely appreciate your time and effort in reviewing our submission and providing valuable insights.

We wanted to kindly follow up regarding our rebuttal as the discussion phase is nearing its conclusion.

We would greatly appreciate any additional comments or feedback you may have regarding our response to your reviews. Your input is invaluable in clarifying and strengthening the work.

If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you once again for your time and consideration. We look forward to any further thoughts you might have.

Best regards

Thanks the authors for the response. After reading the rebuttal, my concerns regarding the performance divergence still remains. This point was also raised by Reviewer QY7Z. Therefore, I would like to maintain my original score.

Dear Reviewer Jua5, Thank you for your thoughtful feedback.

1. We will incorporate the recommended literature into the Related Work section to offer a more comprehensive discussion.
2. Thank you for your feedback regarding the ablation study. To better address your concerns, could you kindly provide detailed requirements about any additional ablation experiments that you would like to include? In Table 3, we have already included results from an ablation study that examines (1) distilling invariant knowledge only, (2) distilling both invariant and specific knowledge in an offline setting, and (3) distilling both invariant and specific knowledge in an online setting. We would add a discussion in the revised version about how conducting an ablation study focusing solely on distilling specific knowledge does not align with the domain generalization context. As discussed in the paper, we aim to demonstrate how domain-specific knowledge can complement domain-invariant knowledge in practice. Relying exclusively on specific knowledge would inevitably result in poor performance, as it lacks the generalization capacity necessary for domain generalization tasks.
3. Thank you for raising this important question. We will provide additional clarification in Appendix A5 of the updated version. Although the learning objectives of domain-invariant and domain-specific knowledge may appear conflicting, they can be complementary. The embedding space of the model is multi-dimensional, and not all dimensions need to serve both objectives simultaneously. By carefully designing the loss function, it is possible to coordinate the coexistence of these two types of knowledge within the embedding space[1][2].

[1]: Sener, O., & Koltun, V. (2018). Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31.

[2]: Chen, L., Fernando, H., Ying, Y., & Chen, T. (2024). Three-way trade-off in multi-objective learning: Optimization, generalization and conflict-avoidance. Advances in Neural Information Processing Systems, 36.

Dear Reviewer,

As the discussion period draws to a close, we would greatly appreciate your feedback on whether our responses have sufficiently addressed your concerns. If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you for your time and thoughtful consideration.

Best regards

Dear Reviewer Jua5

We hope this message finds you well. We sincerely appreciate your time and effort in reviewing our submission and providing valuable insights.

We wanted to kindly follow up regarding our rebuttal as the discussion phase is nearing its conclusion.

We would greatly appreciate any additional comments or feedback you may have regarding our response to your reviews. Your input is invaluable in clarifying and strengthening the work.

If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you once again for your time and consideration. We look forward to any further thoughts you might have.

Best regards

Dear Reviewer QY7Z, Thank you for your thoughtful feedback.

1. There are two key differences between our work and [1]. First, the motivation behind leveraging domain-specific knowledge differs significantly. In [1], the assumption is that there is a correlation between the domain-specific representation and the class label Y. In contrast, our work does not rely on this assumption. Second, our work introduces the novel approach of distilling both domain-invariant and domain-specific knowledge from a large pretrained model to a student model through a carefully designed online knowledge distillation strategy. By comparison, [1] addresses the problem by enabling the model to learn domain-invariant and domain-specific knowledge using an adversarial network with a meta-learning strategy. While both our work and [1] aim to address the challenge of integrating domain-invariant and domain-specific knowledge, they adopt distinct methodologies and pose different research questions, contributing unique perspectives to the field.
2. The performance differences arise from variations in the implementation protocols of baseline methods in domain generalization. In this field, two popular libraries are commonly used as implementation protocols: DomainBed [2] and DDAIG [3, 4]. By following the survey [5], our work follows the DDAIG protocol, which may result in discrepancies in the DomainBed performance of some older baselines, such as ERM, on classic benchmarks like PACS and VLCS. Importantly, all results in our work are presented under a consistent and fair comparison framework. Details of the implementation, including data loader, models, and benchmarks, are provided in the GitHub repository linked in the paper. Lastly, while these discrepancies affect specific baselines on classic benchmarks, they do not impact the conclusions drawn in our paper.

[1] Bui, M. H., Tran, T., Tran, A., & Phung, D. (2021). Exploiting domain-specific features to enhance domain generalization. Advances in Neural Information Processing Systems, 34, 21189-21201.

[2] Gulrajani, I., & Lopez-Paz, D. (2021). In search of lost domain generalization. ICLR.

[3] Zhou, K., Yang, Y., Hospedales, T., & Xiang, T. (2020, April). Deep domain-adversarial image generation for domain generalization. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 07, pp. 13025-13032).

[4] Zhou, K., Yang, Y., Qiao, Y., & Xiang, T. (2021). Domain generalization with mixstyle. ICLR.

[5] Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022). Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4), 4396-4415.

Dear Reviewer,

As the discussion period draws to a close, we would greatly appreciate your feedback on whether our responses have sufficiently addressed your concerns. If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you for your time and thoughtful consideration.

Best regards

Dear Reviewer QY7Z

We hope this message finds you well. We sincerely appreciate your time and effort in reviewing our submission and providing valuable insights.

We wanted to kindly follow up regarding our rebuttal as the discussion phase is nearing its conclusion.

We would greatly appreciate any additional comments or feedback you may have regarding our response to your reviews. Your input is invaluable in clarifying and strengthening the work.

If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you once again for your time and consideration. We look forward to any further thoughts you might have.

Best regards

Dear Reviewer XSvj, Thank you for your thoughtful feedback.

1. Your understanding of the domain loss is accurate. As detailed in Equation 2, the domain loss assigns positive values to the corresponding domain (indicating minimization) and negative values to other domains (indicating maximization). This setup ensures the model learns to distinguish between the corresponding domain and other domains and aligns with common practices for loss function in similar contexts.
2. The KL divergence is indeed an asymmetric distance measure, and the direction of guidance - whether from the teacher to the student or from the student to the teacher differs when computing specific distillation loss. In the current version, we provide a general function for computing the KL divergence without explicitly clarifying this distinction. We will address this oversight in the updated version by clearly specifying the guidance direction in each context and providing the corresponding definitions.
3. In the theoretical discussion section, our analysis focuses on the shared domain-specific knowledge across the source and target domains, not on shared characteristics. The term “shared characteristics” is mentioned in the Introduction as a high-level description of domain-specific features. Then, the knowledge learned from these domain-specific features is what we call domain-specific knowledge. We appreciate your feedback and will ensure this distinction is made clear in the revised version.
4. In Equations (7) and (10), $L(h, D^i_s)$ represents the original student loss on source domain $i$ without guidance from the teacher model. Through the process of knowledge distillation, the student loss is reduced compared to the original loss $L(h, D^i_s)$, and it approximates $L(h_T, D^i_s) + \epsilon$. As a result, leveraging knowledge distillation allows us to replace $L(h, D^i_s)$ with $L(h_T, D^i_s)$. Since $L(h_T, D^i_s)$ is less or equal to $L(h, D^i_s)$, Equation (11) provides a tighter bound compared to Equation (7).
5. While incorporating domain-invariant and domain-specific knowledge may appear to introduce conflicting objectives, these forms of knowledge can also be complementary. The model’s embedding space is inherently multi-dimensional, allowing different dimensions to focus on distinct objectives without interference. By carefully designing the loss function, it is possible to harmonize these objectives, enabling their coexistence and effective coordination [1][2].
6. As discussed in the paper, the performance of knowledge-distillation-based methods is inherently influenced by the teacher model’s performance, a characteristic common to all such approaches. As shown in Table 4, the teacher model utilized in our work, CLIP, performs poorly on the Terra Incognita and Digits datasets. Consequently, methods like NKD, RISE, and BOLD also exhibit suboptimal performance on these datasets. However, by leveraging domain-specific knowledge and employing an online knowledge-distillation strategy, BOLD achieves significant improvements compared to other knowledge-distillation-based methods, demonstrating its relative effectiveness in addressing this limitation.
7. Thank you for highlighting this point. Initially, we did not include the adapter-based methods in our analysis, as doing so might have led to an unfair comparison given that CLIP-Adapter methods involve more parameters than classic RN50 methods. However, in response to your suggestion, we have added both the quantitative results and the qualitative results in Appendix A4 to provide a more comprehensive evaluation.

[1] Sener, O., & Koltun, V. (2018). Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31.

[2] Chen, L., Fernando, H., Ying, Y., & Chen, T. (2024). Three-way trade-off in multi-objective learning: Optimization, generalization and conflict-avoidance. Advances in Neural Information Processing Systems, 36.

Dear Reviewer,

As the discussion period draws to a close, we would greatly appreciate your feedback on whether our responses have sufficiently addressed your concerns. If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you for your time and thoughtful consideration.

Best regards

Dear Reviewer XSvJ

We hope this message finds you well. We sincerely appreciate your time and effort in reviewing our submission and providing valuable insights.

We wanted to kindly follow up regarding our rebuttal as the discussion phase is nearing its conclusion.

We would greatly appreciate any additional comments or feedback you may have regarding our response to your reviews. Your input is invaluable in clarifying and strengthening the work.

If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you once again for your time and consideration. We look forward to any further thoughts you might have.

Best regards

Dear Reviewer de9i, Thank you for your thoughtful feedback.

1. As the theoretical foundation for the effectiveness of online knowledge distillation remains underexplored in the broader field of knowledge distillation, our paper primarily focuses on providing a qualitative analysis. To address the limitations in theoretical rigor, we have conducted extensive experiments and ablation studies to empirically validate the effectiveness of the proposed method. We will provide more rigorous proof in the revised version.
2. Imbalanced dataset distribution is indeed an important practical concern, particularly regarding the adequate training of domain experts. In our framework, domain experts are lightweight adapters consisting of a two-layer fully connected network rather than large-scale, deep neural networks. This design ensures that these adapters can be adequately trained even for domains with only a few hundred images. As demonstrated in our experiments on benchmarks such as Terra Incognita, NICO++, and DomainNet, our method achieves strong performance despite the presence of imbalanced distributions. For instance, our framework achieves 85.3% average accuracy on NICO++ and 50.9% on DomainNet. Although our approach does not explicitly address dataset imbalance, these results suggest that the proposed framework is inherently robust to such challenges. To provide further clarity, we have expanded on this discussion and included a detailed visualization of dataset distributions in Appendix A3. In further work, we aim to extend our framework to better address pronounced imbalance problems, tailoring it more explicitly to such scenarios.

Dear Reviewer,

As the discussion period draws to a close, we would greatly appreciate your feedback on whether our responses have sufficiently addressed your concerns. If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you for your time and thoughtful consideration.

Best regards

Dear Reviewer de9i

We hope this message finds you well. We sincerely appreciate your time and effort in reviewing our submission and providing valuable insights.

We wanted to kindly follow up regarding our rebuttal as the discussion phase is nearing its conclusion.

We would greatly appreciate any additional comments or feedback you may have regarding our response to your reviews. Your input is invaluable in clarifying and strengthening the work.

If we have successfully clarified or resolved the issues raised, we kindly ask you to consider revising your score.

Thank you once again for your time and consideration. We look forward to any further thoughts you might have.

Best regards