AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation

First and foremost, We would like to appreciate the insightful comments and advice provided by reviewer 8qr9. Guided by these recommendations, we have made several amendments to our work, which are highlighted in red within the PDF.

W1: The class-specific mask is essential in anchor learning, but there lacks visualizations on masks to show whether they correctly captured class-specific information.

A1: Thanks for the suggestion. According to reviewer 8qr9's advice, we respectfully add visualization results (Figure 1 and Figure 2) on the mask by t-SNE in Appendix C.1.

Figure 1 (original $z$) can be referred to https://p.ipic.vip/0btpbb.png

Figure 2 (masked $z$) can be referred to https://p.ipic.vip/7riqhk.png

We quote it here:

Appendix C.1 Visualization On Mask Provided by AnchorNet

In Module 2, AnchorNet integrates the mask operator $m$ and the mapping function $\psi$ into a lightweight neural network. The mask is class-specific and plays a crucial role in retaining domain-invariant knowledge in the latent space. To vividly demonstrate the effectiveness of the mask, we conduct t-SNE on the latent variables and the masked version of them. To be specific, we use AnchorNet and Encoder trained under the setting AW$\rightarrow$D in Office-31 and select 32 images for each class (31 classes in total). For each image $(x,y)$, we encode it to get the latent variable $z=E(x;\theta_e)$, and obtain the class-specific mask $m(y;\theta_a)$. Next, we obtain the masked latent variable $z'=(1-m(y;\theta_a))\odot z$. The t-SNE results on $z$ and $z'$ are displayed in Figure 1 and 2 Each displays a distribution of data points plotted against two t-SNE components. The points are colored and shaped differently to represent different classes within the latent space.

In Figure 1, the distribution is quite mixed, with no distinct clusters or separation between the different classes. In contrast, in Figure 2, after applying mask operation on the latent variables, there appears to be a more distinct separation between different classes. Clusters of the same shapes and colors are more evident, indicating that the mask operation has enhanced the class-specific knowledge within the latent space.

W2: According to ablation study 5.3 (a), the contribution of anchor is not significant compared to mixup module, which doubts the effectiveness and necessity of anchor learning module. Here raises a question about why anchor learning is necessary and successfully selectively transfer teachers' knowledge.

A2: Thanks for the question. AnchorNet plays a crucial role in our proposed method. We apologize for the ambiguity caused by the title of ablation results. According to the suggestions, we eliminate the ambiguity and add one more ablation setting.

In Section 5.3.1, we discuss the effectiveness of our modules. As we stated, without Module 1, the proposed method cannot work, we ignore the ablation result w/o Module 1.

Since the whole method is inapplicable without Module 1, we ignore it here.

As a result, conventional ablation studies would discuss the ablation result "w/o Module 2", "w/o Module 3", and "w/o Module 2,3". However, Module 2 is an auxiliary module for Module 3, as stated:

Module 2 helps Module 3 generate mixup samples for training

It is also meaningless to check the ablation result of "w/o Module 3". Therefore, our ablation study should be conducted on "w/o Module 2" and "w/o Module 2,3", which in the text are "w/o Anchor" and "w/o Mixup" individually.

Actually, "w/o Mixup" is NOT verifying the effectiveness of Module 3: Mixup Leaning, but is highlighting the harm of blindly adopting DFKD methods under domain shift. As stated in the second paragraph in Section 5.3.1:

Given that w/o Mixup is a simple process of DFKD, the striking performance degradation highlights the urgent need for solutions to OOD-KD.

Therefore, to eliminate ambiguity, we change the name of the ablation results from "w/o Mixup" to "M1" and from "w/o Anchor" to "M1+M3" accordingly.

To further increase the rigor of this ablation study, we dive into Module 3 and add a new ablation setting: w/o Mixup, where the model trains solely on $D_s$ without mixup samples:

First and foremost, We would like to appreciate the insightful comments and advice provided by reviewer 5Sg9. Guided by these recommendations, we have made several amendments to our work, which are highlighted in red within the PDF.

W1: The detail of the scheduler function is not released. Although the important characteristics of scheduler functions are provided in the main body, I cannot figure out the specific scheduler function used in the experiments.

A1: Thanks for the question. For the sake of space, we did not release the details of the scheduler function we used in our experiments. As stated in the main body, the scheduler function should follow 2 properties:

1. $F(a\cdot\sharp{\rm Epoch};a,b)=1$
2. $F(0;a,b)=b$

Besides, it should be monotonically non-decreasing within its domain. In our experiments, we only use the simplest scheduler function - linear function for all settings. When $a$ and $b$ are given, the scheduler function can be determined.

In fact, the idea of scheduler function draws inspiration from Curriculum Learning [1]. For the performance of different and more complex scheduler functions, we refer readers4 to [1] for further reading.

The discussion of scheduler functions we use has been added to the end of Appendix A.

W2: The ablation study of the effectiveness of each component is not clear. The setting seems ambiguous and in need of further explanation.

A2: Thanks for the advice. We apologize for the ambiguity caused by the title of ablation results. According to the suggestions, we eliminate the ambiguity and add one more ablation setting.

In Section 5.3.1, we discuss the effectiveness of our modules. As we stated, without Module 1, the proposed method cannot work, we ignore the ablation result w/o Module 1.

Since the whole method is inapplicable without Module 1, we ignore it here.

As a result, conventional ablation studies would discuss the ablation result "w/o Module 2", "w/o Module 3", and "w/o Module 2,3". However, Module 2 is an auxiliary module for Module 3, as stated:

Module 2 helps Module 3 generate mixup samples for training

It is also meaningless to check the ablation result of "w/o Module 3". Therefore, our ablation study should be conducted on "w/o Module 2" and "w/o Module 2,3", which in the text are "w/o Anchor" and "w/o Mixup" individually.

Actually, "w/o Mixup" is NOT verifying the effectiveness of Module 3: Mixup Leaning, but is highlighting the harm of blindly adopting DFKD methods under domain shift. As stated in the second paragraph in Section 5.3.1:

Given that w/o Mixup is a simple process of DFKD, the striking performance degradation highlights the urgent need for solutions to OOD-KD.

Therefore, to eliminate ambiguity, we change the name of the ablation results from "w/o Mixup" to "M1" and from "w/o Anchor" to "M1+M3" accordingly.

To further increase the rigor of this ablation study, we dive into Module 3 and add a new ablation setting: w/o Mixup, where the model trains solely on $D_s$ without mixup samples:

Framework ablation studies traditionally involve masking parts of the proposed modules for experimental purposes. Yet, it is essential to recognize: 1. Module 1 is fundamental to our method and is non-removable; 2. Module 2 serves to support Module 3. There is no need to test the results only with Module 1&2. Hence, our investigation focuses on the outcomes absent Module 2, and absent both Module 2 and Module 3, denoted as M1+M3 and M1, respectively. Additionally, our analysis dives into Module 3, where we omit the mixup samples to evaluate their critical role, denoted as M1+M2+M3 (w/o Mixup). It's worth noting that there is no need to add one more setting w/o M2 & Mixup here since it makes no difference to M1+M2+M3 (w/o Mixup). Consequently, we get three distinct ablation scenarios: M1, M1+M3, and M1+M2+M3 (w/o Mixup). To be specific, in M1, we directly choose $S$'s best checkpoint in Module 1 and test it on $D_s$. In M1+M2+M3 (w/o Mixup), the model trains solely on $D_s$. In M1+M3, we mask AnchorNet by equating its output $z'$ with its input $z$ and then proceed with the method.

First and foremost, We would like to appreciate the insightful comments and advice provided by reviewer hjEr. Guided by these recommendations, we have made several amendments to our work, which are highlighted in red within the PDF.

W1: The motivation behind the setting of this paper is confusing. Why do we need to use out-of-distribution teachers when we have student domain data? Wouldn’t it be better to directly train the teacher under the student domain?

A1: Thanks for the question.

First, the problem of OOD-KD is ubiquitous. For practical applications, deploying large models directly onto users' devices, such as using facial recognition at entry points or object detection in autonomous driving scenarios, often isn't feasible due to their size and resource requirements. Instead, we deploy a lightweight neural network (e.g. MobileNet) and train it with i ts data and our large-scale teacher model. However, their data are naturally OOD with each other and of course, the teachers' training data (e.g. ethnicity, skin tone in facial recognition and operating environment and weather condition in autonomous) .

In this case, directly distilling the teacher into our model would lead to great performance degradation. To further improve our model's performance, we should address OOD-KD.

• The reason why we need to use out-of-distribution teachers when we have student domain data is that directly training the student under the student domain is not good enough, especially when the amount of data is not sufficient.

  In our main experiments, we have a baseline "w/o KD":

One more baseline "w/o KD" is to train the student model S without the assistance of $T$, starting with weights pre-trained on ImageNet (Deng et al., 2009).

"w/o KD" trains a model solely with student domain data, which is not good enough compared to other methods.

Due to space limitations, We put part of the experiment results here. For more detailed results, we kindly refer readers to our original submission. In Office-31,

The performance gap between "w/o KD" and our proposed method is nearly 5% in VisDA-2017, 4% in Office-Home, and 20% in Office-31. The performance gap is larger and larger as the scale of student domain data decreases but is still significant with 280000 images (VisDA-2017). The performance gap underlines the importance of applying KD for better student-domain performance, especially when the student domain data is not sufficient.

• Usually, we CANNOT directly train the teacher under the student domain for 2 main reasons.
  1. We may not be able to deploy a large-scale teacher model on our devices like mobile phones, etc, due to resource limitations.
  2. Some large models are immutable to us due to patent restrictions or resource limitations, like GPT-4. It is NOT practical for us to finetune them for better performance.

In conclusion, addressing the problem of OOD-KD is of great importance, since it enables us to transfer knowledge from any teacher model we want and improve the performance of our own models.

First and foremost, We would like to appreciate the insightful comments and advice provided by reviewer aobZ. Guided by these recommendations, we have made several amendments to our work, which are highlighted in red within the PDF.

Major 1: The proposed problem definition appears to closely resemble source-free domain adaptation (SFDA) [cite]. The only distinction being that the target domain network is more lightweight compared to the pre-trained teacher network. As a result, the problem setup seems to simply be a specific instance of the broader source-free domain adaptation scenario. This raises concerns about the novelty of the problem statement.

A1: Thanks for the suggestion. It is of great importance to distinguish OOD-KD from SFDA. We value this input and believe readers will find a more comprehensive exploration of this topic in Appendix C.3. I will quote some discussions below.

To clarify this concern, it is important to note that SFDA and OOD-KD are fundamentally different in 2 aspects:

1. In OOD-KD, source and target model are architecture-agnostic.

   To be specific, in most scenarios, we need a lightweight model suitable for our data. Thus we train it with our data and improve its performance with the assistance of a large-scale teacher model. This is the common scenario of KD, where the source model (teacher) and the target model (student) are naturally different in architecture. However, the inaccessibility of teachers’ training data often leads to the problem of OOD-KD and we should address the domain shift problem (difference in distribution between teachers' training data and our data) for better performance.

   On the other hand, SFDA aims at adjusting current model to fit a new domain, i.e, adapt source model to target domain, where target model and source model share the same architecture and base their methods on this assumption.

   The difference in motivation renders SFDA methods unsuitable for OOD-KD, as they operate under the assumption of architecture identity. Therefore, even we have splendid DFDA methods, we need to address the problem of OOD-KD.

2. In OOD-KD, source model is immutable. To be specific, the immutability of large-scale teacher model is quite common in real-world settings due to privacy or resource limitations. For example, it is not practical for us to finetune models like GPT-4.

These 2 properties are natural in the context of knowledge distillation but is usually neglected in SFDA methods. For example, SDDA[1] (reviewer aobZ mentioned) consists of a Domain Adaptation Module, which consists of a shared feature extractor for source and target domain datasets. We also discuss some other SFDA methods in the last second paragraph of Appendix C.3.

However, SFDA does its adaptation on the source model and assumes that the target model shares the same framework as the source model. ... For example, approaches like SHOT (Liang et al., 2020) and SHOT++ (Liang et al., 2022) divide the backbone model into feature extractor and classifier, sharing the classifier across domains. ...... What’s worse, some SFDA methods base their methodology only on ResNet series models (Yang et al., 2021; Kundu et al., 2022), which is inapplicable to most lightweight neural networks. Major 2: Given the dataset $D_s$, which consists of OOD student domain data with labels, I wonder about the performance of the student when trained solely on $D_s$ in a supervised manner. Did the authors conduct any initial tests to gauge baseline performance? Without such a baseline, I question how the authors determined the severity of the problem.

A2: Thanks for the suggestions. Yes, we conduct some tests to gauge baseline performance. The baseline of training solely on $D_s$ is necessary to demonstrate the urgent need for practical solutions to OOD-KD. We appreciate the insight on this matter, which has been detailed within the main text of our article, which is the baseline w/o KD in the initial submission:

One more baseline "w/o KD" is to train the student model S without the assistance of $T$, starting with weights pre-trained on ImageNet (Deng et al., 2009).

As stated in the last second paragraph in Section 5.1, "w/o KD" adopts the initialization pre-trained on ImageNet and is solely trained with $D_s$ without the help of teacher model. We also put part of the results here.