Reinforced Cross-Domain Knowledge Distillation on Time Series Data

Thanks to Reviewer nLZW for all the comments.

Response to Weak-1 (Theoretical Foundation): We appreciate the suggestion to include a stronger theoretical foundation or analysis to further strengthen the contribution. While we acknowledge the value that theoretical insights and guarantees could bring to the paper, our primary focus in this work has been on the practical implementation and empirical validation of our method. Our goal was twofold: to tackle the prevalent issues of domain shift and model complexity in time series applications, and to demonstrate the superior effectiveness of our approach compared to existing benchmarks through extensive experiments on a range of real-world time series tasks. We believe that the empirical success of our method is also very crucial for research community, and thus, in this paper we emphasizes empirical results over theoretical analysis.

Nonetheless, we will definitely consider incorporating a theoretical foundation in future extensions of this research. Potential directions include: (1) Theoretical Justification for RL: include a theoretical analysis of using reinforcement learning to optimize sample selection, examining the exploration-exploitation trade-off and its impact on performance. (2) Uncertainty Consistency Analysis: explore the theoretical relationship between uncertainty consistency and model performance, supported by mathematical proofs and derivations.

Response to Weak-2 and Q-1 (Computational Complexity): Please kindly refer to Response to Major Weak-2 for Reviewer Ltvb for algorithm complexity analysis and Response to Weak-1-2 for Reviewer bzJ9 for model complexity analysis.

Response to Weak-3 (Hyperparameter Sensitivity): Please kindly refer to Response to Weak-3 for Reviewer E4vw for sensitivity analysis of hyper parameters.

Response to Weak-4 (Failure Case Discussion): Indeed, there are some cases where our approach does not perform as expected, such as $2\to7$ in HHAR and $0\to11$ in SSC. The poor adaptation performance in these scenarios may be attributed to the larger domain shift, as even the complex teacher model and other benchmarks struggle to achieve good adaptation. When there is a significant domain gap that the complex teacher model cannot effectively address, the target knowledge it provides may also be unreliable for the student. To explore this potential failure cause, we conducted further validation on another HAR dataset characterized by larger domain gaps (See Response to Q-4) and we are willing to incorporate a detailed discussion in the revised manuscript.

Response to Q-2 (Scalability to Larger Dataset): Please kindly refer to Response to Weak-2 for Reviewer E4vw for the analysis of our method on larger dataset.

Response to Q-3 (Teacher Sensitivity): As shown in Table 4 of our manuscript, we evaluate our framework with teachers pre-trained with different DA methods. Specifically, we utilize two discrepancy-based DA method (i.e., MDDA and SASA) and one adversarial-based DA method named CoDATS. The difference is that: for teachers pre-trained with discrepancy-based DA method, we have to train the domain discriminator. But for teachers pre-trained with adversarial-based DA methods, we can obtain the well-performed discriminator during teacher's training process. Thus, during student's training, the parameters of discriminator are frozen. From Table 4 of our manuscript, we can see that employing teachers from MDDA and SASA slightly underperforms CoDATS and DANN. The possible reason is that adding additional training steps for domain discriminator will inevitably increase training difficulty in terms of model convergence. But they still perform better than other benchmark methods. It indicates that our approach is not limited by the teacher pre-training strategies.

Response to Q-4 (Larger Domain shift): To evaluate our method under larger domain shift, we conducted some preliminary experiments on another dataset called HPP HAR [15]. This dataset includes 12 subjects, each performing 6 activities, with the smartphone placed in three different positions: pants, a shirt, and a backpack. The results are shown in the table below. As observed, almost all methods experience performance degradation when faced with a significant domain gap. A potential reason for this is that these KD-based cross-domain methods (including ours) are highly dependent on the teacher's performance. If the teacher fails to capture domain-invariant representations, the student model may also be negatively impacted. Due to time constraints, we were unable to explore other SOTA domain adaptation methods to enhance the teacher's performance and verify the above hypothesis. We plan to conduct this verification in future extensions of our research.

Response to Q-5 (Extent to Multi-Source): Some adjustments may be necessary if we extend our approach to multi-source domain adaptation (MSDA) scenarios. Firstly, instead of utilizing RL to select proper target samples, in MSDA we could leverage RL to dynamically select suitable source domains (or source samples), which are most relevant to target domain, to minimize the negative transfer as much as possible. Secondly, some of the key components in RL need to be re-defined. For instance, we need to re-formulate reward function to offer essential feedback to DDQN. A potential solution could be some functions to measure the data distribution discrepancy between source and target domain. Lastly, for MSDA, the teacher should be the expert who globally predicts well on the mixture of source domains. Methods like STEM [16] could be possible solutions for training a proper teacher in multi-source domain scenario.

Thanks to Reviewer E4vw for all the comments.

Response to Weak-1 (Feature Knowledge for Transferability Assessment): Although we have discussed above weakness as one of our limitations in manuscript, we conducted some preliminary experiments as suggested. We investigate several feature-based KD methods to assess sample transferability on SSC as below table shows. Specifically, we utilized L2 [7], attention maps (AT) [8], CORAL [9], and probabilistic knowledge transfer (PKT) [10] to measure the distance between teacher's and student's feature maps. These feature distances are then used to calculate reward $\mathcal{R}_3$ in Eq.(3). From the table, we can see that transferability assessments using feature distance obviously underperform logits-based method in our proposed framework. The potential reason is that: in our current framework, we utilize MCD module to generate $N$ teachers and then average their logits for calculating uncertainty and transferability. However, unlike logits, each data point in the feature maps carries different meaning depending on its spatial location within its feature space. Simply averaging feature maps across multiple teachers appears unreasonable and impractical, potentially resulting in poor performance. Therefore, the feature-based knowledge cannot be directly integrated into our existing framework. A more comprehensive designs need to be carefully considered.

Response to Weak-2 (Scalability to Larger Dataset): To verify the efficiency and scalability of our method on larger time series (TS) dataset, we conduct experiments on another Human Activity Recognition dataset named PAMAP2 [11]. Below table compares the dataset complexity of PAMAP2 with the datasets we employed in our manuscript. Note that it is meaningless to compare the total size of datasets in our settings as our experiment evaluate transfer scenario between single subject. We summarize the averaged number of samples, channels, data length and classes across all transfer scenarios for these datasets. We also report the time complexity of our method across these datasets (i.e., training time per epoch for single transfer scenario). From this table, we can see PAMAP2 is larger in terms of number of samples and more complex in terms of number of channels and classes. Compared with FD, the epoch training time for PAMAP2 only increases 2 times as the number of training samples increases about 4 times, indicating that our method scales well in terms of computational efficiency on larger TS dataset.

Meanwhile, we also conduct the performance comparison between our method and benchmarks on PAMAP2 with randomly selected 5 transfer scenarios. The experimental results are summarized as below table. We can see that our proposed method consistently outperform other benchmarks in terms of average Macro F1-score, even though it cannot achieve the best performance on some transfer scenarios. This observation indicates that the effectiveness of our proposed method can also be generalized to larger time series dataset.

Response to Weak-3 (Hyperparameter Sensitivity): Due to paper space constraints, our sensitivity analysis for hyperparameters $\alpha_1$, $\alpha_2$, $\lambda$, $N$ and $K$ were included in the Supplementary (see Fig. 2,3,4 and Table 5). Please refer to our submitted Supplementary for the details.

For hyperparameter $\tau$ which is the temperature to soften teacher's logits, we added additional analysis ranged from 1 to 16 as shown in below Table. We can see that higher value of temperature (e.g., $\tau = 16$) would over smooth teacher's logits, resulting in poor distillation performance. Generally, $\tau = 2$ or $\tau = 4$ is a good choice for our method.

Thanks to Reviewer bzJ9 for all the comments.

Response to Weak-1-1 (Simple Combination): We clarify that our solution is not a simple two-stage combination of DA and KD. Particularly, we optimize DA loss $L_{DC}$ and KD loss $L_{RKD}$ together, addressing domain shift and model complexity simultaneously. Moreover, we consider the following two solutions as simple two-stage combination. 1. KD$\to$DA: first distill the knowledge from teacher to student on source data, and then directly perform DA on student. 2. DA$\to$KD: first perform DA on the teacher and then conduct distillation. As shown in below table, these two solutions perform better than Source-Only model, but significantly worse than ours. The reason is that if performing KD on source data first, followed by DA, the student would become biased towards the source data during KD stage. And due to its compact network architecture, the student's adaptation performance would also be affected. If we perform DA first and then perform KD on target data, it would result in poor distillation efficiency in KD stage due to the lack of ground truth on target data. These issues are consistent with findings from previous research [12][13].

Response to Weakness-1-2 (Model Complexity): To demonstrate the deployment on edge device, we compared our 1D-CNN teacher and student from four perspectives as shown in below table. Here, we employ Raspberry Pi 3B+ as the edge device for deployment. We can see that the student reduces 15.46 times parameters, 16.98 times FLOPs and 13.54 times memory usage compared to its teacher. Besides, the inference of student is 21.67 times faster than teacher on the edge device. Meanwhile, in our manuscript we have already shown that the student trained with our method is able to achieve comparable performance as the teacher. This enables our compact student to potentially meet the real-time response and on-site deployment requirements for certain time series applications.

Response to Weak-2-1 (Why RL works): We argue that the primary reason why RL performs better is due to its inherent balance between exploitation and exploration. If we solely exploit the gained knowledge (i.e., directly using uncertainty or transferability for sample selection), the student is likely to become stuck at certain sub-optimal points (see comparison of $R_2$, $R_2^\dagger$, $R_3$ and $R_3^\dagger$ in Table 6 of our manuscript). The exploration in RL allows the agent to explore new possibility from the environment. In our implementation, we utilize the ‘NoisyFC’ whose weights and biases are perturbed by a parametric function of the noise to enhance the efficiency of exploration. Meanwhile, we also agree that RL does involve more computational costs than others, but the total training time is still acceptable, especially considering the performance improvement it could bring (see Response to Major Weak-2 for Reviewer bzJ9).

Response to Weak-2-2 (Claimed Issue and No Special Design for TS): We'd like to clarify that our method does not intend to improve teacher's reliability but to adaptively transfer its target knowledge with our RL-based framework. The unreliability of teacher's target knowledge is the underlying reason why existing approaches that simply integrate KD with UDA frameworks often experience unsatisfactory adaptation performance. Thus, we are highly motivated to utilize RL to select samples aligning with student’s capability to mitigate such unreliable knowledge. Besides, we also agree that there is no special architecture design for TS in our proposed framework. We choose to evaluate on TS as model complexity and domain shift issues are very common in TS. As it is general, we will explore to extend our method to other research areas like CV and NLP in the future.

Response to Weak-3 (Domain-invariant Knowledge): The domain discrimination loss originates from Reference [14], which is closely related to adversarial learning. The main idea is to pit two networks against each other. The student is expected to generate invariant representations for both domains and the discriminator is expected to fail to distinguish them. By minimizing this loss, the student would generate similar representations on target domain as the teacher on source domain. In other words, the domain-invariant knowledge would be transferred from teacher to student.

Besides, there are two reasons why we cannot directly transfer the domain-invariant knowledge from teacher to student. First, the compact student model has very limited capacity and cannot capture the same fine-grained patterns in the data as teacher. Coarsely aligning their feature maps, as done in KD-STDA and MLD-DA, would lead to sub-optimal performance on target domain. Secondly, instead of focusing on learning domain-invariant representations, our objective is to improve student's generalization on target domain via teacher's knowledge, which motivates us to adaptively transfer teacher’s target knowledge based on student’s capability. Our ablation study results on framework (see Table 5 of manuscript) also suggest that the performance improvement from domain-invariant loss is very marginal. Conversely, our proposed RKD loss can significantly improve student's generalization capability on target domain.

Response to Q-1 (Reward not clear): We will try to improve the clarity of our framework via providing more description in updated version (See Updates in global Author Rebuttal).

Response to Q-2 (Typos): We will improve the typos in updated version.

Thanks for the response. I suggest detailed discussions on RL and more analysis of learned policies and failure cases (such as related experimental analysis and visualization) be included in the paper. I have updated the rating.

Thanks to Reviewer Ltvb for all the comments.

Response to Major Weak-1 (Comparison with Active Learning): We fully agree that our method is closely related to AL, particularly in selecting critical samples with uncertainty. We are also willing to include the acknowledge of AL in our updated version. Meanwhile, we'd like to clarify that there are also some distinguished differences between our approach and uncertainty-based AL sample selection methods. Firstly, as noted by the reviewer, our method introduces a novel concept by promoting uncertainty to uncertainty consistency. We use sample uncertainty consistency to assess the student's learning capability, rather than directly filtering samples based on their uncertainty. Additionally, this only forms part of our reward function, as we also incorporate sample transferability to evaluate student's network. Secondly, unlike AL, which explicitly leverages uncertainty for sample selection, our approach employs a RL-based module to heuristically learn the optimal target sample selection policy. And our experimental results on reinforced sample selection ablation (Table 6) have demonstrated its superior capability of performance enhancement compared to directly selecting samples as done in AL. Lastly, the selected target samples with our method will not be labeled as AL. Instead, they will continue to be evaluated without labelling after student's state is updated.

Response to Major Weak-2 (Computational Complexity): We performed the time complexity analysis as suggested and the results are shown in below table. Specifically, we measure the training time for our proposed method and other benchmarks with a NVIDIA 2080Ti GPU. The reported results are measured with one epoch on single transfer scenario on FD dataset, which has the largest training samples (about 1,800 samples per transfer scenario) among evaluated datasets.

We can see that our method does require more training time compared to other benchmarks, reflecting its greater complexity. The primary computational costs arise from two factors. The first part is the generation of $K$ historical experiences at each step, which accounts for approximately 73% of total training time. This could be significantly reduced by using a smaller K. The second factor, which consumes about 21% of total training time, is the MCD module which conducts multiple inference processes for uncertainty estimation. This computational burden could be further decreased by adopting alternative uncertainty estimation methods. Nevertheless, although our training time is longer than other benchmarks, we argue that it is still within an acceptable range, especially considering the performance improvement it could bring.

Response to Minor Weak-1 (Literature Review): As suggested, we performed a literature review on uncertainty-based active learning and will include it in our updated version (See Updates in global Author Rebuttal).

Response to Minor Weak-2 (Typo): We would like to revise it in the final revision.

Response to Q-1 (Unreliability): As stated in UNI-KD, the authors employ a data-domain discriminator to estimate sample uncertainty, with inputs derived from feature extractor of compact student trained on both source and target domain (i.e., domain-shared). However, our experiments, which directly applied SOTA UDA methods to compact student (see Table 1 of our manuscript), demonstrated that the compact student cannot fully capture the fine-grained patterns in source and target domains as effectively as teacher. Due to its limited capacity, the student's adaptation performance is inferior, making the uncertainty estimated by UNI-KD unreliable. In contrast, our proposed method leverages more robust teacher to estimate uncertainty via MCD module. Our experimental results further demonstrate its effectiveness on enhancing student’s performance on target domain.

Response to Q-2 (Ablation with AL): We conduct the ablation study on three uncertainty-based active learning (AL) strategies, including least confidence (LC), sample margin (M) and sample entropy (H). The results are presented in below table. We take student trained with our framework using whole target samples as the baseline (i.e., without RL). 'LC' refers to leveraging student's confidence to directly select samples. 'LC Consist.' refers to using the consistency of teacher's and student's confidence for explicitly sample selection. 'LC Consist. +RL' refers to leveraging 'LC Consist.' as reward to learn optimal sample selection policy.

We can see that: firstly, almost all uncertainty-based AL strategies exhibit performance degradation compared to the baseline. This could be attributed to the unreliable uncertainty estimation from student's outputs, especially at early training stage. Additionally, among these strategies, entropy performs the best, likely because it considers the overall probability distribution which might partially address student's unreliable predictions issue. Secondly, utilizing uncertainty consistency instead of uncertainty alone could enhance performance in most settings, as incorporating teacher's knowledge through consistency provides a more reliable measure. Lastly, our RL module could further enhance student's performance via employing any of uncertainty consistency as the reward, indicating its effectiveness.

Thank you to the authors for the detailed response. All of my concerns have been properly addressed, and my questions have been answered. I am happy to raise my score to Strong Accept.