Bayesian Knowledge Distillation for Online Action Detection

Thanks for your detailed comments, here are our answers to your concerns:

1. Th knowledge distillation in PKD (Zhao et al. (2020)) aims at distilling the knowledge of offline model to an online model. Our knowledge distillation aims at distilling the distribution and mutual information of an online teacher model to an online student model and improving the efficiency. The overall mechanism is different. In addition, our student model can quantify the predictive uncertainty in a single forward pass, which is practical for real applications.
2. The teacher model is trained in a deterministic way. The Laplace approximation is performed as a post-processing. Code will be made publicly available. For the teacher model, we include the performance below:

The teacher model has bettern performance than the student model and SOTAs. But it is more computationally expensive, which does not satisfies the objective of our overall goal 3) We will update the paper with visual analysis. Demos will also be available on our project page 4) By increasing the number of self-attention layers, our proposed BKD can achieve 72.3% mAP (SOTA: 71.6 %) on THUMOS’14 with 26.9M parameters (SOTA: 94.6M). Will update the results and efficiency in the newer version.

Thanks for your insightful comments, here are our answers to your concerns:

1. The novelties of this paper mainly lie in: (1) we are the first work leveraging knowledge distillation to improve the inference efficiency of online action detection, which is a big concern for real applications. (2) The student model are trained to identify important features automatically based on the mutual information. (3) We build an evidential student network, which can quantify the predictive uncertainty by a single forward pass.

2. The concept of evidential deep learning is first introduced by [1] to quantify uncertainty for classification problems. By applying a Dirichlet distribution to class probabilities, they treated predictions of a neural network as subjective opinions and learned the function to collect the evidence leading to these opinions from data. The uncertainty estimation is guided by the theory of evidence. Subsequent research has aimed to enhance uncertainty quantification in the framework of [1], particularly by including out-of-distribution data or density models. These developments are comprehensively reviewed in the survey paper [2], which also inspired our use of the term “evidential deep learning”. For a more detailed discussion on this topic, please refer to [2]. We will further clarify evidential deep learning for quantifying classification uncertainty.
   [1] Sensoy, Murat, Lance Kaplan, and Melih Kandemir. "Evidential deep learning to quantify classification uncertainty." Advances in neural information processing systems 31 (2018).
   [2] Ulmer, Dennis, Christian Hardmeier, and Jes Frellsen. "Prior and posterior networks: A survey on evidential deep learning methods for uncertainty estimation." Transactions on Machine Learning Research (2023).

3. We agree the training of Bayesian neural network is expensive. Our proposed BKD performs Laplace approximation (LA) after training one deterministic teacher model. So the main computation gain lies in the LA process. Following the comparison in E2E-LOAD, we made a comparison of training cost with other methods below, the mAP is reported on THUMOS’14 with Kinetics-pretrained features : | Method | mAP (%) | GPU Mem | Time (min/epoch) | # of parameters | |----------|------|-------------------------------|--------|-----| | LSTR | 59.2 | 8x31.4 |1.5 | 105.9| | E2E-LOAD | 72.4 | 8x16.9 |9.6 | 53.5| | BKD (ours) | 71.3 | 2x24 |13.2 | 18.7|

In addition, the inference speed is the first priority of online action detection so the inference FPS is the most important factor. For the unfair comparison concern, the knowledge distillation is one of our contributions. If other methods also leverage knowledge distillation, the performance will drop as the student model usually performs worth than the teacher model.

4. Thanks for your careful reading and pointing out the writing errors, we will proofread the paper and fix all of them.

Question of using CE loss: Our work focuses on evidential deep learning that is designed specifically for classification uncertainty, as detailed in Section 3 of [2]. We use the Dirichlet distribution to efficiently capture uncertainty, allowing it to be determined in a single forward pass of the neural network. This is because Dirichlet distribution is conjugate to the target categorical distribution of y. Since traditional EDL's uncertainty quantification might not effectively distinguish between aleatoric and epistemic uncertainties, we address this through knowledge distillation to minimize the distance between the Dirichlet distribution and the corresponding distribution of the teacher model. By distilling both epistemic and aleatoric information from the Bayesian teacher model, we can employ Eq. 10 as outlined in Section 3.3 of [2] to separate these two types of uncertainty.

Thanks for you detailed and insightful comments, here are our answers to your concerns:

1. We did not introduce Bayesian knowledge distillation for the first time. It was initially introduced by Hinton et al. [1], who transferred knowledge from complex ensemble models to a single deterministic model. This idea was further developed by Korattikara et al. [2], who applied it to distill knowledge from Bayesian neural networks into deterministic networks. However, our work incorporates Bayesian knowledge distillation into online action recognition for the first time. While [1] and [2] focus only on distilling the prediction mean from a Bayesian model, our method also distills the uncertainty from the teacher model, focusing on the entire prediction distribution rather than just the mean.
2. Efficiency concern: We would like to clarify that our approach enhances inference efficiency, not necessarily learning efficiency. The student model, designed to be much smaller than the complex teacher model, ensures efficient inference for predictions. Additionally, the student model shows improved efficiency in uncertainty quantification. A key advantage of our student model is its ability to capture uncertainty, which is distilled from the teacher model during the knowledge distillation process. By employing evidential deep learning, the student model can calculate uncertainty with just a single forward pass, leveraging the Dirichlet distribution's conjugacy with the target categorical distribution of y. This contrasts with the teacher model, which is less efficient in both prediction and uncertainty estimation.
3. For the comparison with MAT (Wang et.al 2023), we only incorporate the results reported in the paper. We will reproduce their results and include them in the ablation study part.
4. For the performance concerns, our proposed BKD achieves very close performance compared to SOTAs on all benchmarks. In Table 2 of the paper, BKD achieves 169.6 FPS with 18.7M parameters, which is 46.3 FPS faster than the second fast TRN. Compared to MAT (72.6 FPS, 94.6M parameters), BKD has 0.3% lower performance drop but it has huge FPS improvement and parameters space saving.
   
   [1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).
   [2] Korattikara Balan, Anoop, et al. "Bayesian dark knowledge." Advances in neural information processing systems 28 (2015).

Thanks for your insightful comments and questions. Here are our answers to your concerns:

1. Weakness 1a: Based on our knowledge, we are the first work to adopt knowledge distillation to improve the efficiency of online action detection. (PKD [1] uses knowledge distillation to transfer the knowledge of an offline model to the online model, which is a different mechanism compared to our work). Traditional knowledge distillation methods typically involve single deterministic models for both teacher and student, aiming to align the student's predictions with those of the teacher. However, our approach employs Bayesian knowledge distillation. This technique not only distills the teacher model's predictions but also its associated uncertainty, focusing on the entire prediction distribution rather than just the mean. We leverage evidential deep learning for its efficacy in Bayesian knowledge distillation, particularly for uncertainty quantification efficiency. By transferring the teacher model's knowledge to a Dirichlet-based network, the student model attains not only accurate predictions but also efficient uncertainty estimation. The student model's uncertainty calculation requires just a single forward pass of the network, thanks to the Dirichlet distribution's conjugacy with the target categorical distribution of y. In contrast, the teacher model is less efficient in both prediction and uncertainty estimation.
   
   Weakness 1b: The student model learns from the teacher model. We aim to transfer as much as the knowledge to the student model, i.e. we aim to distill the distribution of the teacher model to student model. In addition, online action detection is used for many safety-critical appplications such as autonomous drving. Therefore, the uncertainty quantification is important for decision making which another motivation of modeling the uncertainty.
   
   Weakness 1c: In Table 7 of the paper, we used different types of uncertainties for abnormal action detection. Since the abnormal action in our experiment setting denotes the classes that are not used for training, the epistemic uncertainty gives the best detection performance, which demonstrates the correctness of our uncertainty quantification.

2. To validate that the proposed framework can leverage mutual information to select important features, we re-implemented the framework without the attention module. The other parts of the framework remain the same. The experiment results are shown as follows:

The results demonstrate the effectiveness of our proposed attention modeling with mutual information. \

For the effectiveness of predictive uncertainty quantification, the results in Table 7 of the paper show that the uncertainties can be used for detecting abnormal actions accurately.

3. For the insights of teacher-student architectures. We implemented different knowledge distillation methods with a same teacher model for comparison. We did grid searches to select the optimal hyper-parameters for fair comparisons. The experiment results are shown as follows: | Method | THUMOS'14-ANet | THUMOS'14-Kinetics | TVSeries-ANet | TVSeries-Kinetics | |----------|------|-------------------------------|--------|-----| | PKD [1] | - | 64.5 | - | 86.4 | | Feature-based [2] | 65.2 | 68.1 | 87.0 | 88.4 | | Output-based [3] | 66.1 | 68.3 | 87.4 | 88.5 | | BKD | 69.6 | 71.3 | 88.4 | 89.9|

The results shown our BKD obtains better performance than other knowledge distillation methods, which demonstrates the effectiveness of Bayesian distillation.

[1] Zhao, Peisen, et al. "Privileged knowledge distillation for online action detection." arXiv preprint arXiv:2011.09158 (2020).
[2] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).
[3] Romero, Adriana, et al. "Fitnets: Hints for thin deep nets." arXiv preprint arXiv:1412.6550 (2014).