Knowledge Distillation via Flow Matching

Thank you for your valuable comments, we exert our greatest effort to address your comments.

Q1. My major concern is the generalization, given that distillation is a well-defined topic, but this method seems like doesn't work well in heterogeneous architecture settings. Especially when there is a big difference in size between teacher models and student models.

A1. Thank you for your valuable suggestions.

In our experiments, the heterogeneous teacher-student pairs, ResNet50-MobileNetV1 on ImageNet-1k and VGG13-MobileNetV2 on CIFAR-100, did not achieve SOTA accuracy. This outcome can be attributed to suboptimal configurations of the loss function and meta-encoder for FM-KD, rather than suggesting that FM-KD is ineffective in heterogeneous architecture settings.

We experiment with the optimal settings, and the additional experimental results are presented in Table I and II. We can observe that FM-KD beats all previous methods and achieves SOTA performance.

Table I (ImageNet-1k, loss function is set as DIST, K=8)

Table II (CIFAR-100, meta-encoder is set as Swin-Transformer, loss function is set as DKD)

Q2. Can you explain or verify why this method can't work well when the gap is big between teacher and student?

A2. Thank you for your interesting question. FM-KD is effective in your mentioned scenarios. Specifically, as shown in Tables I and II, FM-KD is effective in heterogeneous architecture settings with a big gap between the teacher and the student. In scenarios where a bigger gap exists between the teacher and the student, we conduct experiments using TIMM's ResNet50 as the teacher and ResNet18 as the student. The experimental results in Table III (refer to Appendix G) indicate that FM-KD remains effective.

Table III (ImageNet-1k, epoch is set as 350, teacher is set as ResNet 50 from TIMM, student is set as ResNet18)

Q10. How about the extra cost to the inference with the trained student model? As the architecture of the trained student model is no longer same to the baseline, which violates the basic purpose of knowledge distillation, namely model compression.

A10. We address this review in two Parts:

Part 1: We have presented GPU latency measurements during inference in Table VI. We can substantiate that the additional computational cost of FM-KD in inference is much lower compared with DiffKD.

Table VI (ImageNet-1k, hardware setting is 2*RTX 3080ti)

Part 2: To ensure a fair comparison with other knowledge distillation algorithms such as DKD, DIST, and SPKD, we have further designed FM-KD$^{\Theta}$, which refines the process by distilling $Z_0$ in FM-KD into an existing classification head (i.e. the original student's classification head) $\mathcal{T}_\mathrm{vanilla}(\cdot)$, thereby ensuring that no extra inference cost is incurred. More details can be found in Sec. 3.5 of the revised paper.

Table VII (CIFAR-100)

Table VIII (ImageNet-1k)

The experimental results of FM-KD$^{\Theta}$ are presented in Tables VII and VIII, where the implementation details on ImageNet-1k and CIFAR-100 can be found in Appendix I of our revised paper. By comparing FM-KD$^{\Theta}$ with prior knowledge distillation algorithms like DKD, DIST, and SPKD in Tables 1 and 2 of the revised paper, we can conclude that FM-KD$^{\Theta}$ achieve SOTA performance across a wide range of teacher-student pairs.

Q11. There is no discussion on related works in the main paper. I noticed that the authors put this part in the Appendix, but this is not proper, to the best of my understanding.

A11. Thank you for pointing out this presenting problem in our paper. We have moved Related work to Sec. 5 in the main paper.

References

[1*]. Score-Based Generative Modeling through Stochastic Differential Equations, ICLR 2021.

[2*]. Minimizing Trajectory Curvature of ODE-based Generative Models, ICML 2023.

[3*]. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, ICLR 2023.

[4*]. Flow Matching for Generative Modeling, ICLR 2023.

Q4. Furthermore, the underlying reason for why applying diffusion process to augment feature representation matching in knowledge distillation in not clear enough. What is the reasonable new technical insight here?

A4. It may be helpful to clarify that our method does not employ a diffusion process, typically associated with SDE forward form. Instead, our approach is grounded in Rectified flow, which belongs to ODE forward form. The motivation for applying Rectified flow to augment feature representation matching (i.e. knowledge transfer) is as follows

1. Rectified flow can establish a bridge between two empirical distributions. This characteristic makes the Rectified flow particularly suitable for knowledge distillation in transferring knowledge from the teacher to the student, as both teacher and student features/logits are sampled from empirical distributions.

2. It is difficult to accomplish the knowledge transfer between two complex empirical distributions by one step. Thus, adopting the multi-step sampling from Rectified flow can realize the progressive transformation (i.e. probability path) from the student feature/logit distribution to the teacher feature/logit distribution, which leads to effective knowledge transfer.

3. As proven in Proposition 3.2 of our original/revised paper, FM-KD is implemented through Rectified flow and can be approximated as an implicit ensemble method, which is why FM-KD can achieve more precise knowledge transfer. Unfortunately, other noise schedules, such as VP ODE [*1,*3,*4] and VE ODE [*1,*3,*4], cannot achieve this goal.

4. FM-KD has an property that can be used to realize the trade-off between performance and efficiency by changing NFE in inference.

Q5. In addition, Rectified Flow assumes straight paths between any two steps during the diffusion, so is it reasonable when applying this design to feature representation matching in knowledge distillation? The original purpose of Rectified Flow is for faster diffusion but not more accurate performance.

A5. Thank you for your insightful question, our response consists of three points.

1. In our theoretical derivation, Rectified flow can be approximated an implicit ensemble method for effective knowledge transfer, other noise schedules, such as VP SDE [1*,3*,4*] and VE SDE [1*,3*,4*] cannot. This contrasts with other noise schedules like VP SDE and VE SDE, which may not be as effective due to the varying influence of truncation errors in non-straight noise schedules on the final result $Z_0$ from the Euler method.

2. Rectified flow can accelerate sampling (equivalent to minimizing the curvature) [2*] making it possible to be effective with very small NFE scenarios. Consequently, the fewer the NFEs, the smaller the training and inference computational burden, so we apply Rectified flow for feature representation matching.

3. Rectified flow is not the only implementation of FM-KD. The assumption that the trajectory between two empirical distributions is straight is merely one modeling approach in optimal transport theory. It's viable, but not the only one. We unify VP ODE (w.r.t. VP SDE), VE ODE (w.r.t. VE SDE) and Rectified flow in FM-KD and present this in Appendix M of the revised paper. Interestingly, we discover that Rectified flow performed best, perhaps because Rectified flow can be approximated an implicit ensemble method.

Q6. From Table 2 on ImageNet-1K, the proposed method FM-KD merely performs on par or worse than DiffKD.

A6. The experimental results (i.e. Table 2 of the original paper) for ImageNet-1k are not optimal for FM-KD. The results in Tables 1 and 2 of the original main paper are presented only to ensure consistency of the loss function (i.e. DIST) and the meta-encoder (i.e. Swin-Transformer). As illustrated in Fig. 5 of the revised paper (Table 6 of the original paper) that FM-KD performs best on ImageNet-1k when using MLP as its architecture. Therefore, we are pleased to provide the best performance of FM-KD on ImageNet-1k by using MLP as meta-encoder:

Table III (ImageNet-1k)

More details of Table III can be found in Appendix K of the revised paper.

Q7. The authors did not discuss the limitations of the proposed method.

A7. Sorry that our oversight caused you concern, we have added limitation to Sec. 7 of the revised paper.

Q2. Specifically, the authors directly use Rectified Flow to replace vanilla diffusion in DiffKD, making the diffusion of student representation (either feature or logits) to have straight paths between any two steps. In general, FM-KD is rather incremental.

A2. Thank you for your insightful comment. In our work, we do not employ Rectified flow in its original form for knowledge distillation. There are significant modifications made from Eq. 2 (the original Rectified flow loss function) to Eq. 3 (our proposed serial loss function) in our paper. These enhancements are crucial. Without them, using the original Rectified flow training paradigm (as per Eq. 2) with the WRN-40-2-WRN-16-2 pair on CIFAR-100 would likely have led to the issue of vanishing gradients, resulting in a final test accuracy 3.42%.

Additionally, it may be pertinent to highlight that FM-KD is not a mere incremental improvement. FM-KD represents a substantial deviation from existing training losses, such as those used in DiffKD. The training paradigm (i.e. Eq. 3) we introduced is not a mere extension but a distinct form. Unlike DiffKD, which relies on the original diffusion model training paradigm for training, FM-KD employs a uniquely formulated training paradigm, guaranteed with our proposed Theorem 3.1, to facilitate effective knowledge transfer. This design is quite critical. In our experiments, the original Rectified flow alone demonstrated insufficient results (e.g., achieving only 3.42% with the WRN-40-2-WRN-16-2 pair on CIFAR-100), but it can go up to 77.84% with our proposed unique serial training paradigm.

Q3. The authors try to claim FM-KD as a totally new knowledge transfer framework, which is applicable to different types of teacher-student architectures, different metric-based distillation methods and different representations. However, it is mostly misleading, as DiffKD has already attained this goal when putting it in the context of the authors' viewpoint.

A3. Although DiffKD has shown some progress towards being applicable across various teacher-student architectures and metric-based distillation methods, it has not effectively or adequately achieved it to the expected extent. We illustrate this in three Parts:

Part 1: FM-KD is less sensitive to architecture. FM-KD exhibits good performance using various architectures as meta-encoder, as shown in our experiments with the ResNet50-MobileNetV2 pair (refer to Table 6 in the original paper or Fig. 5 in the revised paper). In contrast, DiffKD's performance appears more sensitive to architectural variations, experiencing training challenges such as loss becoming NAN under different meta-encoder configurations.

Part 2: FM-KD has evaluated this goal experimentally. We have conducted comprehensive ablation studies to demonstrate that FM-KD exhibits generalization ability against variations in architectures and loss functions. On the other hand, it is observed that DiffKD, in its original paper, does not focus extensively on similar ablation studies.

Part 3: FM-KD can perform better. FM-KD achieves better performance than DiffKD in both ResNet34-ResNet18 and ResNet50-MobileNetV1 pairs on ImageNet-1k with lower training and inference costs:

Table I (ImageNet-1k)

More details of Table I can be found in Appendix K of the revised paper.

Table II (ImageNet-1k)

Where x-MLP refers to x-layer MLP. The measurements in Table II were obtained under FM-KD using 2-MLP as meta-encoder. More details can be found in Appendix J of the revised paper.

Thank you for the thorough and constructive comments and suggestions. We think it may be helpful to clarify that probability flow and diffusion processes as generative models are of two different forms. Our research is specifically centered on Rectified flow, a component of probability flow. This flow-based generative model is built on the ODE forward form framework, setting it apart from diffusion processes that are commonly associated with the SDE forward form. A particularly noteworthy aspect of Rectified flow is its capacity to bridge two empirical distributions, a feature that diffusion processes are unable to achieve.

Furthermore, we have carefully incorporated your comments in the revised paper. Please see our below responses to your questions and concerns one by one.

Q1. This paper attempts to improve feature representation matching from knowledge distillation. The proposed method FM-KD heavily relies on two existing works DiffKD (arxiv 2023) and Rectified Flow (ICLR 2023).

A1. Thank you for your valuable feedback. FM-KD draws some inspiration from Rectified flow, yet it is distinct from DiffKD. The primary motivation behind developing FM-KD was to tackle potential limitations we identified in the modeling approach used in DiffKD. Unlike DiffKD, which incorporates three extra empirical losses, FM-KD employs just a single additional loss from theoretical derivation and performs better than DiffKD in both effectiveness and efficiency (please see Tables 1 and 2 of the revised paper).

The main differences between FM-KD and DiffKD consist of three Parts:

Part 1: The loss in FM-KD is different from the form of each loss in DiffKD. The losses in DiffKD are redundant and stacked. DiffKD employs the original training loss of the diffusion model (i.e., $L_{diff}$) and combines it with two additional auxiliary losses (i.e., $L_{ae}$ and $L_{diffkd}$) as detailed in its original paper for training. By contrast, we utilize our proposed training loss $L_{FM-KD}$ (solely one), guaranteed by Theorem 3.1 in the original/revised paper, which differs from each of $L_{diff}$, $L_{ae}$, and $L_{diffkd}$. To the best of our knowledge, this serial training paradigm has not been previously introduced in existing work.

Part 2: FM-KD uses serial training paradigm while DiffKD uses original diffusion model training paradigm. We believe the importance of our proposed serial training paradigm lies in its ability to effectively and stably propagate gradients back to the earlier layers of the student. By contrast, DiffKD might face challenges in functioning solely on the original diffusion model training paradigm, as it may struggle with preventing gradient vanishing. Though DiffKD aligns the sampling result $Z_0$ with the teacher feature/logit $X_T$ (i.e., $L_{diffkd}$ in DiffKD's original paper), ensuring backward gradient propagation to the student's earlier layers, DiffKD is highly sensitive to large Number of Function Evaluation (NFE) and the meta-encoder architecture in our experiments. We are pleased to note that FM-KD seems to avoid this particular issue.

Part 3: DiffKD=$X^S$->Gaussian noise->$X^T$ and FM-KD=$X^S$->$X^T$. DiffKD does not immediately effectuate the transition from $X^S$ to $X^T$. Instead, it first transforms $X^S$ into a Gaussian noise and subsequently translates this noise to $X^T$. This dual-stage transformation process might potentially be overly complex. By contrast, FM-KD offers a more straightforward approach by directly forming a bridge between the distributions of student and teacher features/logits.

We would like to note that the performance of DiffKD, which may be attributed to its heuristically design, could be perceived as not fully meeting the expected standards. For example, with the ResNet50-MobileNetV1 pair on ImageNet-1k, DiffKD obtains Top-1 Val Acc. 73.78% by increasing the GPU latency from 0.0108 to 0.0517 during inference, while FM-KD obtains Top-1 Val Acc. 74.22% by increasing the GPU latency from 0.0108 to 0.0151 during inference in Appendix J & K. Thus, FM-KD is more effective than DiffKD.

Q8. As the proposed method is closely related to DiffKD, diffusion with Rectified Flow (straight paths) vs. diffusion with vanilla diffusion (non-straight paths), DiffKD and DiffKD+RectifiedFlow should be always used as the baselines instead of others.

A8. Thanks for this suggestion, and here we provide more experiments and discussions.

For diffusion with vanilla diffusion (non-straight paths), we clarify that it is not feasible to compare FM-KD with vanilla diffusion processes since vanilla diffusion processes simply do not work in knowledge transfer. Specifically, in our exploratory studies, we try most of the ODE form of popular diffusion processes (non-straight paths), including VP ODE (converted from VP SDE and DDPM) and VE ODE (converted from VE SDE and NCSN). We also try vanilla Rectified flow (straight path). However, if these methods use the vanilla training paradigm:

Score function matching in VP ODE and VE ODE:

$argmin_{v_\theta} E_{(Z_1,Z_0,t)} ||s_{v_\theta}(Z_t,t) - \nabla_{Z_t} \log p(Z_t|Z_0)||_2^2.$

Velocity matching in Rectified flow:

$argmin_{v_\theta} E_{(Z_1,Z_0,t)}||g_{v_\theta}(Z_t,t) - \frac{\partial Z_t}{\partial t}||_2^2.$

In our experiments, their final test accuracy will not exceed 5% with WRN-40-2-WRN-16-2 pair on CIFAR-100 (suffer from vanishing gradient). After we introduce the training paradigm in Eq. 3, as guaranteed by Theorem 3.1, Rectified flow can finally work effectively. Thus, the central focus of our paper is the training paradigm presented in Eq. 3. More importantly, we unify VP ODE, VE ODE, and Rectified flow within FM-KD through Theorem 3.1 in Appendix M, and we experimental demonstrate the effective modeling of both VP ODE and VE ODE. However, Rectified flow outperforms VP ODE and VE ODE in terms of performance and stability.

For DiffKD, we have presented additional comparative experimental results in Table III and substantiated that FM-KD is more effective than DiffKD. Furthermore, we conduct DiffKD to use the meta-encoder architecture (i.e. 2-MLP) from FM-KD, and FM-KD to use the meta-encoder architecture (2-Bottleneck+Conv+BN+MLP) from DiffKD, with the results presented in Table IV. The final results indicate that "DiffKD uses FM-KD's meta-encoder" underwent training collapse at epoch 1, whereas FM-KD (i.e. 74.26%) significantly surpassed DiffKD (i.e. 73.78%) in test accuracy when using the same meta-encoder architecture as DiffKD.

Table IV (with ResNet50-MobileNetV1 pair on ImageNet-1k)

For DiffKD+RectifiedFlow, we implemented subtle adjustments to the parameters $\alpha_t$ and $\sigma_t$ in $p(Z_t|Z_0) = \mathcal{N}(Z_t | \alpha_tZ_0,\sigma_t^2I)$, setting them to $1-t$ and $t$, respectively. These modifications were accompanied by a learning rate warm-up strategy specifically for DiffKD+Rectified flow applied to CIFAR-100. The results of these experiments are presented in Table V. From these results, it appears that FM-KD tends to be more effective in comparison to DiffKD+Rectified flow.

Table V (CIFAR-100)

Q9. From Table 2 on ImageNet-1K, the proposed method FM-KD merely performs on par or worse than DiffKD. This raises a critical problem, is it necessary to replace vanilla diffusion (non-straight paths, e.g. DDIM) by Rectified Flow (straight paths)? Why?

A9. We address your concerns in the following two points.

1. The experimental results of FM-KD (i.e. Table 2 of the original paper) on ImageNet-1k are not obtained from optimal settings. We conduct additional experiments and the additional experimental results in Table III demonstrate that FM-KD substantially outperforms DiffKD.

2. DDIM cannot be used to replace Rectified flow since DDIM is merely an ODE solver and is not related to whether 'the path is straight or not'. We think you are referring to different ODE processes with non-straight paths, such as VP ODE (converted from VP SDE and DDPM) and VE ODE (converted from VE SDE and NCSN). In Appendix M of our revised paper, we unify VP ODE, VE ODE, and Rectified flow into the FM-KD framework for efficient training. The experimental results in Appendix M demonstrate that training with Rectified flow is more effective and stable than using VP ODE and VE ODE in the WRN-40-2-WRN-16-2 pair on CIFAR-100.

Q2. The FM-KD method proposed in this paper modifies the structure of the student network and increases the cost during model inference, which is a shortcoming of a model compression algorithm.

A2. Thank you for your valuable feedback. We address your concerns in the following three Parts.

Part 1: FM-KD leads to significant performance improvements, and the added computational overhead in inference is acceptable. Additionally, FM-KD has the ability to adjust the Number of Function Evaluations (NFE) during inference to balance effectiveness and efficiency.

Part 2: As demonstrated in Tables I and II, in comparison to DiffKD, a method that also elevates computational cost in inference, FM-KD is evidently more cost-effective regarding performance and GPU latency.

Table I (ImageNet-1k)

More details of Table I can be found in Appendix K of the revised paper.

Table II (ImageNet-1k)

Where x-MLP refers to x-layer MLP. The measurements in Table II were obtained under FM-KD using 2-MLP as meta-encoder. More details can be found in Appendix J of the revised paper.

Part 3: We appreciate the opportunity to enhance our work and have designed FM-KD$^{\Theta}$ without extra computational costs in inference. For a comprehensive explanation, kindly refer to our detailed response regarding your Q4.

Q3. The computational cost during inference is not given in the paper.

A3. We have presented GPU latency measurements during inference in Table III. We can substantiate that the additional computational cost of FM-KD in inference is much lower compared with DiffKD.

Table III (ImageNet-1k, hardware setting is 2*RTX 3080ti)

Q4. If refer to Fig. 5, the increase in cost is obvious and there is some unfairness in comparison with other methods.

A4. Thank you for your valuable comment.

To ensure a fair comparison with other knowledge distillation algorithms such as DKD, DIST, and SPKD, we have further designed FM-KD$^{\Theta}$, which refines the process by distilling $Z_0$ obtained from FM-KD into an existing classification head (i.e. the original student's classification head) $\mathcal{T}_\mathrm{vanilla}(\cdot)$, thereby ensuring that no extra inference cost is incurred. More details can be found in Sec. 3.5 of the revised paper.

Table IV (CIFAR-100)

Table V (ImageNet-1k)

The experimental results of FM-KD$^{\Theta}$ are presented in Tables IV and V, where the implementation details on ImageNet-1k and CIFAR-100 can be found in Appendix I. By comparing the accuracy of prior knowledge distillation algorithms like DKD, DIST, and SPKD in Tables 1 and 2 of the revised paper, we can conclude that FM-KD$^{\Theta}$ achieve SOTA performance across a wide range of teacher-student pairs.

Q5. Are there still significant advantages of the FM-KD approach over baseline models that increase the number of parameters to make time-consuming approximations?

A5. Thank you for your interesting question.

Yes, DiffKD is an algorithm that increases the computational cost during inference. As indicated in Tables I and II, by utilizing a lightweight meta-encoder 2-MLP, FM-KD exceeds DiffKD, which uses 2-BottleNeck+Conv+BN+MLP as meta-encoder, by margins of 0.68% and 0.44% in the ResNet34-ResNet18 and ResNet50-MobileNetV2 pairs with a lower GPU latency (less than 1/3 DiffKD's in the ResNet50-MobileNetV2 pair), respectively.

Thank you for reviewing our paper. Below, we provide responses to your comments and queries.

Q1. This paper is similar to DiffKD where it is a combination of generative model and knowledge distillation, except that it replaces the diffusion model with the rectified flow, which has limited innovation.

A1. Thank you for your valuable feedback. Our approach does not merely substitute the diffusion model with Rectified Flow or directly apply Rectified Flow for knowledge transfer. Specifically, DiffKD employs the original training loss of the diffusion model (i.e., $L_{diff}$) and combines it with two additional auxiliary losses (i.e., $L_{ae}$ and $L_{diffkd}$) as detailed in its original paper for training. By contrast, we utilize our proposed training loss $L_{FM-KD}$ (solely one), guaranteed by Theorem 3.1 in the original/revised paper, which differs from each of $L_{diff}$, $L_{ae}$, and $L_{diffkd}$. Moreover, from Eq. 2 (i.e. original Rectified flow loss function) to Eq. 3 (i.e. our proposed serial loss function) in our original/revised paper we implement some critical improvements. Without these improvements, Rectified flow (w.r.t. Eq. 2) in the WRN-40-2-WRN-16-2 pair on CIFAR-100 would have encountered the problem of vanishing gradients, resulting in a final test accuracy 3.42%.

It might be helpful to provide a clear outline of our contributions to ensure a better understanding.

1. We introduce Rectified flow instead of diffusion processes since Rectified flow is able to establish a bridge between two empirical distributions (teacher feature/logit distribution and student feature/logit distribution) and is well suited for knowledge transfer.

2. The serial training paradigm in Eq. 3 is essential for effective and stable training. If trained directly in the form of the original Rectified flow, VP SDE and VE SDE, the student accuracy with WRN-40-2-WRN-16-2 pair on CIFAR-100 $\leq$ 5%. However, as demonstrated in Appendix M, the accuracy $\geq$ 72% if the FM-KD framework is used for training.

3. FM-KD is theoretically guaranteed and has been demonstrated to stable training with various meta-encoder and loss functions by a wide range of ablation experiments. By contrast, DiffKD is an empirical design with no theoretical guarantees. DiffKD prevents vanishing gradient through $L_{diffkd}$ outlined in its paper, but this form leads to unstable training.

4. We prove and explain the effectiveness of this training paradigm as it can approximate an implicit ensemble method in Proposition 3.2.

5. We design OFM-KD, which can be used in the Online KD scenario. Our proposed OFM-KD outperforms prior Online KD algorithms in most teacher-student pairs.

6. FM-KD is more effective than DiffKD on CIFAR-100 and ImageNet-1k, as shown in Table 1 & 2 and Appendix K & L of the revised paper. The experimental results in the original main paper where FM-KD did not reach SOTA performance were merely due to not utilizing the optimal settings.

7. FM-KD is more efficient than DiffKD, as demonstrated in Appendix J.

8. We design FM-KD$^\Theta$ in Sec. 3.5 of the revised paper in this discussion phase, which incurs no additional inference computational cost.

9. We unify VP ODE [1*,2*,3*], VE ODE [1*,2*,3*], and Rectified flow into the FM-KD framework in Appendix M in this discussion phase.

Q6. The experimental performance is not outstanding enough. There is no significant advantage over other knowledge distillation methods in Table 3 of the experiments for object detection.

A6. Thank you for your feedback. We apologize for our inability to resolve this weakness in object detection. This result stems from the fact that in image classification, flow matching with the teacher at the logit level generally yields performance closer to that of the teacher. However, in object detection, flow matching with the teacher at the FPN (Feature Pyramid Network) level does not directly translate into improved performance in the final metric, mAP. We have already incorporated this point into our limitations (please see Sec. 7).

Q7. There are also several tests in the image classification experiments that do not perform as well as other existing methods.

A7. Thank you for your valuable comment. The experimental results of FM-KD in Tables 1 and 2 of the original paper are not obtained from optimal settings. It is demonstrated from Fig. 5 of the revised paper (Table 6 of the original paper) that FM-KD performs best on ImageNet-1k when using MLP as its architecture. Moreover, FM-KD can achieve better performance by using DKD as a loss function on CIFAR-100.

We are pleased to experiment with the optimal settings, and the additional experimental results are presented in Tables VI and VII. It can be observed that FM-KD outperforms all prior methods and achieves SOTA performance.

Table VI (ImageNet-1k, loss function is set as DIST, K=8)

Table VII (CIFAR-100, meta-encoder is set as Swin-Transformer, loss function is set as DKD)

References

[1*]. Score-Based Generative Modeling through Stochastic Differential Equations, ICLR 2021.

[2*]. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, ICLR 2023.

[3*]. Flow Matching for Generative Modeling, ICLR 2023.