Thank you for your constructive review and for expressing strong interest in our work. We are encouraged by your willingness to engage further in discussion and truly appreciate your thoughtful comments on the core contributions of our paper.

Weakness

1.

As explained in the Introduction, the predictive distribution is obtained by $\hat{p}(y|\mathbf{x})=\sum_{i=1}^n p(y|\mathbf{x},\theta_i)/n,$ where $\theta_1,\ldots,\theta_n$ are ensemble members. For DLF, as shown in Figure 1 (“Inference on new data”), we can generate $\theta_1,\ldots,\theta_n$ by sampling $n$ independent latent vectors $\mathbf{z}_1,\ldots,\mathbf{z}_n$ from $\mathcal N_q(\mathbf 0,\mathbf I_q)$, forwarding each sample through the network to have ensemble members. All of the uncertainty measures are calculated based on $\hat{p}(y|\mathbf{x}).$

As the referee suggested, we evaluated DLF and Deep Deterministic Uncertainty (DDU) under the same OOD-detection protocol. CIFAR-10 serves as the in-distribution dataset, while SVHN, CIFAR-100, and Tiny-ImageNet are treated as OOD. AUROC is the evaluation metric, and all tests use a Wide-ResNet-16-1 backbone (≈ 0.17 M parameters).

In (Mukhoti et al,. 2023), the authors report results with a much larger backbone (Wide-ResNet-28-10, $\geq$ 214 $\times$ Wide-ResNet-16-1). After downsizing DDU to match our architecture, DLF achieves higher AUROC on all three OOD datasets. These findings demonstrate that DLF achieves well uncertainty estimation performance relative to its compact model size, which aligns well with the objective of knowledge distillation.

We evaluated OOD detection performance using AUROC as our primary metric. Although LBE achieves slightly higher AUROC scores on CIFAR-100 and Tiny-ImageNet, its predictive probability densities assign nearly many in-distribution samples a probability very close to 1.0. This behavior indicates that the model is essentially overconfident rather than genuinely better at quantifying uncertainty.

2.

In the original manuscript, we did not include Dirichlet distillation (Proxy-EnD²) in the experiments since the papers for Hydra and BE reported that their algorithms are superior to Dirichlet distillation. We focused mainly on comparing these two deterministic distillation methods with our DLF.

In Table below, we add Dirichlet distillation. In addition, we add a new distribution distillation method called Ensemble Distillation via Flow Matching (EDFM; Park et al., 2025), which we found after we submitted our paper. Note that DLF outperforms Proxy-EnD² and EDFM with large margins in terms of the accuracy and NLL while DLF is not much worse than Proxy-EnD² in terms of ECE.

We want to emphasize that superior performance of DLF compared to other distributional distillations is not achieved by chance. Instead, there are various advantages of DLF compared to Proxy-EnD² and EDFM. We think that there are three properties thay any desirable models for distribution distillation are expected to have: (1) large support, (2) easy and stable learning algorithm and (3) flexibility to be applied to downstream tasks. Below, we explain why DLF is desirable in view of these three properties.

• DLF is a Gaussian process, which is well known to cover large class of distributions (Garriga-Alonso et al., 2019; Neal et al., 1996; Calandra et al., 2016; Huang et al., 2015). Even if teacher ensembles are not Gaussian, a Gaussian process can approximate them closely.
• Existence of easy and stable learning algorithms is a critical fact when choosing an algorithm for distributional distillation. Learning the DLF model is numerically easy and stable since the efficient EM algorithm is available. It is well known that estimating the parameters in the Dirichlet distribution is difficult (Minka et al., 2000; Chai et al., 2013) and thus it would not be easy to learn the Dirichlet distillation model which would be a partial reason for inferior performance. For Flow matching distillation, pretrained features should be fed when learning. That is, end-to-end learning is not possible. In contrast, DLF is learned in an end-to-end fashion (i.e. feature vectors and distribution are learned simultaneously).
• Flexibility is also an important aspect. As we have shown in Section 5.3, the learned features by DLF are superior than those by a single DNN or Hydra for downstream tasks.
• To sum up, the idea of estimating the distribution for teacher ensembles is not new but efficient implementation is not an easy task and DLF is such a method.

3.

Thank you for this valuable suggestion. We categorize computational cost into two aspects: the cost incurred during training steps and that during inference. Since inference cost is roughly proportional to the number of parameters, we have reported each model's parameter count. Moreover, during training, the E-step expectation can be obtained in closed form; the formula is given in Appendix B of the paper, so training time does not increase significantly. In the revised manuscript, we will include a discussion of these computational cost. Specifically, we will report (1) Multiply–Accumulate Operations (MAC, measured in MMac), (2) inference latency (ms), and (3) training time (hours).

Preliminary measurements on CIFAR-10/100 are provided below.

4.

Thank you for your insightful suggestion. To verify that our observations generalize to more complex classification tasks, we have conducted an ablation study of the latent factor dimension on the CIFAR-10 and CIFAR-100 datasets. In the revised manuscript, we will include the corresponding figures, and we have also updated the main result tables to report the applied latent factor dimensions where applicable.

Question

1.

We also think that ensemble members trained on the same data with different initializations are not strictly independent. The goal of distribution distillation is to find a proxy distribution that approximates the distribution of teacher ensembles, and the proxy distribution assumes that teacher ensembles are independent. The validity of this assumption in the proxy distribution is justified by the performance of the corresponding predictive distribution.

Note that most methods for distribution distillation assume that teacher ensembles are independent. Moreover, Dirichlet distillation as well as Ensemble Distillation via Flow Matching (EDFM; Park et al. 2025) assume further that prediction values from a same teacher model at different inputs are independent. In contrast, we assume that each teacher model is a Gaussian process and thus the prediction values from a same teacher model are dependent.

That is, DLF estimates the joint distribution of teacher ensembles while Dirichlet distillation and EDFM estimate the marginal distribution only. Estimation of the joint distribution would be useful when we have to predict multiple inputs simultaneously.

For example, consider a problem to decide whether two randomly given images belong to a same class or not. Let $\mathbf{x}_1$ and $\mathbf{x}_2$ be two given images and let $Y_1$ and $Y_2$ be the corresponding class labels. Then, the goal is to estimate $p(Y_1=Y_2|\mathbf{x}_1,\mathbf{x}_2)$.

DLF estimates this predictive probability by $\int_f \sum_{c=1}^C p(y_1=c|\mathbf{x}_1,f)p(y_2=c|\mathbf{x}_2,f) \pi(f)df$ while Dirichlet distillation and EDFM estimate it by $\sum^C_c \int_f p(y_1=c|\mathbf{x}_1,f) \pi(f)df \int_f p(y_2=c|\mathbf{x}_2,f) \pi(f)df$. These two estimates are quite different and the former is expected to perform better.

To justify this conjecture, we investigate the performance of these two estimates on randomly selected pairs of CIFAR-10 test images. The results are as follows:

(AUROC and AUPRC) values are (0.967, 0.816) for Dirichlet distribution and (0.985, 0.90) for DLF while the ECE values are similar (0.11). Moreover, AUROC and AUPRC values of DLF without considering the dependency between $Y_1$ and $Y_2$ (i.e. using $\sum_{c=1}^C \int_f p(y_1=c|\mathbf{x}_1,f)\pi(f)df \int_f p(y_2=c|\mathbf{x}_2,f) \pi(f)df$ where $\pi(f)$ is estimated by DLF) are (0.938, 0.781), which are much lower than those of DLF when considering the dependency. These results amply support that considering dependency of prediction values from the same teacher model would be beneficial and DLF is such a model.

2.

Please refer to our response to Weakness 1.

Reference

• Mukhoti, J., Kirsch, A., Van Amersfoort, J., Torr, P. H., & Gal, Y. (2023). Deep deterministic uncertainty: A new simple baseline.
• Please see the reference list of Reviewer e6dg.

Thank you for your concise review. We appreciate your ability to highlight key aspects of our work, and your feedback helped us refine our focus in the revision process.

Weakness

1.

In the original manuscript, we did not include Dirichlet distillation (Proxy-EnD²) in the experiments since the papers for Hyrda and BE reported that their algorithms are superior to Dirichlet distillation. We focused mainly on comparing these two deterministic distillation methods with our DLF.

In Table below, we add Dirichlet distillation. In addition, we add a new distribution distillation method called Ensemble Distillation via Flow Matching (EDFM; Park et al., 2025), which we found after we submitted our paper. Note that DLF outperforms Proxy-EnD² and EDFM with large margins in terms of the accuracy and NLL while DLF is not much worse than Proxy-EnD² in terms of ECE.

2.

Several prior works have shown that deep neural networks converge to Gaussian processes, including (Lee et al., 2018), (Khan et al., 2019), and (Gal et al., 2016). Based on these findings, we consider it natural to assume that a teacher models trained with a deep neural networks induce a distribution that can be approximated by a Gaussian process.

We want to emphasize that superior performance of DLF compared to other distributional distillations is not achieved by chance. Instead, there are various advantages of DLF compared to Proxy-EnD² and Ensemble Distillation via Flow Matching (EDFM; Park et al., 2025), a recent distribution distillation method. We think that there are three properties any desirable models for distribution distillation are expected to have: (1) large support, (2) easy and stable learning algorithm and (3) flexibility to be applied to downstream tasks. Below, we explain why DLF is desirable in view of these three properties.

• DLF is a Gaussian process, which is well known to cover large class of distributions (Garriga-Alonso et al., 2019; Neal et al., 1996; Calandra et al., 2016; Huang et al., 2015). Even if teacher ensembles are not Gaussian, a Gaussian process can approximate it closely. While Dirichlet distributions are often used to represent class probabilities due to their inherent non-negativity and normalization constraints, they are inherently limited to classification tasks as mentioned in answer of Weakness 1. In addition, DLF treats teacher ensembles as random functions rather than random vectors as Dirichlet and Flow matching distillations do. See our reply to Question 1 of Reviewer jk47 for detailed discussions for this point.

• Existence of easy and stable learning algorithms is a critical fact when choosing an algorithm for distributional distillation. Learning the DLF model is numerical easy and stable since the efficient EM algorithm is available. It is well known that estimating the parameters in the Dirichlet distribution is difficult (Minka et al., 2000; Chai et al., 2013) and thus it would not be easy to learn the Dirichlet distillation model which would a partial reason for inferior performance. For Flow matching distillation, pretrained features should be fed when learning. That is, end-to-end learning is not possible. In contrast, DLF is learned in a end-to-end fashion (i.e. feature vectors and distribution are learned simultaneously).

• Flexibility is also an important aspect. As we have shown in Section 5.3, the learned features by DLF is superior than those by a single DNN or Hydra for downstream tasks.

• To sum up, the idea of estimating the distribution for teacher ensembles is not new but efficient implementation is not an easy task and DLF is such a method.

As explained in Introduction, the predictive distribution is obtained by $\hat{p}(y|\mathbf{x})=\sum_{i=1}^n p(y|\mathbf{x},\theta_i)/n,$ where $\theta_1,\ldots,\theta_n$ are ensemble members. For DLF, As shown in Figure 1 (“Inference on new data”), we can generate $\theta_1,\ldots,\theta_n$ by sampling $n$ many independent latent vectors $\mathbf{z}_1,\ldots,\mathbf{z}_n$ from $\mathcal N_q(\mathbf 0,\mathbf I_q)$, forwarding each sample through the network to have ensemble members. All of the uncertainty measures are calculated based on $\hat{p}(y|\mathbf{x}).$ We will state explicitly this procedure in the revision.

3.

In the revised manuscript, we will number all displayed equations. Citations for the baseline methods (e.g., Hydra, Batch-Ensemble) are already provided in Section 2.2, but were omitted later due to the page limitation. For clarity, we will also include the same citations in the description of our experimental setup.

Question

1.

As stated in line 137, $\mu_{\theta}$ and $\Phi_{\theta}$ are local networks whose output sizes depend only on the target dimension and latent dimension; they do not scale with the number of design points. Thus $m$ increases the amount of data processed during training but does not change the per sample inference cost. Moreover, DLF is that it can be learned easily using the EM algorithm. As detailed in Appendix B, the E‑step has the closed‑form solution, so training time is not computationally demanding. To confirm this argument, we measure the computing times for the training as well as inference in the table below:(1) Multiply–Accumulate Operations (MAC, measured in MMac) for inference, (2) inference latency (ms), and (3) training time (hours).
We will add this result in the revised manuscript.

Refference

• Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., & Sohl-Dickstein, J. (2017). Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165.
• Khan, M. E., Immer, A., Abedi, E., & Korzepa, M. (2019). Approximate inference turns deep networks into Gaussian processes. Advances in neural information processing systems, 32.
• Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). PMLR.
• Park, J., Nam, G., Kim, H., Yoon, J., & Lee, J. Ensemble Distribution Distillation via Flow Matching. In Forty-second International Conference on Machine Learning.
• Minka, T. (2000, November). Estimating a Dirichlet distribution.
• Chai, D., Forstner, W., & Lafarge, F. (2013). Recovering line-networks in images by junction-point processes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1894-1901).
• Garriga-Alonso, A., Rasmussen, C. E., & Aitchison, L. (2018). Deep convolutional networks as shallow gaussian processes. arXiv preprint arXiv:1808.05587.
• Neal, R. M. (1996). Priors for infinite networks. In Bayesian learning for neural networks (pp. 29-53). New York, NY: Springer New York.
• Calandra, R., Peters, J., Rasmussen, C. E., & Deisenroth, M. P. (2016, July). Manifold Gaussian processes for regression. In 2016 International joint conference on neural networks (IJCNN) (pp. 3338-3345). IEEE.
• Huang, W. B., Zhao, D., Sun, F., Liu, H., & Chang, E. Y. (2015, June). Scalable Gaussian Process Regression Using Deep Neural Networks. In IJCAI (pp. 3576-3582).

We sincerely appreciate your thorough review and the thoughtful questions you raised. Your comments not only helped us clarify key components of our work but also encouraged us to consider valuable directions for future extensions. Thank you for engaging deeply with our paper.

Weakness

Quality 1.

A novelty of DLF is that it can be learned easily using the EM algorithm. As detailed in Appendix B, the E‑step has the closed‑form solution, so training time is not computationally demanding. To confirm this argument, we measure the computing times for the training as well as inference in the table below: (1) Multiply–Accumulate Operations (MAC, measured in MMac) for inference, (2) inference latency (ms), and (3) training time (hours). We will add this result in the revised manuscript.

Quality 2.

We would like to clarify that packed-ensembles is not a knowledge-distillation method, but rather an efficient ensemble construction technique. We agree that including a recent distribution-distillation baseline would strengthen the evaluation. We have therefore added experiments with “Ensemble Distribution Distillation via Flow Matching” (EDFM; Park et al., 2025) which we found after the submission.

Quality 3.

In the original manuscript, we omitted Dirichlet distillation (Proxy-EnD²) from our experiments because the Hydra and BE studies demonstrated superior performance over it; instead, we focused on comparing those two deterministic distillation methods with our DLF.

In Table below, we add Dirichlet distillation and EDFM. Note that DLF outperforms Proxy-EnD² and EDFM with large margins in terms of the accuracy and NLL while DLF is not much worse than Proxy-EnD² in terms of ECE.

Quality 4.

We agreed that RoBERTa is not of billion-scale LLMs. However, RoBERTa is used popularly for studying finetuning LLM (Yang et al., 2023). We could try to distill larger LLM but it would take significant amount of times. In our experiments, we only distilled the LoRA part of RoBERTa while we used the pretrained distilled RoBERTa for distillation of RoBERTa. We are trying to apply DLF to large LLMs such as (Liu et al., 2024) but we do not think we can get the results by the due date of rebuttal. If possible, we will report the results in the discussion period.

Clarity 1.

We are sorry for notational complexity. Due to page limitation, we moved most of notations into Appendix. In revision, we will try our best to improve the readabilty.

Clarity 2.

The main motivation of the theoretical result in Section 3.4 is to answer the question about the choice of good design points for distribution distillation. The theoretical result suggests that the design points whose distribution is close to that of test data is promising. We will polish Section 3.4 more in the revision.

Clarity 3.

We used the term LLM for RoBERTa since the previous literature (Yang et al., 2023) use this terms. It would be better to name it just SLM (smalle laguage model) or a high-capacity pretrained transformer. We will clarify this terminology in the revision.

Significance 1.

For a large and hard problem, in Section 5.2, we have conducted experiments using RoBERTa (approximately 355 million parameters) to evaluate our method. As the referee suggested, we analyzed Tiny-ImageNet, a 200-class subset of ImageNet. DLF still outperforms the other baselines even including small ensembles.

Because “small-Ens” consists of multiple models, it uses far more parameters. Although its accuracy gains are modest, DLF matches or exceeds its predictive performance, delivers higher coverage, and does so with only a fraction of the parameters and computation. We believe this efficiency–performance balance is exactly the strength of DLF. Please note that baselines such as Hydra and BE also share parameters in small-Ens.

Significance 2.

We are trying to apply DLF to large LLMs but we do not think we can get the results by the due date of rebuttal. If possible, we will report the results in the discussion period.

Significance 3.

Assumptions like Lipschitz continuity and sparsity are common in theoretical analyses of deep generative models and nonparametric neural network regression (e.g., Schmidt-Hieber 2020; Chae 2023; Wing Hung Wong et al. 1995), and we agree they may not always hold in practice. However, the main motivation of the theoretical result in Section 3.4 is to address how to choose design points for distribution distillation. Even if there are many strong assumptions (Gaussianity of teacher ensembles and so on), the theoretical result suggests that the design points whose distribution is close to that of test data is promising. We will polish Section 3.4 more in the revision.

Originality 1.

We want to emphasize that superior performance of DLF compared to other baselines is not achieved by chance. Instead, there are various advantages of DLF compared to Proxy-EnD² and EDFM. We think that there are three properties any desirable models for distribution distillation are expected to have: (1) large support, (2) easy and stable learning algorithm and (3) flexibility to be applied to downstream tasks. Below, we explain why DLF is desirable in view of these three properties.

• DLF is a Gaussian process, which is well known to cover large class of distributions (Garriga-Alonso et al., 2019; Neal et al., 1996; Calandra et al., 2016; Huang et al., 2015). Even if teacher ensembles are not Gaussian, a Gaussian process can approximate it closely. In addition, DLF treats teacher ensembles as random functions rather than random vectors as Dirichlet and Flow matching distillations do. See our reply to Question 1 of Referee 4 for detailed discussions for this point.
• Existence of easy and stable learning algorithms is a critical fact when choosing an algorithm for distributional distillation. Learning the DLF model is numerical easy and stable since the efficient EM algorithm is available. It is well known that estimating the parameters in the Dirichlet distribution is difficult (Minka et al., 2000; Chai et al., 2013) and thus it would not be easy to learn the Dirichlet distillation model which would a partial reason for inferior performance. For Flow matching distillation, pretrained features should be fed when learning. That is, end-to-end learning is not possible. In contrast, DLF is learned in a end-to-end fashion (i.e. feature vectors and distribution are learned simultaneously).
• Flexibility is also an important aspect. As we have shown in Section 5.3, the learned features by DLF is superior than those by a single DNN or Hydra for downstream tasks.
• To sum up, the idea of estimating the distribution for teacher ensembles is not new but efficient implementation is not an easy task and DLF is such a method.

Question

1.

Because our student model is distilled directly from the teacher ensemble, we chose the ensemble’s empirical coverage as the reference: matching it indicates that the student has inherited the teacher’s calibration. We agree, however, that proximity to the nominal target (e.g., 95 %) is also informative. In the revision we will report both values so that readers can assess calibration against the ensemble and against the nominal level.

2.

Thank you for raising this interesting point. Due to the brevity of the rebuttal period, we were unfortunately unable to conduct experiments with teacher models pretrained on different datasets. We plan to use the upcoming discussion phase to perform these experiments and incorporate the results into the final manuscript. We appreciate your valuable suggestion.

3.

Please refer to our response to Significance 1.

4.

Please refer to our response to Quality 3.

5.

As noted in line 146, DLF Framework depends on the output dimension—so as outputs grow, both the parameter count and $\mathbf{Z}$’s dimensionality increase, which can slow training. However, we believe that selecting an appropriate latent dimension $q$ ensures efficient learning.

Moreover, because DLF framework can estimate class‑specific covariance structures, it captures complex relationships more accurately and delivers superior performance on high-dimensional datasets like Tiny-ImageNet. For these reasons, we believe that DLF not only scales to tasks like pixel‑wise regression but may actually gain relative advantage as output dimensionality increases.

6.

We use an inverse-gamma prior for the variance parameter $\sigma^{2}_{\epsilon}$ because it is conjugate to the Gaussian likelihood in our model. This conjugacy leads to closed-form updates within the EM algorithm and simplifies marginal-likelihood computations.

Reference

• Yang, A. X., Robeyns, M., Wang, X., & Aitchison, L. (2023). Bayesian low-rank adaptation for large language models.
• Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., ... & Chandra, V. (2024, February). Mobilellm: Optimizing sub-billion parameter language models for on-device use cases.
• Please see the reference list of Reviewer e6dg.

Thank you for your detailed and thoughtful review. We appreciate the time and care you took in carefully examining various aspects of our work. Your insightful comments helped us identify points for clarification and further improvement. Thank you for your detailed feedback on items 2, 4, 5, 6, 9, 10, 12, 13 in the “Other comments/suggestions” part. I will address these points in the revision.

Weakness

Quality 1.

In the original manuscript, we did not include Dirichlet distillation (Proxy-EnD²) in the experiments since the papers for Hydra and BE reported that their algorithms are superior to Dirichlet distillation. We focused mainly on comparing these two deterministic distillation methods with our DLF.

In the Table below, we add Dirichlet distillation. In addition, we add a new distribution distillation method called Ensemble Distillation via Flow Matching (EDFM; Park et al., 2025), which we found after we submitted our paper. Note that DLF outperforms Proxy-EnD² and EDFM with large margins in terms of the accuracy and NLL while DLF is not much worse than Proxy-EnD² in terms of ECE.

We want to emphasize that superior performance of DLF compared to other distributional distillations is not achieved by chance. Instead, there are various advantages of DLF compared to Proxy-EnD² and EDFM. We think that there are three properties that any desirable models for distribution distillation are expected to have: (1) large support, (2) easy and stable learning algorithm and (3) flexibility to be applied to downstream tasks. Below, we explain why DLF is desirable in view of these three properties.

• DLF is a Gaussian process, which is well known to cover a large class of distributions (Garriga-Alonso et al., 2019; Neal et al., 1996; Calandra et al., 2016; Huang et al., 2015). Even if teacher ensembles are not Gaussian, a Gaussian process can approximate them closely. In addition, DLF treats teacher ensembles as random functions rather than random vectors as Dirichlet and Flow matching distillations do. See our reply to Question 1 of Referee 4 for detailed discussions for this point.

• Existence of easy and stable learning algorithms is a critical fact when choosing an algorithm for distributional distillation. Learning the DLF model is numerically easy and stable since the efficient EM algorithm is available. It is well known that estimating the parameters in the Dirichlet distribution is difficult (Minka et al., 2000; Chai et al., 2013) and thus it would not be easy to learn the Dirichlet distillation model which would be a partial reason for inferior performance. For Flow matching distillation, pretrained features should be fed when learning. That is, end-to-end learning is not possible. In contrast, DLF is learned in an end-to-end fashion (i.e. feature vectors and distribution are learned simultaneously).

• Flexibility is also an important aspect. As we have shown in Section 5.3, the learned features by DLF is superior to those by a single DNN or Hydra for downstream tasks.

• To sum up, the idea of estimating the distribution for teacher ensembles is not new but an efficient implementation is not an easy task and DLF is such a method.

Quality 2.

For inference overhead, we reported the number of parameters in each model in the manuscript in Section C as inference cost is roughly proportional to the number of parameters. As the referee pointed out, in this rebuttal, we measure the computational costs not only for the inference phase but also the training phase in the table below: (1) Multiply–Accumulate Operations (MAC, measured in MMac) for inference, (2) inference latency (ms), and (3) training time (hours). We did not include EDFM since it uses pretrained features (instead of learning features).

Quality 3.

For a large and hard problem, in Section 5.2, we have conducted experiments using RoBERTa (approximately 355 million parameters) to evaluate our method. As the referee suggested, we analyzed Tiny-ImageNet, a 200-class subset of ImageNet whose results are provided below. DLF still outperforms the other baselines even including small ensembles.

We will conduct further experiments on larger-scale datasets (such as ImageNet) during the revision phase and include those results in the updated manuscript.

Clarity.

As explained in the Introduction, the predictive distribution is obtained by $\hat{p}(y|\mathbf{x})=\sum_{i=1}^n p(y|\mathbf{x},\theta_i)/n,$ where $\theta_1,\ldots,\theta_n$ are ensemble members. For DLF, as shown in Figure 1 (“Inference on new data”), we can generate $\theta_1,\ldots,\theta_n$ by sampling $n$ independent latent vectors $\mathbf{z}_1,\ldots,\mathbf{z}_n$ from $\mathcal N_q(\mathbf 0,\mathbf I_q)$, forwarding each sample through the network to have ensemble members. All of the uncertainty measures are calculated based on $\hat{p}(y|\mathbf{x}).$ We will state explicitly this procedure in the revision.

Originality.

DLF is a kind of the conditional factor model (Chen et al., 2022; Gagliardini et al., 2019) where $\mathbf{Z}$ is the factors and $\Phi_{\theta}(\cdot)$ is the factor loadings depending on covariate $\mathbf{x}.$ Even though there are much literature for the conditional factor models, as far as we know, there is no previous attempt to use deep neural networks for the conditional factor loadings (most models use linear factor loadings) presumably because the model is too complex to fit noisy data. Our key contribution is to suggest DLF to ensemble distillation. As we explained in the reply of Quality 1, our suggestion of using DLF is scientifically well motivated.

Other comments/suggestions:

1.

In lines 137-138, we wrote $\boldsymbol{Z} \sim \mathcal{N}_q(\mathbf{0}, \mathbb{I}_q)$ which indicates that $q$ is the dimension of $\boldsymbol{Z}.$

3 .

Please refer to our response to Quality 1.

7 .

Please refer to our response to Quality 2.

8.

We agree that the wording in Lines 305–306 was too strong. We mentioned Bayesian DNN since we tried to write that the learned DLF can be used as a prior for on-device update of the posterior. Due to the page limitation, we deleted this sentence. We will clarify this point in the revision.

11.

The main motivation of the theoretical result in Section 3.4 is to answer the question about the choice of good design points for distribution distillation. Even if there are many strong assumptions (Gaussianity of teacher ensembles and so on), the theoretical result suggests that the design points whose distribution is close to that of test data are promising.

Regarding the second point, we assume that the distributions of the training and test data are the same, and suggest using validation data randomly sampled from the entire data. To support this choice, we have empirically confirmed that using validation data for the design points works well in Appendix D.1.1. We will clarify this point in the revision.

14.

The entire training dataset is denoted as $\mathcal{D}^{\text{train}} = \{(\boldsymbol{x}_1,y_1), \ldots, (\boldsymbol{x}_n,y_n)\}$. For each $i$-th sample in the training set, another sample $(\boldsymbol{x}_i^c, y_i^c)$ is randomly selected. A new mixed sample $(\boldsymbol{x}_i^m, y_i^m)$ is generated by linearly combining the two samples. $(\boldsymbol{x}_i^m, y_i^m) := \lambda_i(\boldsymbol{x}_i, y_i) + (1-\lambda_i)(\boldsymbol{x}_i^c, y_i^c), \quad \lambda_i \in [0,1]$. We will add more explanation of mixup samples in the revision.

15.

The experiment in Figure 8 uses the Boston Housing dataset, as stated in line 642.

16.

While the values are displayed as identical to two decimal places, DLF slightly outperforms small-Ens (83.094 vs. 83.0915). For clarity, we will report results to three decimal places in the revised version to avoid any potential misunderstanding regarding the bold formatting.

17.

We plan to release the full code and scripts for reproducing all experiments in a follow-up update.

Question

1.

Please refer to our response to Quality 1.

2.

Please refer to our response to Quality 2.

3.

Please refer to our response to Clarity.

Reference

• Chen, Q. (2022). A unified framework for estimation of high-dimensional conditional factor models. arXiv preprint arXiv:2209.00391.
• Gagliardini, P., & Ma, H. (2019). Extracting statistical factors when betas are time-varying. Swiss Finance Institute Research Paper, (19-65).
• Please see the reference list of Reviewer e6dg.