PLD: A Choice-Theoretic List-Wise Knowledge Distillation

• We sincerely thank Reviewers fk7Y, Bcg7, ySdm, and x7Lh, as well as the Area and Senior Area Chairs, for their thorough and constructive feedback on PLD.
• We are pleased that the reviewers recognize our method’s strong theoretical foundation (fk7Y, Bcg7), its efficiency and practical significance (fk7Y), the clarity of our derivations and writing (Bcg7, ySdm), and the robustness of our empirical evaluation (Bcg7, ySdm).
• We greatly appreciate the suggestions to broaden our baseline comparisons, quantify runtime overhead, extend experiments to other tasks, and polish the manuscript’s readability. We will incorporate all additional experiments (CIFAR-100 baselines, COCO detection and runtime comparisons) into the revised manuscript.
• We also thank the reviewers for highlighting minor typos and stylistic improvements, which we have addressed in the updated draft.

Thank you Reviewer x7Lh for the detailed and constructive feedback. We are glad you found PLD a novel and effective direction compared to standard logit-based baselines. Below we (1) address writing clarity and consistency by reducing overused em dashes and standardizing the terminology around ListMLE (now consistently described as the likelihood loss); (2) expand empirical comparisons with additional KD baselines on CIFAR-100 under controlled hyperparameters and with an object detection task; (3) clarify the motivation by explaining the practical drawbacks of a separate CE term and the benefit of list-wise ranking in image classification; and (4) clarify the asymmetric temperature design. We hope these additions and clarifications address your concerns, and we would be grateful for your reconsideration of the score.

Q1: Writing clarity & consistency We have carefully revised the manuscript to improve readability and ensure consistent terminology. All over‑long, dash‑connected sentences have been broken into shorter, standalone sentences, and we’ve replaced excess em dashes with commas or parentheses for smoother flow. We also updated lines 186–187 to standardize every occurrence of ListMLE—short for Listwise Maximum Likelihood Estimation—to “ListMLE (likelihood loss),” removing the term “maximum‑likelihood” to match the original paper’s terminology [38].

Reference [38] F. Xia, T.-Y. Liu, J. Wang, W. Zhang, & H. Li. Listwise approach to learning to rank: theory and algorithm. ICML pages 1192–1199, 2008.

Q2. Inclusion of additional KD baselines and tasks: CIFAR‑100 Baselines and Object Detection

Answer: We included DIST (NeurIPS 2022) as a baseline and structured our ImageNet-1K experiments to compare models horizontally across architectures and scales; to further strengthen the empirical comparison we added CIFAR-100 results against other recent KD methods, all trained under the same controlled hyperparameters: AdamW (β₁=0.9, β₂=0.999), learning rate $=0.001$, weight decay $=0.5$, batch size $=128$, for 250 epochs with a cosine-annealing schedule and a 10-epoch linear warm-up. Results are reported as mean $\pm$ std over three runs. Following prior work (e.g., KD and DIST) we set the temperature $\tau=4$ where applicable.

Table A: CIFAR‑100 performance

As discussed in our Conclusion (Sec 6), PLD naturally extends to any task where student and teacher share the same output logits, including detection or segmentation. To validate this, we distilled Faster R‑CNN detectors on COCO for two teacher→student pairs using DKD  and PLD:

Table B: Object detection performance (COCO)

Q3.1: why having a separate cross-entropy term is a significant drawback in practice?

Answer: The separate CE mixing weight is brittle, expensive to tune, and cannot adapt per sample; PLD subsumes CE into a single data-dependent ranking loss, eliminating that hyperparameter.

Standard KD minimizes$\mathcal{L} = \alpha\\mathrm{CE}(y,s) + (1-\alpha)\\mathrm{KL}(q^T\parallel q^S),$ which requires tuning both the CE weight $\alpha$ and the distillation weight $1-\alpha$. As Figure 1a shows (Sec. 1), small changes in $\alpha$ can shift Top-1 accuracy by more than 1%, and setting $\alpha=0$ (dropping CE entirely) causes a notable performance drop. This global hyperparameter is brittle, it doubles the tuning burden compared to CE alone, and it forces costly grid searches (Appendix B), especially because variants like DIST introduce additional weights such as temperature $\tau$ and balancing terms $\beta,\gamma$.

Moreover, individual teacher output distributions vary per example: highly skewed logits require stronger softening to reveal inter-class structure, while flatter logits require less. A fixed $\alpha$ cannot adapt to this heterogeneity. In contrast, PLD’s first Plackett–Luce factor coincides with CE on the true class, and each subsequent ranking step is weighted by the teacher’s own softmax mass $\alpha_k = q^T_{\pi^*_k}$, making the supervision inherently data-dependent (Sec. 4.2). This unification subsumes CE into a single convex ranking loss and reduces the total hyperparameter count by one (removing $\alpha$).

Motivated by Sutton’s “bitter lesson” that general methods scale best with compute, PLD offers a unified, self-tuning objective that reduces tuning overhead and focuses compute on scale rather than hyperparameter search.

Q3.2: What’s the advantage of list-wise ranking for image classification?

Answer: During inference, classification sorts the output logits—for example, it computes $\arg\max_i s_i$ or selects the top-$k$ values. Training typically uses the cross-entropy loss$\mathcal{L}_{\mathrm{CE}} = -\sum_i y_i \log s_i.$ This loss enforces $s_y > s_i$ for all $i \neq y$. As shown in Section 4.1 (L172–L173), this constraint matches the first selection step in the Plackett–Luce model. Softmax guarantees $s_i > 0$ for every class, and cross-entropy training avoids overconfidence, yielding smooth output distributions. As a result, the detailed ordering among non-target classes is not fully exploited. In this sense, cross-entropy behaves like a relaxed ranking loss.Knowledge distillation captures these residual relationships via the teacher’s soft distribution $q^T$. For example, classes $a,b,c,d,e$ with true label $e$, cross-entropy enforces four independent constraints: $s_e > s_a$, $s_e > s_b$, $s_e > s_c$, and $s_e > s_d$. A list-wise ranking loss instead enforces the full ordering$e > a > b > c > d$in a single objective. This objective also enforces sub-rankings such as $a > b > c$ and $b > c > d$.Empirically, Table 2 shows that PLD’s ListMLE surrogate outperforms cross-entropy. This confirms that supervising the entire teacher-optimal permutation yields stronger Top-1 accuracy gains. Similar benefits of list-wise losses over pairwise losses have been observed in learning-to-rank methods such as ListNet [1].

Reference

[1] Learning to rank: from pairwise approach to listwise approach. ICML 2007.

Q4: Why is temperature applied only to the teacher logits in PLD, rather than symmetrically to both teacher and student as in temperature-scaled KL divergence?

Answer: Applying a temperature $\tau$ to the student logits $s \mapsto s' = s/\tau$ simply rescales the entire PLD loss by $1/\tau$. Specifically,$\mathcal{L}(s') = -\sum_k \alpha_k \log P_{\mathrm{student}}(\pi^*; s/\tau)= \frac{1}{\tau}\,\mathcal{L}(s).$Under gradient descent with learning rate $\eta$, the update$s \leftarrow s - \eta\,\frac{\partial \mathcal{L}(s')}{\partial s} = s - \frac{\eta}{\tau}\,\frac{\partial \mathcal{L}(s)}{\partial s}$is equivalent to using the original loss with an effective learning rate $\eta' = \eta/\tau$. Thus student-side softening does not add modeling expressivity; it only changes the step size. In Supplementary Sec. 3 (Table 2) We find that standardizing both teacher and student logits yields 70.66% Top-1 accuracy, while standardizing only the teacher logits gives 70.81%, compared to 71.16% with no logit scaling.

Baselines & References

1. KD:arXiv:1503.02531
2. DIST arXiv:2205.10536
3. DKD: arXiv:2203.08679
4. AT: arXiv:1612.03928
5. FitNet:arXiv:1412.6550
6. PKT:arXiv:1803.10837
7. RKD: arXiv:1904.05068
8. SP: arXiv:1907.09682
9. VID: arXiv:1904.05835

• We sincerely thank Reviewers fk7Y, Bcg7, ySdm, and x7Lh, as well as the Area and Senior Area Chairs, for their thorough and constructive feedback on PLD.
• We are pleased that the reviewers recognize our method’s strong theoretical foundation (fk7Y, Bcg7), its efficiency and practical significance (fk7Y), the clarity of our derivations and writing (Bcg7, ySdm), and the robustness of our empirical evaluation (Bcg7, ySdm).
• We greatly appreciate the suggestions to broaden our baseline comparisons, quantify runtime overhead, extend experiments to other tasks, and polish the manuscript’s readability. We will incorporate all additional experiments (CIFAR-100 baselines, COCO detection and runtime comparisons) into the revised manuscript.
• We also thank the reviewers for highlighting minor typos and stylistic improvements, which we have addressed in the updated draft.

Thank you Reviewer ySdm for the thoughtful and encouraging feedback. We’re pleased you found the paper’s organization clear, the motivation sound, and the empirical gains convincing across architectures including ViT. Your pointer to DKD (CVPR 2022) helped us broaden our baseline set, and your suggestion to reorganize the experimental section has improved clarity and flow. Without these suggestions it would have been very difficult to coordinate and execute so many additional experiments in the limited time available. In the revised manuscript we will (1) include DKD among the CIFAR-100 baselines and present key results up front with supporting tables moved to follow, (2) extend PLD to object detection on COCO to demonstrate its applicability beyond classification, and (3) refine the experimental layout to highlight these comparisons. We hope these additions further strengthen your confidence in our work.

Inclusion of additional KD baselines and tasks: CIFAR‑100 Baselines and Object Detection

We included DIST (NeurIPS 2022) as a baseline and structured our ImageNet-1K experiments to compare models horizontally across architectures and scales; to further strengthen the empirical comparison we added CIFAR-100 results against other recent KD methods, all trained under the same controlled hyperparameters: AdamW (β₁=0.9, β₂=0.999), learning rate $=0.001$, weight decay $=0.5$, batch size $=128$, for 250 epochs with a cosine-annealing schedule and a 10-epoch linear warm-up. Results are reported as mean $\pm$ std over three runs. Following prior work (e.g., KD and DIST) we set the temperature $\tau=4$ where applicable.

Table A: CIFAR‑100 performance under identical hyperparameters

As discussed in our Conclusion (Sec 6), PLD naturally extends to any task where student and teacher share the same output logits, including detection or segmentation. To validate this, we distilled Faster R‑CNN detectors on COCO for two teacher→student pairs using DKD  and PLD:

Table B: Object detection performance (COCO)

Baselines & References

1. KD: Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531
2. DIST: Huang, T., You, S., Wang, F., Qian, C., & Xu, C. (2022). Knowledge Distillation from A Stronger Teacher (DIST). arXiv:2205.10536
3. DKD: Chen, Z., Liu, W., & Tao, D. (2022). Decoupled Knowledge Distillation. arXiv:2203.08679
4. AT: Zagoruyko, S., & Komodakis, N. (2016). Paying More Attention to Attention. arXiv:1612.03928
5. FitNet: Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). FitNets: Hints for Thin Deep Nets. arXiv:1412.6550
6. PKT: Passalis, N., & Tefas, A. (2018). Learning Deep Representations with Probabilistic Knowledge Transfer. arXiv:1803.10837
7. RKD: Park, W., Kim, D., Lu, Y., & Cho, M. (2019). Relational Knowledge Distillation. arXiv:1904.05068
8. SP: Tung, F., & Mori, G. (2019). Similarity‑Preserving Knowledge Distillation. arXiv:1907.09682
9. VID: Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., & Dai, Z. (2019). Variational Information Distillation for Knowledge Transfer. arXiv:1904.05835

• We sincerely thank Reviewers fk7Y, Bcg7, ySdm, and x7Lh, as well as the Area and Senior Area Chairs, for their thorough and constructive feedback on PLD.
• We are pleased that the reviewers recognize our method’s strong theoretical foundation (fk7Y, Bcg7), its efficiency and practical significance (fk7Y), the clarity of our derivations and writing (Bcg7, ySdm), and the robustness of our empirical evaluation (Bcg7, ySdm).
• We greatly appreciate the suggestions to broaden our baseline comparisons, quantify runtime overhead, extend experiments to other tasks, and polish the manuscript’s readability. We will incorporate all additional experiments (CIFAR-100 baselines, COCO detection and runtime comparisons) into the revised manuscript.
• We also thank the reviewers for highlighting minor typos and stylistic improvements, which we have addressed in the updated draft.

Thank you Reviewer Bcg7 for the detailed and constructive feedback. We are glad you found the paper well written, clear, theoretically sound, and that the improvements of PLD over KD and DIST across teacher-student pairs are convincing. We also appreciate your pointer to relevant prior work such as Contrastive Representation Distillation (Tian et al., ICLR 2020), which helped guide our expansion of baselines and the design of additional CIFAR-100 experiments. Without these suggestions it would have been very difficult to coordinate and execute so many additional experiments in the limited time available. Below we (1) expand the empirical comparison with additional CIFAR-100 baselines, including uncertainty estimates from three runs to provide error bars (2) broaden the set of KD methods evaluated to strengthen the comparison and (3) demonstrate extensions to object detection. We hope these additions and clarifications address your concerns, and we would be grateful for your reconsideration of the score.

Inclusion of additional KD baselines and tasks: CIFAR‑100 Baselines and Object Detection

We included DIST (NeurIPS 2022) as a baseline and structured our ImageNet-1K experiments to compare models horizontally across architectures and scales; to further strengthen the empirical comparison we added CIFAR-100 results against other recent KD methods, all trained under the same controlled hyperparameters: AdamW (β₁=0.9, β₂=0.999), learning rate $=0.001$, weight decay $=0.5$, batch size $=128$, for 250 epochs with a cosine-annealing schedule and a 10-epoch linear warm-up. Results are reported as mean $\pm$ std over three runs. Following prior work (e.g., KD and DIST) we set the temperature $\tau=4$ where applicable.

Table A: CIFAR‑100 performance under identical hyperparameters

As discussed in our Conclusion (Sec 6), PLD naturally extends to any task where student and teacher share the same output logits, including detection or segmentation. To validate this, we distilled Faster R‑CNN detectors on COCO for two teacher→student pairs using DKD  and PLD:

Table B: Object detection performance (COCO)

Baselines & References

1. KD: Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531
2. DIST: Huang, T., You, S., Wang, F., Qian, C., & Xu, C. (2022). Knowledge Distillation from A Stronger Teacher (DIST). arXiv:2205.10536
3. DKD: Chen, Z., Liu, W., & Tao, D. (2022). Decoupled Knowledge Distillation. arXiv:2203.08679
4. AT: Zagoruyko, S., & Komodakis, N. (2016). Paying More Attention to Attention. arXiv:1612.03928
5. FitNet: Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). FitNets: Hints for Thin Deep Nets. arXiv:1412.6550
6. PKT: Passalis, N., & Tefas, A. (2018). Learning Deep Representations with Probabilistic Knowledge Transfer. arXiv:1803.10837
7. RKD: Park, W., Kim, D., Lu, Y., & Cho, M. (2019). Relational Knowledge Distillation. arXiv:1904.05068
8. SP: Tung, F., & Mori, G. (2019). Similarity‑Preserving Knowledge Distillation. arXiv:1907.09682
9. VID: Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., & Dai, Z. (2019). Variational Information Distillation for Knowledge Transfer. arXiv:1904.05835

• We sincerely thank Reviewers fk7Y, Bcg7, ySdm, and x7Lh, as well as the Area and Senior Area Chairs, for their thorough and constructive feedback on PLD.
• We are pleased that the reviewers recognize our method’s strong theoretical foundation (fk7Y, Bcg7), its efficiency and practical significance (fk7Y), the clarity of our derivations and writing (Bcg7, ySdm), and the robustness of our empirical evaluation (Bcg7, ySdm).
• We greatly appreciate the suggestions to broaden our baseline comparisons, quantify runtime overhead, extend experiments to other tasks, and polish the manuscript’s readability. We will incorporate all additional experiments (CIFAR-100 baselines, COCO detection and runtime comparisons) into the revised manuscript.
• We also thank the reviewers for highlighting minor typos and stylistic improvements, which we have addressed in the updated draft.

Thank you Reviewer fk7Y for the detailed and constructive feedback. We’re glad the reviewers (including fk7Y and others) recognize PLD’s choice-theoretic foundation, its convex differentiable surrogate with closed-form gradients, and its elimination of manual tuning between cross-entropy and distillation. Below we (1) show how PLD transfers richer intra-class information beyond soft-label remapping, (2) present expanded CIFAR-100 baselines under identical hyperparameters, (3) provide empirical runtime analyses, (4) demonstrate extensions to object detection, and (5) describe improvements to clarity and consistency in the manuscript. We hope these additions and clarifications address your concerns, and we would be grateful for your reconsideration of the score.

Q1. Is converting the teacher’s outputs into a ranked list and then using the Plackett–Luce model to weight labels based on logits merely a remapping of the original soft labels, or does it extract substantially more useful information?

Answer: PLD extracts richer intra-class information than standard KD by supervising the full teacher permutation with confidence-weighted list-wise ranking, instead of just remapping soft labels.

In knowledge distillation we add extra constraints to the loss to extract richer information from the teacher. Cross-entropy alone only pushes the student toward the top class, whereas distillation terms transfer subtler inter-class relations. PLD uses weighted list-wise ranking: given a teacher ranking $a > b > c > d > e$, supervising this single permutation not only enforces $a$ as the top choice but also embeds every relative preference (e.g., $a>b>c$, $b>c>d$, etc.). Furthermore, each selection step is weighted by the teacher’s softmax confidence (so $\alpha_1 = q^T_a \gg \alpha_5 = q^T_e$), meaning PLD emphasizes high-confidence distinctions like $a>b$ over low-confidence ones like $d>e$, all within one forward pass.

By contrast, standard KD (CE + KL) depends on softmax normalization: when the teacher is highly confident, the target is near one-hot and carries little structure, so the KL term must be softened via temperature (Hinton et al., 2015) just to expose any ordering. PLD instead directly minimizes the negative log-likelihood of the entire teacher-optimal permutation $\pi^* = (y, \text{argsort}(t)\setminus\{y\})$, aligning all ordering relations in a single convex loss (Sec. 4.1, L183–L184), and uses confidence weights $\alpha_k = q^T_{\pi^*_k}$ to prioritize the strongest signals (Sec. 4.2). This yields substantially richer intra-class information than mere soft-label remapping.

Q2: Inclusion of additional KD baselines.

Answer: We included DIST (NeurIPS 2022) as a baseline and structured our ImageNet-1K experiments to compare models horizontally across architectures and scales; to further strengthen the empirical comparison we added CIFAR-100 results against other recent KD methods, all trained under the same controlled hyperparameters: AdamW (β₁=0.9, β₂=0.999), learning rate $=0.001$, weight decay $=0.5$, batch size $=128$, for 250 epochs with a cosine-annealing schedule and a 10-epoch linear warm-up. Results are reported as mean $\pm$ std over three runs. Following prior work (e.g., KD and DIST) we set the temperature $\tau=4$ where applicable.

Table A: CIFAR‑100 performance under identical hyperparameters

Q3. Does introducing the Plackett–Luce model with its closed-form gradients incur significant additional training cost?

Answer: PLD’s runtime is on par with state-of-the-art KD methods while yielding the high efficiency. PLD enforces only a single teacher‑optimal permutation (O(C log C) via sorting) rather than summing over all C! rankings—just as ListMLE  and P‑ListMLE do—making it computationally feasible and directly comparable to other state‑of‑the‑art distillation methods. Below is the run time of Table A’s methods:

Table B: Total training time (minutes)

Table C: Accuracy gain over time (ΔTop‑1 / runtime)

Q4. Why are potential applications beyond image classification (e.g., NLP, image segmentation) not explored experimentally?

Answer: As discussed in our Conclusion (Sec 6), PLD naturally extends to any task where student and teacher share the same output logits, including detection or segmentation. To validate this, we distilled Faster R‑CNN detectors on COCO for two teacher→student pairs using DKD  and PLD:

Table D: Object detection performance (COCO)

Given PLD’s competitive performance, we hope other researchers will adopt it for additional domains such as NLP, as discussed in Section 6 (L303).

Q5. This paper is difficult to understand.

Answer: While we appreciate this feedback, several other reviewers found the paper clear and accessible. For example, Reviewer Bcg7 writes, “The paper is well written and clear. The method is well-motivated and the derivations are fairly easy to follow.” and Reviewer ySdm notes, “The organization of this paper is very clear, so readers can understand the paper easily.” Nonetheless, we’ve further smoothed the narrative flow by breaking down long, dash‑connected sentences into shorter, standalone sentences and adding brief transitional phrases, making the text easier to follow. We further appreciate your detailed suggestions for improving the writing and will incorporate them in the revised manuscript.

Baselines & References

1. KD: arXiv:1503.02531
2. DIST: arXiv:2205.10536
3. DKD: arXiv:2203.08679
4. AT: arXiv:1612.03928
5. FitNet:arXiv:1412.6550
6. PKT: arXiv:1803.10837
7. RKD: arXiv:1904.05068
8. SP: arXiv:1907.09682
9. VID: arXiv:1904.05835

Dear Reviewer fk7Y,

thank you once again for taking the time to review our paper, PLD: A Choice-Theoretic List-Wise Knowledge Distillation. We have submitted our rebuttal and provided clarifications addressing your comments, and as the rebuttal period is progressing, we would greatly appreciate it if you could kindly take a moment to review our response to ensure it adequately addresses your concerns. Your feedback is invaluable to us, and we sincerely hope that our clarifications can help you reconsider our work in light of the points addressed. If you have any further questions or require additional details, please do not hesitate to contact us. Thank you very much for your time and consideration.

Best regards,

Authors of PLD: A Choice-Theoretic List-Wise Knowledge Distillation

Thank you for your constructive and insightful comments on my manuscript. I have carefully considered your feedback, and I am pleased to inform you that your responses have addressed the majority of my concerns. Your suggestions have significantly clarified several aspects of the paper, and I now feel more confident in its contribution to the field.

As a result, I have revised my assessment and have decided to increase the score accordingly. I believe the paper does not deserve a rating of 5 because it only includes two baselines on ImageNet-1K. Please include more comparative experimental results.