Q1: gap between the motivation of the proposed method and its findings

A1: The function of $L_{di}$ is to extract knowledge from unexplored foreground regions within the background areas, rather than merely utilizing background information. These regions present a challenge to the model, and without constraints, the model's limited learning capacity makes it difficult to autonomously explore them. As evidenced in Table 13, when $L_{di}$ is applied alone, the teacher model's performance improves by 0.29 (73.78 vs. 74.07), thereby increasing the upper bound of the student model's distillation performance. This demonstrates the necessity for the teacher model to continuously extract challenging foreground knowledge from the background regions.

We posit that in the online distillation setting, both $L_{co}$ and $L_{di}$ must work in synergy. If only $L_{co}$ is applied without $L_{di}$, the teacher model cannot persistently explore more difficult foreground regions. Conversely, if only $L_{di}$ is applied without $L_{co}$, the student model cannot effectively learn the foreground regions identified by the teacher model from easy to difficult, failing to achieve the intended mutual enhancement of both models' performance.

Q2: explain why they made such design choices instead of merely mentioning them in passing.

A2: In the introduction, we outlined the rationale behind ADM based on two key findings. The proposed method is straightforward yet effective.

Q3: Does the expression in lines 69-70 have any errors?.

A3: There is no error here. Figure 1 illustrates the changes in non-similar regions when comparing our method with vanilla training, which corresponds to Finding 2. It does not address the description of similar regions in Finding 1.

Q1: lack of rigorous theoretical justification

A1: Firstly, we assert that the proposed ADM method adheres to theoretical derivations. To substantiate this claim, we provide additional theoretical support:

Information-Theoretic Analysis of Asymmetric Decision-Making (ADM)

1. Notation and Fundamental Assumptions

Symbolic Convention

• Input data: $$X \in \mathcal{X}$$
• Task labels: $$Y \in \mathcal{Y}$$
• Student features: $$Z_s = f_s(X) \in \mathbb{R}^{H \times W \times C}$$
• Teacher features: $$Z_t = f_t(X) \in \mathbb{R}^{H \times W \times C}$$
• Spatial indices: $$(i,j) \in {1,\dots,H} \times {1,\dots,W}$$
• consensus mask in Eq6[L196-L200 in paper]: $$M_f = \frac{1+\mathcal{S}}{1+ \bar {\mathcal{S}}}$$
• divergence mask in Eq6[L183-L187 in paper]: $$M_d = \frac{1-\mathcal{S}}{1- \bar {\mathcal{S}}}$$

Information-Theoretic Quantities

• Entropy:

$

$H(Z) = -\mathbb{E}_{Z \sim p(Z)}[\log p(Z)]$

$

• Mutual Information:

$

$I(Z; Y) = H(Y) - H(Y|Z)$

$

• Conditional Mutual Information: $I(Z_s; Z_t | X)$= ${E_X} \left[ D_{\text{KL}}\left( p(Z_s,Z_t|X) \,\|\, p(Z_s|X)p(Z_t|X) \right) \right]$

---

2. Consensus Learning: Feature Alignment via Mutual Information Maximization

Objective Function

The student model maximizes mutual information in foreground regions:

$

$\mathcal{L}_{\text{co}} = -I(Z_s^{M_f}; Z_t^{M_f}), \quad \text{where } Z^{M_f} = Z \odot M_f$

$

Variational Lower Bound

Theorem 1 (Barber-Agakov Variational Bound):

$I(Z_s^{M_f}; Z_t^{M_f}) \geq {E}_{Z_s^{M_f}, Z_t^{M_f}} [ \log q(Z_t^{M_f} | Z_s^{M_f}) ] + H(Z_t^{M_f})$

Minimizing the negative log-likelihood yields:

$$ {L}_{co} = E [ -\log q(Z_t^{M_f} | Z_s^{M_f}) ]$

$

Information Bottleneck Interpretation

Constraining foreground alignment enforces:

$$ \max , \left[ I(Z_s; Y) - \beta I(Z_s; X) \right] \quad \text{s.t. } I(Z_s^{M_f}; Z_t^{M_f}) \geq C$

$

Resulting in the lower bound:

$$ I(Z_s; Y) \geq I(Z_t^{M_f}; Y) - \beta I(Z_s; X) + C$

$

---

3. Divergence Learning: Entropy Maximization for Feature Space Expansion

Objective Formulation

The teacher model maximizes entropy in low-consensus regions:

${L}_{di}$ = ${-H(Z_t^{M_d})}+ \lambda {I(Z_t^{M_d}; Y)}$

Jaynes' Maximum Entropy Principle

Lemma 1 (Optimal Feature Distribution): Under constraint $$I(Z_t^{M_d}; Y) \geq \epsilon $$, the optimal distribution satisfies:

$$ p^*(Z_t^{M_d}) \propto \exp\left( \lambda \log p(Y | Z_t^{M_d}) \right)$

$

Coverage Capacity Analysis

Theorem 2 (Feature Space Coverage):

$$ \mathcal{R}(Z_t) \geq \sqrt{2 H(Z_t^{M_d})}$

$

where $$\mathcal{R}(Z_t)$$ denotes the radius of feature coverage.

---

4. Convergence Analysis via Dynamical Systems

Coupled Dynamics

The training process is modeled as:

$$ \begin{cases} \frac{d}{dt} I(Z_s; Y) = \alpha \left( I(Z_t; Y) - I(Z_s; Y) \right) \ \frac{d}{dt} H(Z_t) = \beta \left( H(Z_t) - H(Z_s) \right) \end{cases}, \quad \alpha, \beta > 0$

$

Lyapunov Stability Proof

Theorem 3 (Global Convergence):

Define the Lyapunov function:

$$ V(t) = \left( I(Z_t; Y) - I(Z_s; Y) \right)^2 + \left( H(Z_t) - H(Z_s) \right)^2$

$

The time derivative satisfies:

$$ \dot{V}(t) = -2\alpha \left( I(Z_t; Y) - I(Z_s; Y) \right)^2 - 2\beta \left( H(Z_t) - H(Z_s) \right)^2 \leq 0$

$

Guaranteeing convergence to equilibrium $$I(Z_s; Y) = I(Z_t; Y)$$ and $$H(Z_t) = H(Z_s) $$.

---

5. Empirical Validation and Error Bounds

Fano's Inequality Application

Theorem 4 (Classification Error Bound):

$$ P_e \geq \frac{H(Y|Z_s) - 1}{\log |\mathcal{Y}|} \implies P_e^{\text{ADM}} \leq \frac{H(Y) - I(Z_s; Y) - 1}{\log |\mathcal{Y}|}$

$

---

6. Generalization to Structured Tasks

Semantic Segmentation

Pixel-wise regional mutual information:

$$ I_{\text{region}}(Z_s, Z_t) = \sum_{c=1}^C I(Z_s^c; Z_t^c)$

$

Diffusion Model Distillation

Temporal trajectory alignment:

$$ I_{\text{traj}}(Z_s, Z_t) = \sum_{t=1}^T \gamma^{T-t} I(Z_s^t; Z_t^t), \quad \gamma \in (0,1)$

$

---

Q2: statistical significance testing

A2: As detailed in lines 272-273 of the paper, the majority of experiments were conducted multiple times, and we report the mean results. This approach demonstrates the robustness and consistency of the performance improvements.

---

Q3: analyzed the computational trade-offs

A3: Please refer to lines 241-247 of the paper. ADM introduces no additional trainable parameters, and the incremental computational cost is significantly minimal compared to the original setup.

Q1: the effects of consensus learning need to be further clarified

A1: Applying $L_{co}$ to the student model does not imply that the performance improvement of the student model is primarily due to $L_{co}$. As evidenced in Table 13, when $L_{co}$​ is used alone, the teacher model's performance remains unchanged at 73.78 (73.78 vs. 73.78), which inherently limits the potential improvement of the student model through distillation. Conversely, when only $L_{di}$ is applied, the teacher model's performance increases by 0.29 (73.78 vs. 74.07), thereby raising the upper bound of the student model's distillation performance. Hence, although $L_{di}$ is applied to the teacher model, it significantly enhances the student model's performance.

We contend that in the online distillation setting, both $L_{co}$ and $L_{di}$ need to work synergistically. If only $L_{co}$ is used, the teacher model cannot continuously explore more challenging foreground regions. Conversely, if only $L_{di}$ is used, the student model cannot effectively learn the foreground regions identified by the teacher model from easy to difficult, failing to achieve the intended mutual enhancement of both models' performance.

Q2: whether the gradient from Lco and Ldi propagates back to the cosine similarity matrix (S) via the attention scores A?

A2: No gradients are propagated in there. Please refer to the code in Appendix L629, where the detach operation is implemented.

Q3: Evaluation of the proposed method upon vision transformer-based (ViT) models is missing.

A3: We conducted experiments using DiT-based diffusion model distillation, which may indirectly support the effectiveness of our method in transformer-based distillation scenarios.

Q4: the 1T2S and 2T1S variants of DML are missing as important baselines.

A4: We adhered to the experimental settings outlined in the SwitOKD paper to demonstrate the extendibility of our approach for training multiple networks.

Q5: The experimental setup of the extension to diffusion distillation is unclear.

A5: As detailed in lines L292-L296 of the referenced paper, for CIFAR10, we followed the experimental settings from the RCFD paper. For DiT model distillation, we adhered to the experimental settings from the DiT paper. All parameters were reused from the original papers without any modifications.

Q6: The visualisation provided in Fig.4 and Fig.5 is hard to understand.

A6: There was a typo error. The third comparison is included in the appendix, while only two comparisons are presented here.