Query-based Knowledge Transfer for Heterogeneous Learning Environments

We sincerely thank the reviewer for their thoughtful feedback and recognition of our work. We are pleased that you found the paper well-motivated, with clear writing and figures that effectively convey our objectives. Your acknowledgment of our innovative use of noise to identify expert models validates our approach and its practical value in addressing key challenges in collaborative learning. Here are our responses to the concerns.

1. Addressing Low Baseline Accuracies in Table 2

Thank you for highlighting this observation. The performance of baseline collaborative learning approaches under data heterogeneity in our experiments aligns with findings from prior works addressing similar scenarios. Specifically, federated learning with non-IID data distributions is known to result in performance degradation due to local data imbalance and limited diversity. Studies such as "No Fear of Heterogeneity" (NeurIPS 2021) and "Personalized Cross-Silo Federated Learning on Non-IID Data" (AAAI 2021) discuss how this heterogeneity leads to suboptimal performance for generalized and personalized FL methods. The baselines used in our experiments, such as FedAvg and FedProx, are tested under configurations comparable to those in prior research. Methods like PFNM and CLUE, designed for single-round learning, struggle with heterogeneity in data distributions and complex model structures, leading to suboptimal performance.

This emphasizes the limitations of these methods in handling data heterogeneity effectively and highlights the need for targeted approaches like QKT, which addresses these challenges through query-specific, communication-efficient knowledge transfer.

2. Overconfidence, Tuning of λ, and Handling Multiple Expert Scenarios

Thank you for this insightful comment. We appreciate the opportunity to elaborate on these points.

Overconfidence and the Role of λ: The concept of overconfidence is leveraged in our masking strategy to identify and filter irrelevant knowledge contributions. The parameter λ in Equation (3) controls the trade-off between focusing on query classes and retaining existing knowledge. While tuning λ is useful to balance this trade-off, our experiments show that λ values between 1.5 and 2 (we used 1.7 for most datasets) were effective, based on empirical observations from CIFAR-10. Even the default value of λ=1 demonstrated notable improvements, as shown in the ablation studies (Section 4.3). This indicates that QKT is not overly sensitive to λ under typical settings.

Clients as Experts in Multiple Classes: When a teacher model is proficient in multiple classes, our approach naturally incorporates this by using the masking mechanism to retain knowledge relevant to all those classes. The synthetic Gaussian noise samples utilized during masking allow the model to effectively identify and focus on the teacher’s areas of expertise.

Multiple Clients as Experts in a Single Class: In scenarios where multiple teachers are proficient in the same class, QKT aggregates knowledge from these teachers by combining their masked outputs, allowing the student to benefit from diverse perspectives while maintaining robustness.

Our experiments evaluate both scenarios where a teacher is an expert in a single class (single-class query) and where a teacher is proficient in a subset of classes (multi-class query). In both cases, multiple teachers can be utilized simultaneously to achieve the student’s learning objectives, ensuring flexibility and robustness in real-world applications. We hope this response clarifies these aspects and demonstrates how QKT effectively handles such scenarios.

4. Regarding Convergence Guarantees

Thank you for raising this insightful question. Convergence in the traditional sense of iterative optimization does not directly apply to QKT and, accordingly, we do not devise a theory for this. Instead, our empirical results across various datasets and scenarios demonstrate that QKT is effective in practice.

As shown in Section 4.2 (Table 2), Section 4.3 (Table 3), and Appendix A.5 and A.6, QKT achieves stable and competitive performance after a single round of knowledge transfer and refinement. These results consistently highlight QKT's ability to balance learning the query task and mitigating forgetting, even under heterogeneous data distributions.

While QKT’s empirical effectiveness is empirically supported, formal theoretical guarantees for convergence in decentralized and heterogeneous settings remain an open research area. Exploring such guarantees is a promising direction for future work, potentially building on theoretical frameworks from personalized federated learning and knowledge distillation. We will acknowledge this potential direction for future work. We hope this response clarifies QKT’s approach and its demonstrated ability to meet its learning objectives efficiently within a single round. Thank you for the opportunity to address this aspect.

5. Privacy Concerns with Noise-Based Filtering

Thank you for your comment regarding the potential privacy concerns of using noise to determine teacher expertise. In QKT, the synthetic noise-based filtering is designed to avoid revealing direct information about client data distributions. The generated noise inputs are independent of client-specific data, and the predictions reflect only the model’s inherent biases, as detailed in Section 3.3. These outputs are used solely for filtering irrelevant teachers and are not shared or used to infer other clients’ data.

It is important to note that FL primarily aims to protect raw data from being shared. While sharing model weights or gradients is common, these can still be susceptible to privacy attacks such as membership inference (Shokri et al., CCS 2017), model inversion (Fredrikson et al., CCS 2015), and property inference (Melis et al., CCS 2019). However, general statistical information, such as the number of training samples, is often used in FL for model aggregation (McMahan et al., AISTATS 2017) or per-class statistics to handle heterogeneity (Wang et al., ICLR 2020).

In highly privacy-sensitive scenarios, techniques like differential privacy (Abadi et al., CCS 2016), secure multi-party computation (Bonawitz et al., CCS 2017), or homomorphic encryption (Gentry, STOC 2009) could be investigated and potentially integrated into QKT.

We hope this response clarifies how QKT aligns with FL privacy principles and its adaptability to stringent privacy requirements.

3. Comparisons with Relevant FL + KD Baselines

Thank you for highlighting the importance of including comparisons with relevant FL + KD baselines. In response to this and to a similar concern raised by another reviewer, we conducted additional experiments incorporating PerAda, a recent personalized FL baseline that integrates both KD and FL.

These experiments were performed on CIFAR-10 using both pathological and Dirichlet data distributions, focusing on single-class and multi-class queries. To ensure a fair comparison, the shared parameters in PerAda were initialized by distilling knowledge from all local models. In this context, PerAda (local) refers to the average performance of personalized models trained individually for each client, while PerAda (global) reflects the average performance of the global model concerning each client’s query.

Detailed results for PerAda are included in the table below, alongside additional baselines in Table 2 and Appendix A.7, Table 20. The results show that QKT consistently outperforms PerAda, achieving higher query accuracy gains, improved overall accuracy, and reduced forgetting rates. These results highlight QKT’s superior ability to address the challenges posed by heterogeneous data distributions and client-specific objectives in collaborative learning scenarios.

Table 1: Pathological Distribution

---

Table 2: Dirichlet Distribution

We sincerely thank the reviewer for their feedback. We are pleased that the efficiency of our approach in reducing communication overhead and its adaptability across diverse datasets and tasks were appreciated. Here are our responses to the concerns.

1. Motivation and Practical Applications of QKT

Thank you for this insightful comment. We appreciate the opportunity to clarify the practical applications that motivate QKT and how it differs from personalized federated learning (PFL). We have included the motivations in lines 34–39 and expand on them here.

Motivation and Practical Applications:
QKT is particularly suited for scenarios where clients or entities own specialized knowledge that can enhance the performance of other clients with similar needs, even if the required knowledge is limited or absent in their local data. For instance, in healthcare, hospitals may need specific expertise on rare or emerging diseases not represented in their local datasets. With QKT, these hospitals can benefit from the relevant knowledge of peer institutions with similar cases, enhancing diagnostic capabilities without sharing sensitive data. Other practical applications include finance, where institutions may need insights on regional fraud patterns, and cybersecurity, where organizations require knowledge of emerging threats observed in other sectors.

Distinction from Personalized FL:
While PFL aims to tailor models to individual clients, it typically relies on iterative, collaborative training rounds and may not effectively address cases where critical knowledge is limited or entirely absent in a client’s local data. QKT fills this gap by enabling direct knowledge transfer from pre-trained peers to clients needing specific information, without requiring iterative rounds. This approach provides a scalable, efficient solution for cases requiring targeted knowledge adaptation, especially when immediate integration of specialized expertise is essential.

2. Novelty of QKT Contributions

Thank you for this comment. We acknowledge that both knowledge distillation and classification head refinement have been applied in various forms within personalized FL, as indicated in recent works. However, FL is not a well-suited solution to our problem, which we describe below.

Novelty of QKT:
QKT introduces a query-based, selective knowledge transfer mechanism that addresses scenarios where critical knowledge may be limited or absent in a client’s local data. QKT directly enables a client to acquire targeted knowledge from multiple peer models that are already pre-trained on relevant tasks, without the need for iterative communication rounds. Additionally, our approach leverages a data-free masking strategy to focus the knowledge transfer only on relevant classes from appropriately selected teacher models. This selective transfer reduces interference from irrelevant knowledge, enhancing task-specific model performance in a way that standard PFL methods may not support, as they typically rely on local training rather than targeted, query-specific adaptation.

We wanted to express our gratitude once again to the reviewer for their valuable feedback and for raising critical points about our work. We hope that the detailed responses and additional experiments we provided have clarified the motivations behind QKT, its distinction from personalized FL, and the innovations introduced by our approach, including query-specific knowledge transfer, data-free masking, and the two-stage framework.

In response to your comments, we also included additional baselines, such as PerAda and standalone local fine-tuning, as well as expanded results in Appendix A.7. These experiments consistently show that QKT outperforms alternative methods across key metrics, demonstrating its ability to handle query-based objectives and heterogeneous data effectively.

If there are any remaining concerns or further clarifications needed, we’d be happy to address them before the discussion period concludes. We also kindly invite you to reconsider your score in light of the evidence and improvements we’ve presented.

Thank you again for your time and thoughtful feedback.

We sincerely thank the reviewer for their detailed follow-up questions and for engaging with our work. Here are our responses to the questions.

Q1: The authors mentioned that the proposed algorithm is designed to address specific scenarios that traditional FL cannot handle, such as rare or emerging diseases in healthcare and regional fraud patterns in finance. However, I didn’t see a direct evaluation of these scenarios in the experiments. Did I overlook something?

Thank you for your question. While we do not explicitly evaluate rare diseases or fraud patterns, our experiments simulate analogous challenges.

Specifically, we utilize heterogeneous data distributions, including pathological and Dirichlet distributions, to mimic real-world scenarios where certain classes are under-represented or absent locally. These distributions align with challenges in applications like rare diseases or regional fraud patterns, where specific knowledge is unevenly distributed among clients. This experimental setup aligns with prior work in decentralized learning (e.g., McMahan et al., 2017; Fallah et al., 2020) to ensure fair comparisons, as described in Section 4.

Additionally, we included medical benchmarks such as PathMNIST and BloodMNIST to evaluate QKT’s applicability in domains like healthcare. PathMNIST comprises colorectal cancer images, and BloodMNIST features blood cell microscopy images. These datasets represent realistic settings where knowledge transfer can enhance performance on rare or specialized tasks without direct data sharing.

While the current experiments simulate these dynamics, we acknowledge the value of more direct evaluations in such domains. Extending QKT to explicitly target these application areas represents an exciting direction for future work.

---

Q3: Could the authors clarify how the hyperparameters for the baseline methods used in Figure 4 were selected?

Thank you for your question regarding the hyperparameters for the baseline methods in Figure 4. Detailed information about the hyperparameters and training settings is provided in Appendix A.1 (lines 616–635) of the manuscript.

We maintained consistency in the training setup (e.g., optimizer, learning rate) across all methods to ensure fair comparisons. The other hyperparameters used for each baseline generally follow the values recommended in their respective original papers.

---

Q4: In Appendix A.7, which row in the tables corresponds to standalone local fine-tuning?

Thank you for your question regarding standalone local fine-tuning in Appendix A.7.

The Standalone Local Fine-Tuning baseline corresponds to the rows labeled as "Local" in the tables. These rows represent client performance without external knowledge transfer and serve as a reference for evaluating query accuracy gains.

Note that the metrics "Query Accuracy Gain" and "Forgetting" are not applicable to this baseline, as it does not involve any knowledge transfer process.

---

Q7: Do the clients have prior knowledge of the total number of classes $C$?

Thank you for this insightful question. QKT does not require clients to have explicit prior knowledge of the total number of classes across all clients. Instead, during Phase 1, clients infer relevant class information locally by analyzing teacher models’ responses to Gaussian noise inputs, as described in Section 3.3.

It is important to clarify that FL and collaborative learning frameworks typically assume implicit consistency in the number and order of classes across clients. This standard assumption in FL (e.g., McMahan et al., AISTATS 2017) ensures that models trained collaboratively or shared among clients can interpret class labels consistently. Under this assumption, clients know that the class set is consistent globally but do not necessarily need explicit knowledge of how all other clients' data is distributed or labeled.

In the context of QKT, this assumption allows clients to operate within their local knowledge while dynamically inferring class relevance through interactions with teacher models.

That said, we acknowledge that some real-world scenarios may deviate from this assumption—such as tasks where class definitions or orders vary dynamically across clients. Addressing such challenges would involve mechanisms like label mapping or ontology alignment, which represent exciting opportunities for future work.

We hope this clarification addresses your concerns and reinforces QKT’s adherence to established FL practices.

---

Remark

We deeply appreciate the reviewer’s engagement and thoughtful questions, which have significantly enriched our discussions and strengthened the manuscript. If further clarifications are required, we are happy to address them promptly before the discussion period concludes. We kindly invite you to reconsider your score based on the evidence and refinements we have provided.

We sincerely thank the reviewer for their feedback. We're glad that our proposed decentralized collaborative learning problem is appreciated. Here are our responses to the concerns.

1. Some of the claims do not seem well justified

Thank you for this insightful comment. We agree that the goals of QKT and FL differ, and we are happy to reword that sentence toning down the claim. However, we would like to highlight that this sentence is in the context of a general discussion on efficiency considerations and we point out that QKT by design aims to minimize communication volumes.

2. The algorithmic details are not well presented

Thank you for highlighting this point. We are pleased to clarify the algorithmic details regarding teacher model training and the knowledge transfer process in QKT.

Each teacher model is independently pre-trained on its local data, as detailed in Appendix A.1. As outlined in Section 3.3, QKT adopts an approach similar to peer-to-peer federated learning, in which the student client receives the relevant teacher models (the teacher set as defined in line 218) and then applies knowledge transfer locally. This single-round transfer reduces communication overhead and avoids iterative communication rounds.

The KD loss in Equation 4 is computed locally by combining cross-entropy and KL divergence, both calculated on the student’s dataset. The KL divergence term is derived from the filtered teacher predictions, ensuring that only relevant knowledge from each teacher contributes to the KD process. Our data-free masking approach further enhances this relevance by filtering out unnecessary information, keeping the student’s focus task-specific.

We hope this explanation clarifies the training and knowledge transfer mechanism in QKT.

3. how communication rounds were set for other approaches and what synthetic datasets were used

Thank you for raising this point. We clarify these aspects below:

Communication Rounds: For FL baselines such as FedAvg, FedProx, and Moon, we use 100 communication rounds, a common choice in federated learning, aligning with standard practices and allowing for fair comparisons across methods (e.g., MOON, CVPR 2021; FedAPEN, NeurIPS 2023; Cross-Silo Prototypical Calibration, ICML 2023).

Synthetic Dataset for Masking: As outlined in Section 3.3 and examined further in Section 4.3, our masking approach leverages Gaussian noise (we used 20 samples) shaped to match the model’s input dimensions. This synthetic noise enables the identification of relevant classes for knowledge transfer, ensuring that only essential information is utilized by the student model.

We hope this response clarifies our chosen settings and the role of synthetic data in QKT’s masking strategy.

4. methods in more homogeneous data settings

We have experimented with the various levels of heterogeneity in Appendix A2. We further elaborate on this point here. To evaluate the impact of data homogeneity, we conducted experiments across varying levels of heterogeneity, from highly heterogeneous (M=2) to increasingly homogeneous (M=6) settings under the pathological distribution using CIFAR-10. As data becomes more homogeneous (higher M), the performance gap between QKT and other methods decreases. However, QKT consistently outperforms other approaches across all levels of heterogeneity.

5. Impact of Teacher Model Quality

Teacher quality directly affects knowledge transfer. This is addressed in our experiments. In Section 4.3 (Table 3), we show that using all teachers in KD introduces noise, which negatively impacts performance. QKT’s filtering mechanism effectively mitigates this by focusing on relevant teachers, significantly improving the accuracy gain of KD.

While QKT’s data-free filtering criterion prioritizes teachers with knowledge relevant to the query task, its masking approach further mitigates the risks posed by weaker teachers. This allows QKT to extract valuable specialized knowledge even from less reliable teachers while minimizing the impact of irrelevant or noisy contributions.

Additionally, in larger-scale experiments with more clients (Appendix A.7), we compare KD with random participant selection to QKT, which incorporates teacher filtering. The results demonstrate that QKT consistently outperforms by effectively identifying and leveraging relevant teachers, even as the number of clients increases.

These findings highlight the importance of teacher quality and demonstrate QKT’s ability to ensure reliable and efficient knowledge transfer across diverse and large-scale learning settings.

6. Teacher Selection Criterion

QKT currently only employs noise-based filtering to decide whether to select a teacher model or not. Sec 4.3 provides a corroboration for this choice. We will highlight that future work can consider expanding the criteria for teacher selection. For instance, teachers could be scored based on additional factors, such as the similarity between their predictions and the student model’s predictions. High-scoring teachers, reflecting greater task relevance and alignment, could then be prioritized for communication, ensuring scalability, efficiency, and reduced risk of forgetting previously acquired knowledge.

7. practical use case; difference with knowledge distillation from multiple teachers in centralized settings

Thank you for your question. QKT is particularly useful in scenarios where different clients or entities possess specialized knowledge that can be leveraged to enhance the performance of other clients with similar needs, as mentioned in lines 036-039. Practical use cases include healthcare, where hospitals could benefit from sharing expertise on rare or emerging diseases without exposing patient data, and autonomous driving, where one system could learn from diverse geographic and environmental data collected by others.

In contrast to centralized settings, QKT operates in a decentralized manner, transferring knowledge directly from multiple pre-trained peer models to a student model without requiring central aggregation. This design is particularly advantageous in environments where data centralization is impractical or restricted due to privacy regulations.

Furthermore, QKT does not rely on an additional transfer set, which is often required for knowledge distillation. Such datasets may not be readily available and, if shared or centralized, could violate privacy policies. Instead, QKT leverages the student’s own local data for knowledge transfer, ensuring privacy compliance while maintaining effectiveness.

By enabling the integration of specialized knowledge from multiple sources, QKT allows the student model to quickly adapt to specific tasks. This makes QKT a scalable, flexible, and privacy-preserving solution for distributed learning environments.

We sincerely thank the reviewer for their feedback. We're glad that our proposed decentralized collaborative learning approach, its strong motivation, and its superior performance across diverse datasets compared to baselines were appreciated. Here are our responses to the concerns.

1. The authors should discuss the recent development of ensemble learning

Thank you for this thoughtful comment. We agree that ensemble learning shares some conceptual similarities with QKT, particularly in using multiple teacher models to improve the learning process. However, QKT diverges from traditional ensemble learning in several ways:

• In this work, we focus on the problem of transferring knowledge from models trained on heterogeneous distributions, which aligns closely with personalized federated learning. Ensemble learning methods, such as deep ensembles (Lakshminarayanan et al., NeurIPS 2017) and snapshot ensembles (Huang et al., ICLR 2017), typically aggregate outputs from multiple models to improve prediction performance. However, applying such mechanisms to models trained with unique objectives or on heterogeneous data distributions, as seen in federated learning, remains an open challenge (Zhang et al., ICML 2021). Additionally, ensemble methods may still face knowledge inference challenges outlined in Section 3.1.
• In our main experiment, we included a basic ensemble baseline (line 315, table 2) to demonstrate how ensemble methods yield unsatisfactory results (typically, with a large gap) in meeting clients' unique requirements in heterogeneous learning environments.
• Besides, ensemble methods often require multiple forward passes and increased storage during inference, as predictions are combined across models. In contrast, QKT consolidates knowledge into a single student model, reducing both inference time and storage costs.

2. The authors should discuss the pre-training. Moreover, it leads to a significant storage burden if the client does not have sufficient storage/computation resources.

Thank you for raising this important point. Pretraining is indeed an integral part of our setup. As detailed in Appendix A.1, each teacher model undergoes independent pretraining on its local dataset for up to 100 epochs, with early stopping applied if validation performance stagnates.

It is indeed common in the collaborative training setting, as targeted by many federated learning methods, for each client to participate in training and to be able to locally train its own copy of the model on its local dataset. Our work makes similar assumptions regarding the ability of clients to store and train locally.

Moreover, QKT introduces significant improvements in resource efficiency compared to other collaborative learning approaches. Unlike peer-to-peer collaborative learning methods, which typically require multiple rounds of communication and storage of intermediate models, QKT performs knowledge transfer in a single communication round. This minimizes both computational and temporary storage demands at the client level. By consolidating relevant knowledge into a single student model, QKT also reduces the storage burden associated with ensemble methods, which would otherwise require maintaining and evaluating multiple models during inference.

In addition, our data-free filtering approach could be integrated into peer-to-peer federated learning frameworks to further reduce storage and communication overhead. Typically, peer-to-peer personalized frameworks require all-to-all communication to identify suitable collaborator clients for each task. By adopting our data-free filtering approach, instead of transmitting (and temporarily storing) all models to each client—a process that could incur significant storage burdens—clients could instead share only the predictions from Gaussian noise samples. Each client can then determine suitable collaborators by selecting clients with similar tasks based on these predictions, ensuring more efficient, targeted communication with lower storage requirements.

We hope this clarifies our approach and addresses your concerns regarding pretraining and resource efficiency.

We have evaluated FedBoost using both uniform and weighted sampling strategies for pre-trained base predictors under different configurations of communication budgets ($C=5$ and $C=10$) using CIFAR-10 with pathological data distributions for single-class and multi-class queries. These experiments followed the setup detailed in Section 4, with additional baselines and results presented in Appendix A.7.

The table below summarizes the results. Across all configurations, QKT consistently outperforms FedBoost on accuracy, query accuracy gain, and forgetting metrics. Notably, FedBoost’s reliance on multiple communication rounds and global ensemble optimization limits its adaptability to heterogeneous data and client-specific objectives.

We thank the reviewer for suggesting this comparison and hope these results clarify QKT’s advantages in decentralized and heterogeneous learning scenarios.

4. how the performance changes during stages 1 and 2

Thank you for your insightful comment regarding the dynamic performance across the two phases of QKT. We have an ablation study to show the performance of the two stages in QKT (lines 413-418, table 3). We further include an attached plot (Appendix A.5.1 in revised manuscript) that compares the validation accuracy after each epoch during Phase 1 and Phase 2 for a random subset of clients trained on CIFAR-10, alongside the initial local accuracy (orange) before Phase 1. The results show that

• Both Phase 1 and Phase 2 of QKT show significant improvements (more than 80%), far surpassing the local accuracy baseline (less than 45%).
• Phase 2 demonstrates remarkable stability compared to Phase 1, as freezing the feature extractor preserves its learned representations. By focusing exclusively on classification head refinement, Phase 2 achieves smoother and more consistent accuracy improvements, avoiding the fluctuations seen in Phase 1 caused by simultaneous updates to both components.

This two-phase design effectively balances adaptability and stability, addressing the challenges of heterogeneous collaborative learning environments.

5. baselines and the QKT method perform with an increasing amount of clients

Thank you for your request. In response, we expanded our evaluation to include 50 clients using CIFAR-10 under a pathological distribution, building on the experimental setup described in Section 4.1 of the paper. We followed the same evaluation metrics outlined in lines 355–366 and Appendix A.1 to ensure consistency with our main experiments.

We selected the main baselines that achieved competitive results in our earlier experiments. To account for the increased number of clients, we adopted typical participation rates (P%) of 10% (Table 1) and 20% (Table 2), as commonly used in cross-device federated learning. For FL methods, this represents the number of clients selected for each round, while for single-round methods like QKT, it represents the number of clients participating in the single knowledge transfer round. Additionally, we included the "Ensemble (all models)" baseline to assess performance when utilizing all client models simultaneously.

At a larger scale, with 50 clients, QKT consistently outperformed other baselines, achieving higher accuracy, greater query accuracy gain, and lower forgetting rates. It is worth noting that QKT is highly efficient in that it completes knowledge transfer in a single round, significantly reducing communication, storage, and computational demands.

Table1: Performance of QKT and baselines with 50 clients (CIFAR-10, pathological distribution) at a 10% participation rate.

Table2: Performance of QKT and baselines with 50 clients (CIFAR-10, pathological distribution) at a 20% participation rate.

We appreciate this opportunity to demonstrate QKT’s effectiveness in handling larger-scale collaborative learning scenarios.

Thank you for your valuable comments and suggestions. We're glad that you found our approach innovative, particularly our masking strategy, and that our comprehensive experiments across popular benchmarks effectively demonstrated strong improvements. Here is our responses to each point.

1. It is unclear what Figure 2 is showing.

Thank you for your feedback. We appreciate the opportunity to clarify and improve the Figure presentation. The background color inconsistencies were unintended and may have contributed to confusion. We have revised the plot to ensure consistent background colors, enhancing clarity and focus on the decision boundaries.

The improvements in decision boundaries achieved by QKT are evident in the progression of the plots. As mentioned in lines 184-187, QKT Phase 1 introduces more structured and accurate boundaries for the query class compared to Naive KD. Phase 2 further refines these boundaries by focusing on the classification head, resulting in better separation and alignment for both query and local classes. This improvement is supported by the accuracy gains observed for certain classes, as discussed in lines 180–187 and 194.

2. it is not clear why the second stage helps to maintain the performance

Thank you for this insightful question. We hope to clarify that the intuition behind the stage is detailed in section 3.1 (Critical Role of Task-Related Parameters) and Figure 2. Here, we re-explain the intuition. The purpose of our two-stage approach, despite both stages using Equation 4, is fundamentally tied to the targeted refinement in each phase.

In the first stage, we focus on enhancing the feature extractor by transferring relevant query-specific knowledge with masking techniques that filter out irrelevant knowledge. However, this query-focused learning can potentially introduce interference with local class representations.

To address this, we begin Stage 2 by replacing the classification head, which helps to retain the original local knowledge by restoring decision boundaries learned for local classes. Then, with the feature extractor frozen, we fine-tune only the classification head to incorporate the query knowledge. This approach stabilizes the knowledge captured in the feature extractor during Phase 1, allowing us to focus adjustments solely within the head. This selective refinement helps the model retain prior local knowledge while effectively learning the query classes.

This strategy supports robust performance on both query and original classes, and we hope it clarifies the intuition behind our approach.

3. speed and computational costs of QKT compared to the baselines

Thank you for this insightful comment. We would like to clarify that the synthetic dataset generation and query-class identification in QKT are highly computationally efficient. Specifically, only a small number of Gaussian noise samples (20 in our experiments, line 443) are generated, with no additional computation or optimization required. This is followed by a straightforward inference step to collect the probability distribution and detect relevant models for the query task, minimizing both time and computational overhead.

In terms of communication, QKT is designed for a single-round transfer, contrasting with multi-round federated learning approaches that often require iterative updates and communication. We quantify communication by the number of rounds required (line 361), and report this metric in Table 2. This gives QKT an advantage in terms of communication efficiency, particularly when network resources are limited.

Regarding dataset size, we acknowledge the value of experiments on larger datasets to further demonstrate scalability. However, we hope to clarify that our experimental setting closely follows the prior work to ensure fair comparisons (e.g., McMahan et al. (AISTATS 2017), Li et al. (MLSys 2020), Fallah et al. (AAAI 2021), and Deng et al. (ICLR 2022)). It is also a common practice, as we do, to apply various heterogeneous distributions on the datasets we use to show the potential scalability and performance of the algorithm.

We hope this explanation provides clarity on QKT’s computational efficiency and scalability.

We wanted to express our gratitude once again to the reviewer for their valuable feedback and for highlighting key aspects of our work. We hope that the detailed responses and additional experiments we provided have clarified the points you raised, including the revised Figure 2, the intuition behind our two-stage approach, the computational efficiency and scalability of QKT, and the dynamic performance across stages.

Additionally, we expanded our evaluation to include experiments with 50 clients to further demonstrate QKT’s scalability. These results consistently show that QKT achieves superior performance compared to the baselines across various metrics.

If there are any remaining concerns or additional clarifications needed, we’d be more than happy to address them before the discussion period concludes. We also kindly invite you to reconsider your score in light of the additional evidence and improvements we’ve presented.

Thank you for taking the time to review our work and for helping us improve it.