We thank the reviewer for the insightful comments. Below are our responses.

[1/6] The paper lacks quantitative evaluations of image quality (e.g., FID, IS, CLIP score), making it difficult to establish a direct link between data quality and performance gains.

The data generation in Flick follows two steps: the server first obtains a set of text prompts from LLM, then feeds them into a text-to-image model (i.e., LDM) for synthetic images. We use an out-of-the-box image generation model (e.g., stable-diffusion-v1-5 used in the experiments) without fine-tuning. Therefore, metrics such as FID, IS, or CLIP score, which primarily evaluate the pretrained generator, are not the focus of our evaluation. Instead, as shown in Figure 12 in Appendix C, Flick consistently improves performance across different image generators, highlighting its resilience to the image generative backend.

We clarify that the performance gain mainly lies in the generation of informative, task-relevant text prompts, which is the core contribution of this work. To this end, we propose a prompt template that instructs LLM in extracting information from the collected cross-client-specific knowledge while instilling inherent commonsense knowledge to produce informative text prompts for each class. As shown in Table 2, integrating cross-client-specific knowledge leads to significant gains over local generation. Further, the ablation results in Table 3 confirm that the inclusion of commonsense knowledge improves text prompt effectiveness.

[2/6] The applicability of Flick to non-vision tasks (e.g., text classification, speech, time series) or more complex structured tasks (e.g., object detection, segmentation) is not discussed.

We clarify that the Flick can be extended to other vision tasks with minimal modification, particularly in how to obtain the label $y_s$ of a generated data point $(x_ s, y_ s)$. In the image classification task investigated in this paper, the label $y_ s$ refers to the image class, which is known a priori to the server. When adapting Flick to more complex tasks such as object detection, $y_ s$ would include both the class label and the bounding box of the object. To this end, the server can leverage an automated annotator, such as SAM [1], to generate structured bounding boxes for synthetic images.

We further highlight that the core contribution of Flick lies in its data-driven pipeline, which integrates low-sensitive client-specific knowledge distillation, LLM-based commonsense knowledge instillation, and dual-use of generated samples to address data heterogeneity in federated learning. This pipeline is modality-agnostic and not limited to vision tasks. Nevertheless, we acknowledge two key challenges when applying Flick to non-vision modalities such as text, speech, or time series: 1) identifying suitable methods for converting raw data into meaningful yet low-sensitive text representations (akin to image captioning); and 2) finding the generative models with the capacity of producing realistic synthetic data from text prompts. We leave this as a promising direction for future work.

[1] Alexander Kirillov, et al. "Segment anything". ICCV, 2023.

[3/6] While Flick aims to alleviate data heterogeneity, its mechanism—relying on "client feedback per round → server-generated data → client-side compensation"—may introduce new forms of client imbalance (e.g., favoring clients with stronger descriptive capabilities), due to variations in the amount of feedback, description quality, and generated data quality. No fairness or consistency analysis is provided in this regard.

We highlight two essential fairness-enhancing features built into Flick. First, all clients have an equal opportunity to share their client-specific knowledge with the server. In each communication round, the server randomly selects a set of clients, and each participating client is allowed to report the same number of local summaries -- specifically, one representative local sample per class for image captioning. Second, the quality of generated data used for local compensation is consistent across clients, as the server centrally employs a text-to-image model (i.e., LDM) to synthesize data. This generation process is decoupled from individual client differences and thus not influenced by client diversity.

While variations in description quality, mainly arising from different local captioning models across clients, may still introduce fairness concerns, we mitigate this by enforcing a fixed output length (default: 30 tokens) across all captioning models. This constraint helps limit variations in description quality. Moreover, designing a fairness-aware prompt template could further alleviate the negative impact of local summary disparities on LLM analysis, which we leave as a promising direction for future work.

[4/6] Token length and communication cost statistics are not reported.

For the local summary phase, each client uses a VLP model to selectively caption one representative sample per class. We set the maximum token length of each image caption to 30 tokens, balancing the informativeness of the content with the risk of privacy leakage.

We break down the communication overhead in Flick into three parts: uploading local summaries, downloading synthetic samples for local data compensation, and exchanging model parameters, where the first two components represent additional overhead compared to FedAvg. Taking the PACS experiment in Table 1 as an example, each client reports local summaries with a minimal cost ranging from $1.01kb$ to $1.48kb$ per round, which is negligible compared to the $42.72Mb$ for exchanging model parameters. For a fair comparison, we report the overall communication overhead when Flick and FedAvg reach the target accuracy (defined as the best accuracy of FedAvg). Flick requires only 11 communication rounds with the server sending 510 synthetic images to clients. Thus, the total communication cost of Flick is: $42.72 (Mb/round) \times 11 (round) \times 2 + 0.14 (Mb/sample) \times 510 (sample)=0.99Gb$ where the factor 2 refers to clients offloading the local updates and downloading the global model. For FedAvg, it takes 142 rounds to reach the same target accuracy, and the total communication cost is $42.72 (Mb/round) \times 142 (round) \times 2 =11.85Gb$. Moreover, after 150 rounds, Flick further improves the global model accuracy by 11.43% over FedAvg, by compensating 2,405 synthetic samples at an additional cost of only $0.33Gb$. It indicates that the performance gain by Flick justifies the additional costs.

[5/6] Compared with traditional federated learning methods, the proposed method is very complex, introducing four additional large-scale models and multiple hyperparameters. Therefore, it brings concerns on training burden and optimization difficulty.

We clarify that the image captioning model (i.e., VLP model), LLM, SBERT embedding model, and image generation model (i.e., LDM) are all out-of-the-box models without any fine-tuning or prompt-tuning when applied in our Flick framework. As a result, they introduce zero training burden to Flick. Moreover, the inference overhead introduced by these models is controllable and flexible: 1) three of the four models (excluding the VLP model) are running on the server, which typically has rich computing resources; 2) the performance gain from Flick is consistent to the choice of image captioning model, LLM, and image generation model, as evidenced by the results in Figures 10, 11, and 12 in Appendix C, respectively. This flexibility allows both clients and the server to select models that are best suited to their available computational budgets.

We also investigate the sensitivity of Flick to the key hyperparameters: text similarity $T_ s$, validation accuracy $T_ v$, and data pool size $\lvert \mathcal{G}_ {s}\lvert$ in Figure 9, and data generation budget $G$ in Table 5 of Appendix C. The extensive experimental results show that Flick generalizes well without requiring careful hyperparameter tuning.

[6/6] The authors did not provide any code for review, so the reproducibility of this paper is open to question.

We appreciate the reviewer for pointing it out. Due to the policy prohibiting external links in the rebuttal, we are unable to share the code at this stage. We are committed to releasing the code once anonymity is no longer required.

We thank the reviewer for the insightful comments. Below are our responses.

[1/5] Lack practical applications, and why not leverage powerful LLMs for the task?

Despite the increasing availability of versatile foundation models, we emphasize that real-world deployment, especially inference at the edge, still requires task-specific lightweight models due to privacy constraints and limited resources. This also necessitates privacy-preserving training on data that remains distributed across users, thereby leading to the emergence of FL. On one hand, even if a strong pretrained model (e.g., ResNet-18 trained on ImageNet) is available, it still requires refinement using task-relevant, decentralized data residing on edge devices. On the other hand, while foundation models such as GPT-4o or Qwen2-VL may achieve superior performance on tasks like PACS classification, they are typically too large to run efficiently on edge devices. Relying on remote cloud-based inference via APIs introduces significant privacy risks, as it requires users to upload private or sensitive raw data to third-party services.

[2/5] Authors can provide a breakdown of time and resource consumption involved in the Flick process; API fees associated with using GPT.

Thanks for the suggestion. Below, we provide a detailed breakdown of the additional computational overhead introduced by Flick, which varies across rounds. For example, we report results from an experiment running on an NVIDIA A40 GPU, using the FedProx baseline on PACS dataset. The table below shows the wall-clock time (in seconds) for six components: extra local training, local summary, LLM inference, sample retrieval, data generation, and global model fine-tuning. We observe that the server-side computational overhead (i.e., the last four components) takes the majority of the overall cost. However, server-side overhead decreases over time, as Flick no longer heavily requires data generation after 20 rounds. We also clarify that the data generation time using the LDM is approximately 1 second per sample, which is lower than the reviewer's estimation. This latency can be further reduced with a more powerful server.

For a fair comparison, we report the wall-clock time required for both Flick and FedProx to reach the same target accuracy (i.e., best accuracy by FedProx): 0.48 hours for Flick (at round 19) and 1.24 hours for FedProx (at round 142). The accumulated additional computational overhead by Flick in the first 19 rounds is 0.28 hours, as shown at the bottom of the table. Moreover, after 150 rounds, Flick further improves the global model accuracy by 10.09% over FedProx, with only an additional 0.66 hours cost. These results highlight that the overhead introduced by Flick is well justified by its superior efficiency and accuracy improvements.

For real-world deployment, we also provide the wall-clock time to the target accuracy when local training and summary running on an NVIDIA Jetson AGX Orin: it takes 0.96 hours for Flick while 2.37 hours for FedProx. The accumulated additional computational overhead is 0.62 hours, with the breakdown across the six components being 57.3s, 1220.0s, 291.5s, 51.7s, 598.1s, and 14.1s, respectively. We emphasize the strong flexibility of Flick for inference overhead by allowing clients and the server to choose models according to their available resource budgets. Please see our further response [4/6] to the comment of Reviewer t7ek for the details.

In terms of the API fees, we use the gpt-4o-mini model for local summary analysis. There are a total of 388 API requests across 150 rounds, at a cost of \$0.05, with \\$0.03 for input tokens and \$0.02 for output tokens.

[3/5] Why not fine-tune a global model on all synthetic data at the server? This would reduce communication to just a single round.

We clarify that simply fine-tuning a global model on synthetic data, particularly in a one-shot way, is a suboptimal solution for two main reasons: 1) obtaining a rich task-related synthetic dataset, especially within a single round, is privacy-sensitive and resource-intensive; and 2) the performance gain from global fine-tuning alone, as shown in Tables 1 and 3, is limited.

In Flick, the fine-tuning dataset $\mathcal{G}_s$ is built from scratch and dynamically updated over rounds. In contrast, one-shot fine-tuning requires collecting a rich and substantial dataset within a single round, which leads to privacy risk and significant latency. For example, the baseline method FGL bootstraps all clients to caption all local images and then uploads them to the server-side generator to construct a large-scale fine-tuning dataset in the first round. As shown in Figure 14(a) (Appendix D), this results in ~0.6 hours of warm-up latency. Although the global model achieves high accuracy after the first round, its performance over time is suboptimal. Moreover, collecting large-scale local summaries in a single round raises privacy risks and relies on full client participation, which is often unrealistic in real-world settings.

Moreover, global fine-tuning alone is insufficient. Flick introduces dual usage of synthetic data: the local data compensation progressively provides clients with additional samples to approach the IID condition, yields long-term benefits, while fine-tuning the global model offers a more straightforward yet slight improvement. As shown in Table 3, ablation of local data compensation causes a larger accuracy drop than removing global fine-tuning. Table 1 further supports this: Flick, which combines both local compensation and global fine-tuning, consistently outperforms FGL that is only based on global fine-tuning.

[4/5] Lack of experiments under IID settings, e.g. using datasets like CIFAR.

We clarify Flick is specifically designed for heterogeneous FL scenarios, which are far more realistic scenarios in life than ideal IID settings. In such heterogeneous settings, data silos are non-IID, exhibiting both label skew and domain shift. Such non-IID settings are practical and pose significant challenges to global model performance. In contrast, simple IID settings are idealized and offer limited space for improvement. Flick aims to bridge the gap between realistic non-IID distribution and the IID ideal cases through local data compensation. Besides, we emphasize that Flick explicitly alleviates the domain shift issue, which is not captured in datasets like CIFAR-10 or CIFAR-100. Therefore, evaluation on such datasets cannot reflect or validate the core capabilities and contributions of Flick.

[5/5] Questions and Suggestions.

We have provided in Appendix B.2 the details of the data generation budget $G$, LLMs, and image generators used for the two datasets in Table 1. Specifically, we set $G$=5, use stable-diffusion-v1-5 for image generation, and employ gpt-4o-mini for generating text prompts. We appreciate the suggestion and will move this information from Appendix to the main text in future revisions, to make the paper easier to follow. The table below summarizes the number of samples across three datasets -- PACS (7,984 images, 20 clients), Office-Caltech (2,003 images, 8 clients), and DomainNet (12,740 images, 100 clients) -- used in our experiments. For each baseline, we report the number of generated and training images under Flick over 150 rounds (before /) and up to reaching the target accuracy (after /).

Thank you for your efforts. I have carefully read your response and would like to provide further comments as follows。

Regarding weakness 1:

1. I understand that deploying a model like Qwen2.5-VL may require computation resources. However, in Flick, the client side also needs to host a VLP model, which likewise demands significant resources. Moreover, many recent MLLMs are relatively lightweight and can be deployed efficiently using frameworks such as vLLM or LMDeploy, which help keep resource requirements acceptable. Additionally, the field of knowledge distillation has seen extensive progress, enabling the compression of larger models into much smaller, more efficient versions.
2. The overall cost — including local training, image generation using diffusion models, prompt generation with LLMs, and global fine-tuning — incurs far greater overhead than simply using off-the-shelf LLMs. Since your method already relies on LLMs and LDMs, it would be more convincing to compare your approach with them in a few-shot setting. A more detailed comparison of the required resources and the resulting performance is necessary to support your claims.

Therefore, I strongly suggest making comparisons with MLLMs like Qwen2-VL and Qwen2.5-VL.

Regarding weakness 2:

1. Based on my experience with LLMs and diffusion models, the generation time for LDMs is typically much longer than 1 second per image. Could you clarify how you achieve this inference speed? The specific inference parameters and settings are missing from your paper — can you please provide these details?
2. More importantly, you have not provided any statistics on resource consumption, such as GPU usage. Given that Flick heavily relies on both LLMs and LDMs, its resource requirements are likely to be much higher than those of traditional federated learning. Additional analysis and comparisons are needed to properly justify the feasibility and efficiency of your approach.

Thank you for your detailed response. I have carefully read your replies, and I think you have provided very good explanations that address at least half of my concerns regarding the basic setup. Although I still hold my reservations, I am convinced that using LLMs/LDMs to train small models like ResNet can make sense in certain scenarios, as you pointed out—particularly in highly specific and resource-constrained cases.

Therefore, I have raised my score from 2 to 3. And increase the sub scores for originality and significance.

Additionally, regarding the comparison with few-shot settings using MLLMs such as Qwen: I understand that few-shot models incur more inference costs, but using an off-the-shelf MLLM requires no training at all. I still have concerns here—when I suggested this comparison, I actually meant comparing the performance. While efficiency is indeed difficult to compare (since Flick requires training whereas few-shot Qwen incurs higher inference costs), performance can and should be compared directly. That was the main point of my suggestion.

Lastly, about my concern that “Given that Flick heavily relies on both LLMs and LDMs, its resource requirements are likely to be much higher than those of traditional federated learning”: in your Q5 response, you still did not provide an explicit resource usage comparison with traditional FL methods. I understand that Flick will inevitably require more resources, but they should remain within an acceptable range.

I believe adding these comparisons will further justify the performance-efficiency trade-off of your work.

Thank you again for your efforts in addressing my concerns. If the two comparisons mentioned above can be added, I will make sure to raise my score further in recognition of your efforts.

Thank you for your hard work in addressing my concerns.

The results you provided confirm with my expectations regarding Flick:

1. Compared to using off-the-shelf MLLMs such as Qwen, Flick yields lower performance. I suspect that with carefully selected examples, the performance gap could be even more pronounced.
2. Compared to traditional federated learning training using ResNet, Flick incurs much higher GPU usage.

Despite these inherent limitations and my reservations about the practical value Flick may offer, I am convinced by the authors’ arguments that it presents a trade-off between performance and efficiency and may be valuable in certain scenarios.

I also greatly appreciate the detailed experimental results that directly answer my questions. Therefore, I will raise my score to 4 to acknowledge the authors' efforts.

I strongly recommend including these additional comparisons and statistics in the revised paper to better support and justify the positioning of Flick. Best of luck with your acceptance.

[Response to Q5] An explicit resource usage comparison with traditional FL methods.

In response, we evaluate the resource usage from three aspects: 1) peak GPU memory (GB) during training, 2) GPU memory hours (GB·hour), akin to GPU hours but weighted by runtime memory usage, and 3) energy consumption (kilowatts·hour). We collect these statistics on an NVIDIA A40 GPU for both FedProx and Flick across the three benchmarks. The table below reports the per-round resource usage and the accumulated usage required to reach the target accuracy (i.e., best accuracy of FedProx). Since GPU memory hours and energy consumption vary across communication rounds, we provide both the dynamic range and the average value over all rounds.

For Flick, GPU memory hours and energy consumption show a wider dynamic range because the overhead from key components changes across rounds. As discussed in our previous response [2/5], Flick relies 'heavily' on LLM and LDM for synthetic data generation during the early training stage, which takes the majority of the overall cost. As training progresses, both GPU memory hours and energy consumption per round in Flick decrease. In contrast, the variation in these two per-round measurements for FedProx is mainly due to the randomness of participating clients in each round, where heterogeneous data silos introduce varying local training overhead.

From the results, we observe that for the Office-Caltech dataset, Flick requires more GPU memory hours than FedProx to reach the target accuracy. This is because the additional GPU memory for LLM and LDM far exceeds that required for local training, thereby counteracting the resource savings from faster convergence (i.e., fewer communication rounds). However, Flick still offers shorter training latency to the target and achieves a higher best accuracy than FedProx. For more complex tasks, such as the DomainNet benchmark with large-scale clients and data, Flick outperforms conventional FL across all metrics: it reaches the same performance as counterpart FL frameworks with lower training latency, GPU memory usage, and energy cost; furthermore, it achieves better accuracy in the long run.

To conclude, although Flick introduces additional overhead, primarily on the server, it enables faster convergence of the global model compared to FedProx. In most cases, Flick further reduces resource consumption to achieve the same target accuracy, particularly in challenging settings with large-scale clients and datasets. Furthermore, with continued training, Flick offers substantial accuracy gains; for example, on the PACS dataset, Flick achieves 11.43% higher accuracy than FedProx after the same number of communication rounds.

Dear Reviewer 28YM,

Thank you again for your time, great effort, and swift actions on reviewing our rebuttal and for the insightful discussions. Your constructive feedback has greatly helped us better reflect the novelty and contributions of our Flick. We will definitely add the suggested comparisons and resource usage statistics in the revised manuscript. Once again, we sincerely appreciate your detailed review, constructive suggestions, and timely engagement throughout the process.

Best, Authors

We thank the reviewer for the insightful comments. Below are our responses.

[1/6] Implementation Details: Could you provide more details on the prompt generation process with LLMs? Specifically, how do you ensure that the prompts effectively capture the necessary client-specific knowledge without introducing significant privacy concerns?

In the data generation phase, Flick instructs the LLM to generate a set of class-specific text prompts using a predefined prompt template. These text prompts are then passed to a text-to-image model (e.g., LDM) to synthesize data. The prompt template, shown in Figure 3, takes as input the local summaries $\mathcal{T} ^{n}$, class $i$, and budget $G$ (number of text prompts). Appendix F also provides detailed examples of LLM responses for generating $G=5$ text prompts in the classes dog (Figure 15) and headphones (Figure 16), given the corresponding local captions.

The privacy concerns mainly arise from the local summaries $\mathcal{T} ^{n}$, while these summaries are deliberately designed to contain low-sensitivity textual information. Extensive evaluation in Appendix E supports our privacy claims, and further discussion is provided in the response [3/6]. In a nutshell, $\mathcal{T} ^{n}$ is resistant to image reconstruction and satisfies an $\epsilon$-DP guarantee, making local summary a privacy-preserving design.

[2/6] Real-World Applicability: How do you envision Flick performing in real-world scenarios where data distributions may be more complex and nuanced?

We clarify that our framework design, particularly the calculation of decision matrix $\mathbf{D}$ (lines 200-207 in the paper), naturally enables its resilience to complex and heterogeneous real-world data distributions. Extensive experimental results support such a conclusion, including settings with both extreme label skew and domain shift, which are commonly used to simulate realistic non-IID scenarios.

Specifically, data compensation for each local silo follows the matrix $\mathbf{D}$, where entries are obtained based on the performance of local models on a server-held, task-related validation dataset. Such a dataset is built from scratch and dynamically updated across communication rounds, making the data-driven compensation process fully automated and agnostic to the actual local distribution.

To validate the performance of Flick under heterogeneous data distribution, we report experimental results under the Dirichlet distribution with the concentration parameter $\alpha=0.1$ for PACS dataset and $\alpha=0.05$ for Office-Caltech dataset, where a smaller $\alpha$ indicates stronger heterogeneity. We also evaluate Flick on the large-scale DomainNet dataset with 100 clients with $\alpha=0.1$ (Table 7 in the Appendix). Furthermore, we evaluate Flick under a more extreme case where each client holds samples from only two random classes (please see our response [4/4] to the comment of Reviewer Bcqt for the detailed results). Overall, Flick consistently improves convergence speed and accuracy of the global model by a large margin compared to the baseline methods, especially under more heterogeneous settings.

[3/6] Privacy Considerations: While the paper outlines a low-sensitivity approach to data summarization, could you elaborate on any privacy assessments conducted?

The local summary phase in Flick provides the server with essential client-specific information with minimal privacy risk by 1) selectively captioning only one representative sample per class and 2) using textual summaries that carry informative yet low-sensitivity content. We evaluate the privacy risk of such a data summarization approach from two perspectives: providing a formal $\epsilon$-DP guarantee and investigating the risk of raw data reconstruction from local summaries. As shown in Table 9 (Appendix E), we add Laplace noise (parameterized by $\epsilon$) to the textual summaries and observe that Flick maintains performance gain even under increasing noise levels. Moreover, Table 10 (Appendix E) shows image reconstruction results using a wide range of image generators. We observe that reconstructed images remain significantly dissimilar to the original raw data, as measured by three widely used image similarity metrics. It indicates that recovering the original data from local captions is highly challenging.

[4/6] Computational Resource Requirements: How does the framework perform in environments with limited computational capabilities?

We highlight two features of our Flick that support deployment in real-world scenarios: zero training burden and flexible inference overhead. Specifically, the four models within the framework, including the captioning VLP model, LLM, SBERT embedding model, and LDM model, are all out-of-the-box models without any fine-tuning or prompt-tuning when applied to the Flick framework. As a result, they introduce zero training burden to Flick. In terms of inference overhead, only the VLP model for image captioning runs on the edge side, while the other three models run on the server, which typically has rich computing resources. Moreover, Flick offers strong flexibility in resource requirements through: 1) tunable hyperparameters such as generation budget $G$ and server-held data pool size $\lvert \mathcal{G}_ s \lvert$, and 2) consistent performance gains across different choices of image captioning models, LLMs, and image generators, as shown in Figures 10, 11, and 12 in Appendix C, respectively. This flexibility enables both clients and the server to choose models that are best suited to their available computational budgets.

[5/6] The potential impact of synthetic data on model bias and fairness is not fully explored. Could you provide insights into how Flick addresses these issues, or suggest directions for future work to ensure equitable model performance across diverse client data?

The bias and fairness of synthetic data primarily depend on the text prompts fed into the text-to-image model. In Flick, the server uses a predefined prompt template to instruct LLM in generating these text prompts, as discussed in our response [1/6]. Within the template, the local summaries $\mathcal{T}^{n}$ capture the cross-client-specific knowledge. We emphasize that all participating clients have an equal opportunity to contribute to $\mathcal{T}^{n}$ in Flick, as each client shares the same number of local image captions with the server. Moreover, designing a fairness-aware prompt template could further mitigate the potential bias during text prompt generation and the subsequent data synthesis. We leave this as a promising direction for future work.

[6/6] While innovative, the framework's dependence on LLMs could be seen as an extension of existing generative approaches, rather than a fundamentally new paradigm.

While the availability of versatile generative models has made data synthesis more accessible, effective data generation in the context of FL remains non-trivial, primarily due to privacy and resource constraints. We highlight that Flick is a novel data-driven framework tailored to the challenges of heterogeneous FL. We solve the problem of designing a generative pipeline leveraging limited, low-sensitivity cross-client knowledge and task-relevant commonsense knowledge of LLMs.

Rather than merely stringing together existing generative tools, Flick is carefully designed to address three core challenges: 1) distillation of client-specific knowledge in a low-sensitivity way; 2) prompt design that effectively guides the instillation of task-relevant knowledge from LLMs into the data generation process; 3) usage of the synthetic samples, balancing both effectiveness (long-term improvements in model performance) and efficiency (controllable and flexible overhead). The effectiveness of each key design module is thoroughly validated through the ablation study presented in Table 3.

We thank the reviewer for the insightful comments. Below are our responses.

[1/5] Since federated learning often involves private or domain-specific datasets, off-the-shelf LLMs may lack the relevant commonsense knowledge to accurately represent or augment such data.

Typically, the off-the-shelf LLM (e.g., gpt-4o-mini used in our experiments) contains rich knowledge spanning a wide range of domains. We want to highlight that our proposed pipeline in Flick can effectively query task-relevant information from such an LLM and instill it into the data generation process. Furthermore, in practical deployments, the server in Flick can substitute the off-the-shelf LLM with a task-oriented LLM to better capture domain-relevant commonsense knowledge.

[2/5] The retrieval mechanism seems to introduce duplicate generated samples across different clients or even on the same client. Could you provide quantitative insights on how frequently this duplication occurs in practice? Additionally, how does this duplication impact model performance, either positively (e.g., through regularization) or negatively (e.g., by reducing data diversity)?

There indeed exist some duplicate generated samples that can occur both within a client and across different silos. We define the intra-client duplication as the proportion of duplicated samples within a single client, and inter-client duplication as the proportion of duplicated samples shared across clients. For example, in our PACS experiment using FedAvg as the baseline (Table 1), the intra-client duplication across all clients ranges from 0% to 2.86% (with an average of 1.35%), while the inter-client duplication ranges from 4.89% to 19.30% (with an average of 12.79%).

While such duplication may slightly reduce data diversity, it also brings benefits by reducing generation overhead through sample reuse. This reflects a trade-off between the efficiency and effectiveness of the data generation process. The degree of duplication is controlled by the hyperparameter $T_ s$, which defines the text similarity threshold for data retrieval. As shown in Figure 9(a), a higher $T_ s$ reduces duplication by favoring the generation of new samples over retrieval. We observe consistent performance gain by Flick across $T_ s$ ranging from 0.6 to 0.9, and we set $T_ s=0.8$ as the default in our experiment.

[3/5] The paper mentions that stale sample removal is applied only to the server-side dataset. Would applying a similar strategy to the client-side local datasets also be beneficial? Could you clarify the rationale for restricting this operation to the server dataset, and whether client-side stale samples could similarly degrade learning performance?

We clarify that the update mechanism for the server-side data pool $\mathcal{G}_s$ is tightly tied to its function. Removing stale samples helps prevent overfitting problems and ensures that the dynamic $\mathcal{G}_s$ remains effective both as a validation set for evaluating local updates and a fine-tuning dataset for enhancing the global model. Moreover, this strategy keeps the retrieval and storage overhead constant, as the size of $\mathcal{G}_s$ remains stable across communication rounds.

In contrast, applying such an update mechanism to the client-side local dataset $\mathcal{D}_j$ is not suitable, since $\mathcal{D}_j$ serves as the training dataset where every sample matters. After local compensation in Flick, we expect the local silos to become domain-rich and progressively closer to an IID distribution, making stale sample removal on the client side unnecessary.

[4/5] In line 126, the paper suggests that client datasets progressively approximate an IID distribution over time. However, this assumption may not hold if certain classes inherently present greater learning difficulty. Could you expand on how the proposed method addresses or accounts for such class-specific learning challenges in the non-IID setting?

We clarify that the client in Flick progressively approximate an IID distribution through local compensation, which holds even in the presence of “hard” classes. Specifically, Flick tends to compensate for such hard classes across all participating clients via the decision matrix $\mathbf{D}$ (lines 202-207), since such learning challenges are not client-specific but lead to generally underperforming local models on validation samples of these classes. In such cases, the local data still approximates an IID distribution. We want to highlight the distinction between an IID and a balanced distribution: Flick compensates the local data across clients towards an identical distribution (i.e., $\mathcal{P}_ {j_ 1}(y)\simeq$ $\mathcal{P}_ {j_ 2}(y)$ with $\forall j_ 1, j_ 2 \in \mathcal{J}, j_ 1 \neq j_ 2$), while the resulting label distribution ($\mathcal{P}_ {j_ 1}(y)$ and $\mathcal{P}_ {j_ 2}(y)$) may still be imbalanced.

[5/5] In line 166, the sample selection strategy favors choosing samples with the smallest loss for underperforming classes, under the assumption that such samples are less likely to be noisy. This makes sense in early training stages. However, in later stages, prioritizing harder (i.e., higher-loss) samples might better facilitate learning. Could you clarify why your method consistently favors small-loss samples?

We clarify that our strategy does not consistently favor small-loss samples. Specifically, for well-performing classes, we intentionally select the sample with the largest training loss for image captioning, as such samples provide larger gradients and thus contribute more to model updates. Moreover, the choice between small-loss and large-loss samples varies across communication rounds, even for the same class within the same client, depending on the dynamic relationship between the class-wise loss and the averaged training loss (lines 161-166).

We thank the reviewer for the insightful comments. Below are our responses.

[1/4] The use of local summarization methods, i.e., sample annotation by VLP model, may lead to the loss of key features and insufficient representation of heterogeneity, thereby introducing new issues during the generation process. How can the generated summaries be evaluated?

When generating the local summaries, we aim at an underlying trade-off among informativeness, privacy, and overhead. Specifically, we aim to ensure that the textual summaries are low sensitive, thereby making it difficult for the server to reconstruct the original data (as discussed in Table 10 in Appendix E). Besides, the overhead of image captioning constrains the scale of local samples for summarization. Therefore, in our design, each client follows a loss-based rule to selectively caption one representative data point per class, thus providing the server with effective yet privacy-preserving local summaries at an acceptable cost. The effectiveness of the local summary is also resilient to the choice of the VLP model, as demonstrated by the results in Figure 10 in Appendix C.

While designing a reasonable metric to directly and quantitatively evaluate the quality of the summaries is non-trivial, the ablation results in Table 3 (w/o local summary) and Table 6 (random local summary) show a clear accuracy drop, which strongly shows the effectiveness of our local summarization design.

[2/4] During sample selection, there is a risk of failing to capture intra-class diversity, as well as overfitting or selecting low-quality data. Are there more reasonable selection methods for sample choice?

On one hand, each participating client selects one representative data point per class with minimal captioning overhead. We clarify that the selected sample varies across the communication rounds, even for the same class within the same client, which benefits capturing intra-class diversity in the long run. On the other hand, the loss-based selection strategy ensures that clients choose clean, undistorted samples for underperforming classes and impactful samples for well-performing classes, thereby avoiding captioning low-quality data. The ablation results shown in Table 6, where random sample selection leads to a clearly lower accuracy, further validate the effectiveness of our design.

While methods such as hard-sample selection [1] could potentially improve the sample selection in Flick, directly adopting them is not suitable due to the overhead constraints and the limited, imbalanced nature of data silos in our case. We leave this as a promising direction for future work.

[1] Angelos Katharopoulos and François Fleuret. "Not all samples are created equal: Deep learning with importance sampling", ICML, 2018.

[3/4] More evidence is needed to support privacy claims.

We have already investigated the privacy guarantee of our selective sample captioning mechanism from two perspectives: 1) the formal $\epsilon$-DP guarantee (Table 9 in Appendix E), and 2) the potential risk of reconstructing raw data from the textual captions (Table 10 in Appendix E). We would appreciate it if the reviewer could clarify the additional evidence that should be further considered to support our privacy claims.

[4/4] Experiments under extreme data heterogeneity are lacking.

In response, we conduct additional experiments under a more extreme heterogeneous setting, where each client holds samples from only two random classes on the large-scale DomainNet dataset with 100 clients. We report the results in the following table.

Compared to the Dirichlet distribution with $\alpha=0.1$ (Table 8 in Appendix C.4), we can observe that the accuracy of FedAvg drops from 74.13% to 65.03%, indicating an extreme heterogeneity under this setup. The experimental results show that our Flick consistently outperforms the baseline methods, and the performance gain is actually more significant as the data distribution across clients becomes extremely heterogeneous.