Prompt Public Large Language Models to Synthesize Data for Private On-device Applications

Thanks for your time and valuable feedback!

"the NWP metric should probably be used along some perplexity scores..."

We ran extra experiments to collect the perplexity metrics for the pre-trained LMs in en-US:

Note:

• For fair comparison, the numbers were collected this week (during May 27-29) over 100M+ examples from 200k+ mobile devices. That's why the NWP values are slightly different from Table 3 of our paper. Note that this week overlaps with Memorial Day, which may affect the result.

• While the perplexity scores seem high, they are in the normal range from our previous experience. LLM-mix-166G achieves both the best NWP and perplexity. The perplexity generally aligns with NWP accuracy, though not always, which is also normal from our previous experience. In this paper, we follow [4] and focus on NWP, because it is shown to correlate well with the live metrics in A/B testing (which we also report in Section 4). Besides, reporting NWP accuracy is standard for federated learning LM experiments [1,2,3].

[1] Reddi et al., “Adaptive Federated Optimization”, ICLR 2021

[2] Nguyen et al., “Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning”, ICLR 2023

[3] Li et al., “Private Adaptive Optimization with Side information”, ICML 2022

[4] Xu et al., “Federated learning of gboard language models with differential privacy”, ACL industrial track, 2023

"...the novel contributions might not be enough."

Though prompting LLMs to synthesize data has been explored before, we specifically designed the prompts to mimic user data in mobile virtual keyboards. Besides, we are the first to systematically study this in a production federated learning application and verify it over real-user data from millions of mobile devices. Running large-scale real-world experiments is challenging because of complex system issues and user behaviors.

There are a few interesting and inspiring observations from our production experiments: e.g., LLM synthetic data are closer to the private distribution but there is still a gap (Figure 4); LLM filtered C4 is better than LLM generated chat (Table 3); using a privately-trained LM to filter data only gives a slight gain (Table 4). We hope that our work can inspire more future research in this area.

Thanks for your time and valuable feedback!

1. "Whether ... be achieved by other data enhancement methods."

The space of the LLM prompts (and other data methods) is too large to be explored in a real-world production system. Our key contribution is to show that even a simple idea can work. We are the first to verify this simple idea in a production federated learning application over real-user data from millions of mobile devices. Running large-scale experiments is challenging due to complex system issues and real-user behaviors. We hope that our work can inspire more future research.

2. "...the paper (Wang et al., 2023) are not compared in the experiment."

We did not compare it, for two reasons:

• As pointed out in the Introduction, the two approaches “require non-trivial changes to the current" production pipeline. E.g., knowledge distillation requires changing the on-device LM’s original (word-level) tokenizer to LLM's wordpiece tokenizer, which is difficult from production deployment standpoint. Distribution matching, an enhancement on top of knowledge distillation, adds another layer of complexity (requires careful allocation of privacy budgets).

• Besides, Wang et al., 2023 did not explore the LLM generative abilities, which is the focus of our paper.

2. "...Is the effect of direct mix filter-C4 and convert C4 to chat much worse...?"

Following your advice, we created a new dataset by merging LLM-filtered-C4-136g and LLM-converted-C4-10g, and evaluated the pre-trained LMs:

Note:

• For fair comparison, the numbers were collected this week (May 27-29) from 100M+ examples of 200k+ mobile devices. That's why the first 3 rows are slightly different from Table 3 of our paper. Note that this week overlaps with Memorial Day, which may affect the result.

• The new data (last row) is slightly worse than LLM-mix-166G, indicating the potential of only relying on C4. Due to resource limitations, we only converted ~20% of filtered C4 to chat. If given enough resources, it is interesting to convert all C4 and see how it performs.

• Last, the diversity issue that we learned from directly prompting LLMs (vocab coverage in Table 2) is interesting, and worth sharing with the community.

Thanks for your time and valuable feedback!

1. "...I found this paper...Please discuss this paper..."

Thanks for suggesting this paper. Below we discuss the differences (will add them to our paper):

• Hou et al. (2023) proposed FreD, a method to measure the distance b/t a server-side dataset and the private federated dataset on users’ devices. FreD metric is used for server-side dataset selection.
• By contrast, we use LLMs to synthesize good server-side datasets.
• FreD can be combined with our approach, e.g., use LLMs to get a few datasets (our paper), and use FreD to select a good one. This requires careful allocation of privacy budgets in FreD and federated fine-tuning.

We have discussed a few related work in our paper, e.g., Wang et al., 2023 (see Introduction) and Zhang et al., 2023 (see Section 2).

This is an active research area, and a few new papers appear after our COLM submission, e.g.,

• Hou et al., https://openreview.net/forum?id=SbyqM0cJM0, 2024
• Li et al, https://arxiv.org/abs/2405.14212, 2024

Compared to the new papers, our work provides unique value by running real-world federated learning (FL) experiments.

2. "Not too surprising..."

We agree most of the results match the expectations, and thanks for clarifying that this is not the main factor. We highlight that this is the first time the results are observed in a production FL system: e.g., LLM synthetic data are closer to the private distribution but there is still a gap (Figure 4); LLM filtered C4 is better than LLM generated chat (Table 3); using a privately-trained LM to filter data only gives a slight gain (Table 4). We hope that our work can inspire more future research.

3. "Last section on using privately finetuned LM to filter public data......"

Section 5 presents a preliminary study on whether a privately-trained LM can help reduce the distribution gap between the server-side data and the private user data. The goal here is not training new on-device LMs, instead, the goal is to see if we can use an existing privately-trained LM to obtain a better dataset. The obtained dataset could be of independent interest, e.g., used for server-side research simulations. The private user data is only used in training the on-device LM (so it has the same privacy guarantee as before, see Appendix C.1). Using this LM to filter server-side data does not incur extra privacy cost, following the post-processing property of differential privacy.

Thanks for your time and valuable feedback! Below we address your concerns on the privacy guarantees of our method:

In our paper, the private data are user typing data in the on-device mobile keyboard applications. In Section 3, we design LLM prompts to synthesize data similar to user tying data without access to any private user data, so the manual prompts are public and carefully examined. As also remarked in footnote 1 of our paper: "Pre-trained LLMs are considered to be public because their training data do not contain the on-device user data in mobile keyboard applications. The privacy concerns of LLMs and their training data is an important independent topic (Brown et al., 2022; Tramer et al., 2022)."

In Section 4, the on-device LMs are pre-trained on public data (either C4 dataset or the synthetic data from LLMs), and then fine-tuned on user data with federated learning (FL) and differential privacy (DP). The device/user-level DP guarantees are detailed in Appendix C.1, which are widely used in the literature (McMahan et al. 2018, Kairouz et al. 2021a, Xu et al. 2023). By improving the public pre-training data, we achieve a better privacy-utility trade-off than previous work.

Section 5 is a preliminary study, where we want to see if DP FL fine-tuned LM can be used to improve the server-side data. Note that the goal here is not training new on-device LMs, instead, the goal is to see if we can use an existing privately-trained LM to get a better server-side dataset. The obtained dataset could be of independent interest, e.g., used for server-side research simulations. Specifically, the pipeline that we explore in Section 5 is:

• pre-training on LLM-mix-166g + private fine-tuning via FL -> privately-trained LM (used for on-device inference, same as Section 4) -> use the privately-trained LM to filter LLM-mix-166g to obtain LLM-prox-32g.

This pipeline gives two outputs: 1) privately-trained LM (same LM obtained in Section 4), and 2) a LM filtered server-side data. The private user data is only used in training the private LM. Obtaining the LM filtered server-side data does not incur extra privacy cost, following the post-processing property of differential privacy. Therefore, the pipeline we considered in Section 5 has the same DP guarantee as that given in Section 4 and Appendix C.1.