Hot PATE: Private Aggregation of Distributions for Diverse Tasks

We appreciate the feedback and interest. Our revised version has improved readability. Here we respond specifically to the "weaknesses" (which is where the questions are):

The primary application of our privacy and diversity-preserving aggregation method is the Hot PATE setting. The case when teachers have output distributions arise naturally with prompts and token generation so this is a very compelling application. Hot PATE broadly applies when (1) models have an output distribution and (2) it is important to transfer the diversity (e.g. the diversity is not "noise" on a ground truth but the signal itself). It does not have to be prompts.

Thank you for pointing out B.1 in [1]. Relation to our work: It is not directly relevant except for the general message that transferring diversity is helpful for student learning (in the setting of distillation). What B.1 in [1] say is that with cold PATE, transferring more of the teacher vote histogram (distribution) rather than just the argmax helps with accuracy. They also say there is additional privacy loss of reporting more and it may not be worth the gain. What we consider is the case where each teacher has a distribution and it is valuable to transfer this diversity to the student. An application of this setting is generation by LLMs. Limiting LLMs to say argmax takes away from their value. Our technical contribution is a method that preserves the diversity without incurring privacy loss over the cold PATE baseline.

The "questions" seem more about writing style and readability and we will address them in the revision. Thank you!

Thank you for the review and the references! Indeed, it seems that unfortunately there is confusion about understanding our contribution. We attempt to clarify by addressing the listed "weaknesses".

W1. We do not "build directly" on Duan et al 2023. Our technique addresses what we view as a significant deficiency in the design of Duan et al -- not effectively working with model diversity. Our technique is general, but we found the use case of learning via prompts (which Duan et al considered) very compelling.

The core of our contribution is formalizing a notion of a diversity and privacy-preserving aggregate distribution. We give a method where privacy loss does not increase with diversity that is critically important in diverse settings.

Our description of Duan et al is indeed simplified. We focused on what we deem relevant to make the points. The point is that the Duan et al approach did use unlabeled training examples for knowledge transfer. The reviewer points out that a non iid method was also used. But this detail is not very relevant to us. The point is that the mechanism for knowledge transfer is through unlabeled examples whereas with our approach, the diversity (and its preservation) allows for compelling alternatives (simply transfer the distribution). As for selecting a token among multiple: In Duan et al it is still the case that each teacher provides a vote for a token and these votes are privately aggregated. With increased diversity, there will not be a dominant vote and we incur higher privacy loss to identify a lower maximum. This is exactly the issue our method corrects. Privacy loss does not increase with diversity!

W2. Indeed the main technical contribution is a method for diversity-preserving private aggregation of distributions. We view learning via promps as a very compelling use case. This use case motivated our design.

W3. We simplified to focus on what is relevant. Indeed, "PATE" is broadly used for any sample-and-aggregate approach to DP machine learning and there are many works with this term. The reviewer pointed to a GAN method called G-PATE. This is not relevant to the way our approach extends on the original PATE or prompt-PATE. G-PATE can be viewed as a different "branch" in the sample-and-aggregate private machine learning literature-- use private aggregation of gradients from teacher "discriminators" to train a student "generator." G-PATE and the related works do not address the issue we were out to address. We will explain this in a "related work" paragraph in the revised version.

Thank you for your quick response. In general, I still think that this paper requires a substantial amount of work and is definitely not ready to be accepted. The main problems are:

1. Poor writing and the lack of statement about what the main contribution is.
2. The experimental results should compare to the PATE method in general and not only to the PromptPATE from Duan et al. (Duan et al. addressed privacy protection from all types of prompts using both PATE and DP-SGD).
3. The scalability of the proposed solution may prove impractical for tasks of greater complexity.

• W1. "Our technique addresses what we view as a significant deficiency in the design of Duan et al -- not effectively working with model diversity." - Duan et al. do not consider models but model adaptations.
• W2. This needs to be clearly stated and then the relevant experimental design is required. The current one is too narrow.
• W3. In my review, I pointed out the lack of sufficient related work in this submission and incorrect statements.

I also read through the other reviews and fully agree with them. I maintain my score.

A revised version is now uploaded. We thank the reviewer for the comments and tried to address to the extent we could what seemed to lead to some misunderstandings by the reviewer. We also included a "related work" section that better explains the relation to Duane et al (which did not address the core problem that we address) and includes other references and relation to our work.

Regarding the points made: Problem (1) (lack of statement of what the main contribution is) is very subjective. Other reviewers seemed to understand it at a high level. We did try to improve this based on all comments in the revision.

We disagree with problem (3) "the scalability of the proposed solution...." This is in fact the whole point of coordinated ensembles. That we do not "pay" for increased diversity. It is established analytically (proofs) and there is really no actual need for an experimental design. Experiments are needed in the event when there is hidden complexity in the practical setting that is not captured in the abstraction. In this case, the analysis makes the point well. The small demonstration (that had limited diversity due to limitations of the API) shows the gains over a "naive" approach. The naive approach is exactly what you would get if you apply Duane et al for the aggregation step. There is sharp drop in utility with diversity.

We disagree with problem (2). We do not think a comparison with DP-SGD fine tuning is relevant here. It is true that Duane et al also considered DP-SGD fine-tuning (that is, not using prompts). We are not touching that space. We only consider PATE-like settings with prompts as a motivation. We disagree with the characterization that it is "too narrow". In-context learning via prompts is very promising and effective and we feel it is important to understand how to "do it right" with privacy. Our approach is the way to do it. It address the major specific technical hurdle. Orthogonal issues are domain-specific prompt engineering and potential light-wait enhancements to the API (which we propose).

We appreciate the feedback and interest.

Response to "weaknesses":

W2 [empirical validation of student model] We propose a meta scheme and demonstrate by running the numbers that large privacy-preserving students prompts can be generated effectively. Large scale applications would require orthogonal components such as datasets and tasks, prompt engineering, and more. This is beyond the scope of this work. Our work can be viewed as proposing a method that SHOULD be used IF applications arise in which PATE is applied with diverse tasks. This is established with theory and simulations. The method itself might have other applications.

W3 We cited the related work we are aware of that seemed most relevant. There is significant number of works on PATE, Prompts, LLMs, out there. If the reviewer is aware of relevant works that we had missed we would appreciate sharing that.

W4 The revised version would include additional details.

Response to questions:

Q1: The notation w_j are the model weights that are transformed to probabilities using softmax. This is very standard and the purpose was just to explain the process in which diversity is introducted and tuned in models. Our method just assumes there is an output distribution but applies regardless of how it is computed. Q2:The demo is indeed limited by the interface of OpenAI GPY 3.5 which is beyond our control. It only provides the top 5 probabilities. We would have larger gain over baseline with more diversity, and this is demonstrated in the simulations. We used this interface because this is what is available. Regardless, the purpose of the demo was to demonstrate that the models are indeed diverse. Importantly, we included simulations in Section 7 that cover much more general problem parameters.

Q3: The private information is the dataset that is used to generate the teacher prompts. The dataset can be in the form of examples (e.g. medical records) that can be partitioned to teachers.

Q4: The student model is a new prompt that is privacy preserving. It can be a set of example "shots" or have another form.

We appreciate the feedback and interest! We agree that the experiments were fairly limited scope but we believe that the theory, together with simulations and the (limited) experiments, demonstrated the value of the method. We expect that our method is the way to apply PATE with diverse models and any effective method in future should use this approach. The method for private diversity-preserving aggregation may have additional applications as well.

W1, W2, W3 The demo was indeed limited by the GPT3.5 interface which only returns the top-5 probabilities. The purpose of the demo was meant to show that real models behave in the way we expect and when tasked with diverse tasks, indeed have diverse responses. Importantly, we demonstrated through the theory and also simulations that the more diverse the distributions are, the larger the gain from our approach over the prior naive methods of dealing with diversity. Moreover, with our method there is no utility loss with diversity. Utility only depends on the extent of agreement between teacher distributions, not on the "entropy" of the agreement components. This follows from the theory and demonstrated with simulations (Section 7). So as long as you believe the premise that models tend to cover the diversity of the answer space to a query, you need to see the value of the method.

W2, W3 The use case of prompts is interesting in that it allows for a very large number of teachers, when we have enough data,compared with traditional PATE (where each teacher is a trained model). We included a numerical analysis of privacy cost per yield (token) in some regimes and number of yields dependence on number of teachers. The number of yield queries increase quadratically with the number of teachers. For example, if we have 1M sensitive examples, we get 10 examples for each of 100k teachers. This can allow for large (10^5+ student prompts). Indeed, we do not have a concrete evaluation on a particular task and particular dataset, but we did run the numbers and they demonstrate the feasability.

W4 we will expand on this (discuss regimes and guidelines) in the revised version.

Response to questions:

Q1 [Performance on fully open-ended queries] This submission is a proof of concept. We propose a method, and demonstrate its potential with theory and simulations and running the numbers. Actual implementation requires orthogonal components including datasets and prompt engineering that might be tailored to an application. When we have access to a large model beyond public interfaces or when publicly available interfaces provide more of the token distribution we can extend the experiments. As said, the beauty of our method is that there is no cost for diversity. This follows from the theory.

Q2 [Quality of output] The revised version (will be uploaded soon) will include more discussion to that end to address questions we envision might come up in some applications.

Q3 [Scalability/privacy] See Section 7 and response to W2,W3. We did run the numbers. As for impact on privacy. We use DP, with mathematical guarantees and run the numbers with overall epsilon<1. We have in mind a use case of highly sensitive data.