Hot PATE: Private Aggregation of Distributions for Diverse Tasks

We thank the reviewer for the time and comments.

Question 1 ״Try Hot PATE on real-world datasets to substantiate the practical impact of Hot PATE and improve confidence in its general applicability״

Hot PATE is a mathematically rigorous framework, which we consider to be our main contribution. Hot PATE can only improve over the baseline (cold PATE) in terms of privacy utility tradeoff, and this benefit increases with the diversity.

We include here additional evidence of large potential gains of hot vs cold PATE on natural texts that represent real-world datasets. We are happy to add such evaluations to the paper:

We generated the tokens output distributions of Llama 3.2 1B on all prefixes of the first 500 tokens of a WSJ top page article from Saturday (this is representative, as other news articles and texts we tried gave similar results). The temperature setting was the default $T=1$.

For the purpose of this evaluation, consider a case when all $n$ teachers share the same output distribution. In this case, a coordinated ensemble would yield a histogram where a single token (different one for different shared randomness) has a count of $n$ and remaining tokens have count $0$. The diversity-preservation is perfect as the probability of any one token is equal to its probability value in the distribution. As for cold PATE, we computed the agreement level (minimum fraction of teachers) required for meeting different diversity-preservation levels on a sequence of 10 tokens. These values are inversely related to the privacy parameter $\varepsilon$. Results are averaged over a sliding window of 10 consecutive tokens from the article:

Hot PATE (coordinated ensembles): 100% diversity transfer, privacy parameter $\varepsilon$.

Cold PATE (independent ensembles):

As you can see, for transferring 50% of the distribution, we would incur X$38$ privacy loss with cold PATE. For transferring 95%, the loss is X$5446$. This is very significant. We can extrapolate the same relative gains when the teacher distributions are not identical, as the gains would apply to a common "transferable" part that is supported by enough teachers.

Question 2 ״Can you provide some results on different privacy levels to better understand empirically the diversity-privacy trade-off.״

Yes. There are two regimes:

(1) The teacher distributions are "close" in TV distance. In this regime, using Hot PATE allows us to generate the privacy-preserving tokens (called "yields") for "free". We achieve this using the SVT (sparse-vector technique) with "BetweenThresholds" as there is a high agreement among the teachers. In contrast, in this regime, the baseline "cold" PATE incurs a privacy cost that increases with the diversity, and does not obtain "free yields" as our method.

(2) The teacher distributions are less similar (higher TV distance). In this regime it could be that when sampling a token from each teacher, the resulting samples are in complete "disagreement" and we need to re-sample (so multiple samplings might be needed per generated token). What we showed is that in Hot PATE we essentially need to pay only once per "yield", and we can generate tokens as long as most teachers agree on some small fraction of the distribution (in TV distance). In contrast, in this regime, the baseline "cold" PATE fails completely, in the sense that the probability of it generating any tokens at all is close to zero.

We did include some calculations in the appendix using one particular Laplace-based analysis method. For $\varepsilon=1$, we get $0.005/(2 ln(1/\delta)) * n^2$ tokens for $\delta=10^{-6}$. So this is 180 tokens with 1000 "teachers" (partitions of the data), 18000 tokens with 10K teachers, or 7 tokens for 200 teachers. These tokens are the combined length of synthetic examples or summaries generated from the sensitive data.

Finally, if more utility is needed, then hot PATE also applies with weaker privacy notions that are often used in practice, such as DP with high $\varepsilon$ or $k$-anonymity (as used in Clio by Anthropic): The "boost" in the privacy parameter facilitated by high counts in the histogram translate to a "boost" in $k$.

We thank the reviewer for the constructive feedback and excellent comments and suggestions. We will use them to improve the presentation.

Question: “Besides sequential text generation, what other applications would hot PATE work well for?”

Response:

Hot PATE is suitable for "soft" tasks where the desired output is a sample from a distribution over multiple "possible answers" or "responses". It is a way to aggregate multiple sensitive distributions in order to obtain one or multiple samples that preserve "diversity" and "privacy" (this could also be a single task; not necessarily sequential generation).

Example usage that is not LLM token generation: A game where each sensitive expert suggests a distribution over possible next moves. We want to choose a move that reflects the experts but guards their privacy.

Hot PATE is also suitable for a weaker notion of privacy, a variant of $k$ anonymity, when we only require that the output is “supported” by at least $k$ sensitive units. To achieve this, we simply only return a response that has count at least $k$ without adding noise. With hot PATE there is a much higher likelihood of a token having a large count than with cold PATE.

Weaknesses:

The reviewer pointed out that it is hard to tell how hot PATE would “perform across different applications” and “utility guarantee”.

Since indeed our demonstration was on a particular task, we include here some additional evidence of large potential gains of hot vs cold PATE on natural texts. We are happy to add such evaluations to the paper. We also discuss utility guarantees (see our response to reviewer GoNr).

We considered the token output distributions of Llama 3.2 1B on all prefixes of the first 500 tokens of a WSJ top page article from Saturday (other texts gave similar results). The temperature setting was the default $T=1$.

For the purpose of this evaluation, consider a case when all $n$ teachers share the same output distribution. In this case, a coordinated ensemble would yield a histogram where a single token (different one for different shared randomness) would yield a count of $n$. The diversity-preservation is perfect as the probability of any one token is equal to its probability value in the distribution. As for cold PATE, we computed the agreement level (minimum histogram count) required for meeting different diversity-preservation levels on a sequence of 10 tokens. These values are inversely related to the privacy parameter $\varepsilon$. Results are averaged over a sliding window of 10 consecutive tokens from the article:

Hot PATE (coordinated ensembles): 100% diversity transfer, privacy parameter $\varepsilon$.

Cold PATE (independent ensembles):

As you can see, for transferring 50% of the diversity, we would incur X$38$ privacy loss with cold PATE. For transferring 95%, the loss is X$5446$. This is very significant. We can extrapolate the same relative gains when the teacher distributions are not identical, as the gains would apply to a common "transferable" part that is supported by enough teachers.

We thank the reviewer for the comments and will do our best to improve the presentation.

Question 1:

-- our demo “only counted the diversity”.

Our demo reports on the diversity-privacy tradeoff. Diversity is measured by the number of returnable tokens for a given privacy level (measured by the threshold setting).

-- “grammatically correct”

Our proposed method returns text that is generated by the LLM. So if the LLM generates grammatically correct sentences, then so does our output. This is true in our construction because all of the teachers are consistently applied with the same sanitized prefix (which was privately computed in the previous iterations). So all of the teachers are predicting the next token of the same prefix. This is explained on page 4 (below Figure 4). In our demo, we used the Lamma 3 8B model and text prompts. The prompts were designed to generate a single token response that is a number and the LLM behaved as expected. Sequential text generation repeats such steps.

-- “practical application value”.

Our hot PATE method is designed as a plug-in replacement for cold PATE. It is validated mathematically, always improves the utility-privacy tradeoff, and its benefits increase with the entropy of the generation process.

We include here an additional evidence for the benefits of hot PATE over baseline cold PATE on natural texts. We considered the token output distributions of Llama 3.2 1B on all prefixes of the first 500 tokens of a WSJ top page article from Saturday (other texts gave similar results). We are happy to include such evaluation in our paper.

For the purpose of this evaluation, consider a case when all $n$ teachers share the same output distribution. In this case, a coordinated ensemble would yield a histogram where a single token (different one for different shared randomness) would yield a count of $n$. The diversity-preservation is perfect as the probability of any one token is equal to its probability value in the distribution. As for cold PATE, we computed the agreement level (minimum histogram count) required for meeting different diversity-preservation levels on a sequence of 10 tokens. These values are inversely related to the privacy parameter $\varepsilon$. Results are averaged over a sliding window of 10 consecutive tokens from the article:

Hot PATE (coordinated ensembles): 100% diversity transfer, privacy parameter $\varepsilon$.

Cold PATE (independent ensembles):

As you can see, for transferring 50% of the diversity, we incur X$38$ privacy loss with cold PATE. For transferring 95%, the loss is X$5446$. This is very significant. We can extrapolate the same relative gains when the teacher distributions are not identical, as it applies to the "transferable" part.

Question 2: As we establish mathematically, and demonstrate in our demo, hot PATE (coordinated ensembles) provides high utility regardless of diversity. The tradeoff is not inherent.

Weaknesses:

– "focuses more on theoretical advantages and does not thoroughly explore the feasibility, computational costs, and scalability of the coordination integration method in real-world scenarios"

The benefits of our method are established via a rigorous mathematical analysis and kick-in whenever there is entropy in the responses (which is well established for LLMs). We do, in fact, include a discussion of scalability and computational costs for different API types (Section 4.3). The additional evidence provided above shows that on "real-world" text generation we can expect orders of magnitude benefits over "cold" PATE.

– “repeat the same core idea across scenarios” by describing “homogeneous and heterogeneous” ensembles

These two scenarios warrant separate treatment. Standard PATE is designed for heterogeneous ensembles, which make sense both in diverse and non-diverse settings, whereas homogeneous ensembles are only relevant in diverse settings (and as far as we know were not previously studied with PATE). They require not only different threshold settings but also different private aggregation methods, in order to preserve diversity.

– ״The proposal of "coordination sets" is one of the main contributions of the paper, but it is mixed with other technical details (e.g., privacy analysis methods)״

Our submission is about privacy. It is therefore necessary and central for us to present coordinated ensembles together with their benefit to our privacy analysis.