Language Model Inversion

The paper proposes an estimator based on bisection that can be done with API calls, but this also assumes at least top-1 logit is exposed.

To clarify, we do not assume the top-1 logit is exposed, only the argmax output (which is given in all APIs we know of). Our only assumption is that we can provide a "logit bias" argument. We do not require the API to return logits or log probabilities.

Given the high similarity between [1] and this submission, every nook and cranny of their differences must be disclosed and analyzed to make the submission's contributions clear. I found comparisons lacking with many unanswered questions like: why not use the iterative version, which is shown to be extremely useful in [1]?

We agree that the method here is not novel. We apply ideas from past inversion work including [1] to a different application domain (inverting LM outputs). The contributions are developing a new architecture, a dataset of LM prompts, new baselines, and an algorithm for making this practical to run in an API setting with logit-bias.

We tested the iterative refinement approach in [1] but unfortunately did not find it worked when the input is LM logits. We have added these negative results to the appendix, along with a small experiment explaining why our corrector module might not work recursively (due to a narrow range of BLEU scores in the training distribution).

The bisection-based estimator seems effective, but I find it a bit abrupt and lacking an explanation.

We’ve completely rewritten Section 5 in an effort to make our logit recovery algorithm more clear, and smooth the transition between sections. We plan to release our algorithm as a Python package to allow others to recover full probability vectors from argmax-only API access.

The method does not seem to consider the setting in which the model probabilities are not available at all (i.e., generation-only APIs), which is maybe the most relevant setting for extracting prompts given that the most powerful LLMs are private.

Sorry for the confusion. We do consider this baseline – “Sample Inverter” is a T5 encoder-decoder trained to directly predict a prompt from LLAMA-2 Chat outputs. We have edited the paper to make this more clear in both the theoretical discussion (Section 3.2) and presentation of results (Section 7).

Additionally, the algorithm in Section 5 (originally known as “Distribution Extraction”) is proposed for precisely this reason: it enables logits access to models when APIs only provide argmax directly, by use of the logit bias argument. (As of November 2023, logit bias and argmax output are available in OpenAI, Anthropic, Cohere APIs.)

Thank you for the clarification and continued help. Hopefully the new name makes the purpose of our algorithm clearer to readers nonetheless... Please let us know if there are any improvements you recommend after reading the updated manuscript!

The descriptions of the algorithm and the results... are not very clear. It's hard to fully grasp the methodology and the results without a more detailed explanation.

Thanks for the suggestion. We have renamed the algorithm to “API-Based Logits Extraction”, but are open to suggestions for clearer terminology here. We rewrote the pseudocode and the explanatory text in Section 5.

While the paper introduces a novel concept, the execution in terms of explaining the methodology and results could be enhanced. The paper would benefit from more detailed and clearer descriptions.

Thank you for this recommendation. We’ve completely written Section 5 (the section about stealing logits via API) and changed our description of the method, as well as added more details interspersed throughout the methods and results sections.

Can the authors provide a more detailed explanation of Algorithm 1 and its workings?

We have completely rewritten Section 5 and the pseudocode shown in Algorithm 1. Please indicate any further points of confusion so that we can continue to improve our explanations.

How were the models chosen for the experiments, and can the authors provide a more in-depth description of these models?

We chose the 7B parameter versions of LLAMA-2, and consider both the chat and base checkpoints. We chose LLAMA-2 as it was the among most powerful base language models available at the time of submission. We have clarified this point in experimental design.

For the ablations in the appendix (previously Section 9), we use GPT-2 instead of LLAMA due to computational constraints; these models require ~20x less compute to run.

How do the results in Figure 2 correlate with the methodology described? It would be helpful to have a clearer description or perhaps an example to illustrate the findings.

We rewrote the explanations and captions to better explain Figure 2. In particular, we explain these sampling-as-a-defense experiments differently in Section 8; please let us know if anything is still unclear.

Method is ad-hoc and not theoretically motivated.

We have added a new Subsection 3.1 (“Logits Contain Residual Information”) providing empirical and theoretical motivation why language model logits may reveal information about past tokens.

In terms of the method, we feel we are following standard practice in modern deep learning using Transformers. The main architectural challenge is how to feed in a high-dimensional vector (32,000 numbers) into a transformer. Main contributions were collecting a large dataset of prompts, extracting next-token probability distributions, and demonstrating that inversion is possible and effective in a variety of settings. We are not sure what would make a deep learning approach more theoretically motivated.

The formal methodology is difficult to understand. For example, it’s unclear how the prompt is actually decoded from the embedded output probability distribution, i.e., what happens after their proposed encoding.

We havve tried to clarify this subsection of the paper; please let us know what you still find confusing. The output vector is provided to the encoder of an encoder-decoder language model, where the decoder autoregressively predicts the prompt. This approach is roughly standard in most conditional NLP systems.

I don’t understand what is going on in section 5. What does it mean to “control one logit?”

We consider the most common setting of LLM APIs, where we can provide a “logit bias” argument that upweights or downweights the relative importance of a token in the output. This feature is key to our algorithm. We have completely rewritten Section 5 and Algorithm 1 to be clearer and more contiguous with the rest of the paper, which may alleviate your concern.

In the beginning of section 4 where it’s stated that you “train on samples from the conditional model”, what is “the conditional model”?

This line was simply meant to indicate that we train the inversion model on output from a (conditional) language model. It was not a critical point, and we’ve removed the statement entirely.

Given that the vector v was projected from R^d to R^v (as part of the final linear layer of the generation model), the argument that projecting it back to R^d would lead to a loss of information feels a bit strange

This is a small notational inconsistency. The original dimensionality d is that of the victim language model (4096 for LLAMA) while the final dimensionality is the input embedding of the inverter language model (typically 768 for T5). So in this case, projecting directly from LLAMA to T5 space would lead to a loss of dimensionality with a factor of 4096/768≈5.3.

Where is the theoretical discussion/experiments corresponding to inversion when only the text output is available? The methodology discussed in section 5 still assumes access to probabilities from the model, which are often not available in “models as a service” platforms.

We did consider this baseline, sorry for not making it more clear. For Llama2-chat, we train a model to predict the prompt input given language model output samples . This is shown in Table 1 as “Sample Inverter”. We add an additional paragraph of theoretical discussion in this matter to Section 3.2 (“Inverting from outputs”).

In section 4, it mentions that there is a “fixed-length input sequence of 42 words.” Is this in reference to the sequence of embeddings that are fed to the model? Or is this alluding to how the embeddings are decoded into the expected input?

This is the sequence of embeddings that is fed into the encoder of the inversion model.

How does the ability to reconstruct text change as a function of the pretrained model used to construct the distribution p(x | v)?

We consider this question through many ablations shown in Table 9. Notably, we conclude that we could have seen greatly improved inversion performance if we had trained an inverter with more parameters.

Thanks for the comments and suggestions.

the absolute numbers are fairly low – the exact match scores are discouraging and the BLEU scores are also not very high. So I am not convinced if this is a serious concern for security in terms of prompt leakage.

We view this work as a proof-of concept, to show that prompts do indeed leak through LM outputs, which had not been known before. Bigger models and more training would likely improve absolute numbers. The work, as is, improves upon the best jailbreaking prompts in most settings.

In addition, we include new experiments that significantly improve both exact match and BLEU score for all out-of-distribution settings. Please see the general response for more detail.

A more pointed analysis on the recoverability of sensitive information instead of recovering general purpose prompts might help establish the significance of the proposed attack more clearly.

To answer this question, we created a new dataset of prompts that include private data such as names, birthdays, and nationalities. Our models do a good job at reconstructing categorical national information (80% accuracy at countries and 50% on nationalities) while they struggle with exact numbers such as dates. Clearer analysis and examples are available in the new appendix section "Private Prompt Dataset" and the new Table 11.

Would using token distributions at multiple positions improve the attack?

Inverting a prompt using probability outputs from multiple positions is an interesting suggestion. Since this setting would allow us to model the prompt with strictly more information, it would likely improve inversion performance. Although this is easy to test with local models, our threat model is the one represented by standard APIs: a single API call provides probabilities for the next token at a single position.

Distribution extraction from access to argmax/top-k logits is very expensive.

We agree it has some cost and requires lots of calls, but not that it is "very expensive". Extracting the full logit distribution with argmax-only access requires a number of API calls proportional to the vocabulary size V. For example, in the case of GPT-3.5 Instruct Turbo, using November 2023 pricing, extracting a single prompt can cost as low as $7 (`16 tokens * 3 calls per token * 100k tokens *$.0015 / 1k tokens) = $7.20`). Given the current value of these systems, this cost is not nothing, but it is also not that expensive compared to the engineering cost required to find the most effective prompt for some use cases.