Better Language Model Inversion by Compactly Representing Next-Token Distributions

Thank you for your valuable feedback. We hope we can address your concerns.

1. Though many current LLM APIs do not provide logprobs/logits, some do, and understanding implications of providing logprobs can inform practitioners making decisions about APIs in the future. As an aside, logprobs can still be extracted from APIs that do not provide logprobs if they provide a logit bias parameter (Morris et al., 2024).

2.a (ALR formula) The formula for the alr transform is given on line 80. Using python indexing, alr(p) = log(p[:V]) - log(p[V]).

2.b (How to train inverter) We train the inverter as a standard encoder-decoder model, with the source sequence being the logprob sequence, and the target sequence being the hidden prompt, using cross entropy loss. Section 4 describes how we train our model and additional information is provided in Appendix D. The supplemental material contains the training code.

2.c (Why encoder-decoder?) At a high level, inversion takes a sequence (the LM output), and returns a sequence (the hidden prompt). Since this is a sequence-to-sequence (seq2seq) task over natural language, we believe that encoder-decoder transformers are the most obvious and appropriate architecture to use. This has precedent in previous work on inverting text embeddings (Morris et al. 2023) where they found that the encoder-decoder models significantly outperform decoder-only ones.

3/5. (Related work placement) Thank you for pointing this out. We will consider either moving the related work section, or making it clearer in our introduction what aspects of our work are novel before going into technical details. In doing so we can also resolve your concern about lack of intro for O2P.

4. (Compression requires full logprob outputs?) I believe this is a slight misunderstanding. Our compression only requires access to a number of logprobs equal to the hidden size D of the model (plus one), as opposed to V logprobs for a naïve method. In particular, we choose a list of D tokens plus a reserved token, get the logprobs of those tokens, and subtract the logprob of the reserved token from the D other logprobs. This requires only D+1 logprobs, and you can check that this is exactly the “recovered hidden state” on line 116. For an API that charges per logprob (e.g., gives the logprob of a single token per query), this leads to significant savings versus a method that requires V logprobs per generation step.

5. (Training cost comparison) Our method has a training cost similar to L2T, since the only difference is the shape and size of the model input, the latter of which ranges from smaller to larger than the L2T size, depending on the number of output steps we are training on. Since training efficiency is not the focus of our method and it remains similar to a previous method, we do not believe that an efficiency analysis would be particularly interesting.

7.a (Baselines in Table 4) You are correct that the baselines are different for the different models. As mentioned on line 240, the L2T transfer method is incompatible with out-of-family models, so it cannot be compared for Mistral models. For O2P, we are restricted to the numbers reported in their paper, and they do not report transfer to Llama 2 13B. We chose not to replicate the O2P transfer experiments to obtain these numbers due to computational constraints, as this is a relatively less important part of the paper. We expect O2P to perform at least as well for Llama 2 13B as it would on Mistral, since it was trained on Llama 2 7B. Confirming this would only further strengthen our conclusion from Section 5.5 that O2P is best for model transfer.

7.b (Tables 1 & 2 target models) target models here are Llama 2 7B and Llama 3 8B, as mentioned in Section 4 paragraph 1. We can add this information to the caption as well.

7.c (Intro to baselines) We are confident that our baselines include all important existing language model inversion methods, but we agree that these could be better introduced. We will incorporate an overview of existing methods in the revised related work section.

Questions:

1. (How to get logprobs from OpenAI API) As an illustrative example, observe in the table below how applying a logit bias of 2 causes token 2 to become the most likely token.

If we know that token 1 has logprob -0.5, then it must be the case that token 2 has logprob between -0.5 and -2.5. You can continue to bisect this range until you have token 2’s logprob within a small error tolerance. You can read more about this method in Section 5 of Morris et al. (2024). “Minimum logit bias” refers to the minimum logit bias that you provide to the API to get token 2 to be the most likely token.

2. (GPT-3.5 baseline in Table 3) The goal of Table 3 is to compare O2P to PILS for inverting system prompts. Since the O2P paper only provides results with O2P+GPT-3.5, we use these as our baselines. To help rule out the possibility that different target models explain the performance gap, we include an O2P+Llama2 baseline in the non-finetuning setting, and see that the results are highly similar to O2P+GPT3.5. We will clarify this in the final version of the paper.

3. Prompt (avg.) is the average performance from a set of adversarial prompts designed to get the model to reveal its hidden prompt. Prompt (best) is the performance of the best prompt from this pool. These are obtained from Morris et al. (2024), as mentioned at the end of Section 4; we can make this clearer.

4. (Acceptability of training cost) Training our model takes 3-7 days to complete on four RTX A6000 GPUs, so it is not prohibitively expensive, as detailed inAppendix D. We would like to point out that it is a one-time cost per target model to train an inverter, and it is likely that this cost could be significantly reduced with a bit of engineering effort.

5. (Test set sizes) For Tables 1, 2 and 4, we have used 1000 samples from each dataset, as was done in previous work. We will make sure to add this detail to the final draft. For Table 3, as mentioned in Appendix D.2, we used 103 samples for Awesome Dataset and 29 for Store Dataset, as was done in previous work.

Thank you for your valuable feedback and positive evaluation. Your input will help us to improve our paper.

(Confusion about logprobs): To simplify the analysis in section 2 and the start of section 3, we assume access to all V logprobs for each output. Line 127 makes the observation that we don’t actually need all of these logprobs to calculate $\text{alr}(p)_{1:D}=(\log p_1-\log p_V, \log p_2-\log p_V,\ldots,\log p_D-\log p_V)$---we only need the first D logprobs and $\log p_V$.

(Prompt Length) We limited the inversion of prompts to 64 tokens during both training and testing, consistent with L2T. While our qualitative examples (Appendix E) show prompts of varying lengths, we agree that a systematic analysis of how prompt length affects inversion performance would be interesting. We will consider adding such an analysis to our final version.

(Missing Metrics): Your assumption is correct that these scores are too low to be informative. Exact match scores for system prompt recovery (Table 3) are indeed very low (near 0%), and similarly, exact match and BLEU scores for model transfer (Table 4) are substantially lower than the reported Token F1 scores. Additionally, L2T does not report BLEU and exact match scores for their transfer experiments. We are happy to include these numbers for completeness.

(Missing Baselines): For Table 3 (system prompt recovery), the L2T did not conduct any system prompt inversion experiments, so we only compare against O2P. Since O2P generally outperforms L2T, we did not deem it necessary to run these experiments ourselves. For Table 4 (transfer performance), the baseline coverage reflects methodological constraints:

• L2T's model transfer only works for models with the same tokenizer (we report results for Llama 2 13B which shares a tokenizer with Llama 2 7B, but cannot transfer to Mistral 7B)
• O2P did not report numbers for Llama 2 13B in their original work, so we only use their reported Mistral 7B results. We do not believe that adding this would be particularly informative or change our conclusion that O2P outperforms our method in the transfer setting.

Thank you for your positive review and valuable feedback. We will make sure to improve the paper accordingly. We will address your questions and concerns as best we can below

(Grid plots and figure 3) In figure 3, the target model receives the prompt (xtick labels) and generates an output (ytick labels) one token at a time. At each generation step, we feed the outputs so far to the inverter, and force it to generate the hidden prompt, measuring the probability of each prompt token according to the inverter. These probabilities are a single row of the grid. The rows with the most high probability tokens (mostly gray) are the ones where the model is closest to correctly guessing the hidden prompt. Token is “recoverable” when its probability is high. Filled red grid cells mean that the target model had to generate the hidden token before the inverter was able to guess what it was. We will try to make these figures easier to parse in the final draft.

(Choice of model) We agree that Llama 2 is a weaker model, so we also provide Llama 3 results in Tables 1 and 2 for the benefit of future work. We used Llama 2 models in order to make our results comparable to previous work without replicating all of their experiments on Llama 3. We will make sure that this is explicit in the final draft.

(Intuition for why our method works) You are correct in that logprob-based inversion can recover the prompt even when it does not appear in the text outputs of the model. Line 227 is only trying to convey that inversion success appears to be correlated with how likely the model is to generate the secret prompt, even for logprob based methods. As to the mechanism for how our method excels, compared to text-based inversion, logprob-based inversion accesses richer information about the prompt directly from the model’s internal representation. Compared to the previous logprob-based inversion, our method gives the inverter access to information that only surfaces later in the generation process. We are happy to discuss more about these intuitions.

(Why do model providers allow logprob access) We are not sure about all the reasons, but we presume that logprobs and logit bias are useful to API users for one reason or another (e.g., fine-grained control of generations), and so removing these options from the API would cost providers in some way. It may be that these costs outweigh the risks of providing logprobs, at least for now.

Thank you for your positive score and valuable feedback. We appreciate your comments and will use them to improve our paper. Regarding the listed weaknesses, we hope you agree that poor model transfer is a limitation of the method but not a weakness of the paper, since an honest evaluation of the limitations should be part of any analysis of a new method. See the answer below to question 2 for more details on our transfer experiments. Similarly, we believe that our answer below to your question 1 addresses the second listed weakness.

Questions:

1. (ALR implementation) In our preliminary experiments, we observed that tokens whose corresponding unembedding submatrices are low-rank consistently underperformed compared to those with full-rank submatrices. This makes sense because linearly dependent tokens provide redundant information about the hidden state. Empirically, we find that the first D tokens tend to be low rank (this may depend on the model), but a random choice of D+100 tokens usually has high rank, and therefore performs well. We do not observe sensitivity to the choice of random tokens, and choosing random tokens is much easier than searching for a full rank submatrix. We also explored using other transforms like CLR, but observed no significant performance improvements over our proposed approach. We adopted ALR primarily due to its efficiency advantages: while CLR requires computing the mean over the entire logprob vector (necessitating access to all vocabulary tokens), ALR operates with only D+101 logprobs, which is substantially smaller. This design choice significantly reduces API costs (as measured in logprobs) without any impact on performance.

2. (Model Transfer) We experimented with several variations on our logprob alignment technique and did not find any improvements, though this does not rule out the possibility that other techniques like CKA might work. We report these findings despite their limitations because our method is the first ever cross-family logprob-based inverter transfer method, and could precipitate better results in future work. We speculate in the paper (line 257) that transfer suffers because the inverter “overfits” to model-specific features from the source model that are not present in logprobs from other models.

3. (Positional embedding generalization ablation) This would be an interesting experiment to run, though the rebuttal period is not long enough to accommodate it. We will consider adding this for the camera ready. Since this claim is speculation, and not central to the findings of the paper, we can consider removing it, though we think that it is an interesting avenue for future work.

4. (Practical Cost): Great idea, we will add this to the paper. Using some back-of-the-envelope calculations, let’s calculate the cost for a T-step attack on an API model with hidden size D, max logit bias B. We need $T\times D$ logprobs, each logprob requires $\log(B/\epsilon)$ queries. Each query has up to $T$ input tokens and 1 output token. Suppose the input cost is $C_\text{in}$ per token and the output token cost is $C_\text{out}$ per token. So the cost will be $\sum_{i=0}^{T-1}D(i\times C_\text{in}+C_\text{out})\log(B/\epsilon).$ If we are using an GPT-4.1 mini (one size up from the cheapest model GPT-4.1 nano), this model has an unknown hidden size D, but perhaps we assume it is similar to GPT-3.5-Turbo which is less than 4600. Max logit bias is 100, cost is $0.40/1M tokens and cost per output is$1.60/1M tokens, and we will ignore caching, though that could reduce cost further. We will require logprobs with precision of 0.001. This gives us $\sum_{i=0}^{15}4600\times(i\times\frac{0.1}{1000000}+\frac{0.4}{1000000})\log_2(100/0.001)=\$5.50$. For GPT-4.1 nano, this cost is $1.38. Using the batch API, both of these costs can be halved. This cost per prompt is relatively low, and creating a small dataset to fine-tune an inverter model to target GPT-4.1 would be feasible.