Logits of API-Protected LLMs Leak Proprietary Information

Thank you for your positive review and for noting that our findings affect millions of LLM users and could lead to better transparency and accountability for API providers.

Logit bias could go away

We agree that logit bias may not be a permanent feature, and some APIs do not offer it at all. Our work remains relevant because it deals with common real-world APIs that would likely continue to have this vulnerability otherwise. Alternatively, if logit bias is useful for model auditing, it could become common practice to expose this feature as part of responsible AI deployment.

Limited information leakage, robustness of model signature for accountability

Thank you for this thoughtful discussion! As this a new area of research, it is indeed an open question just how far this type of attack can go. As established in our paper, these attacks may be relatively easy to counter, but they are nonetheless important considerations when building LLM interfaces now and in the future. While we agree that our paper is mostly concerned with details about the LLM final layer, we believe this is an important first step. Indeed, we are working on a followup that gleans information by looking at deeper LLM layers.

Other models

For open-source models, we found it sufficient to use Pythia models as generic stand-ins since there is no theoretical or practical reason the attack would fail with, say, GPT2. For closed-source models, we found GPT-4 turbo to be more expensive to attack, and smaller models would be of less interest. Moreover, the experiments would be considerably more expensive to run now since OpenAI has patched their API.

Post-processing outputs

The watermarking method in [1] would not change these logprobs, since it affects sampling, not logprobs. Changing temperature, so long as the temperature is constant for each query, should not affect the image, since this simply scales the logit vector by a constant.

Routing models, mixture of softmax

We found that, for the same query, the logprobs reported by the API were randomly selected from a set of values, leading us to develop the “stochastic logit extraction” method (Sec. 3.3). Some of the cases you mention are handled by this method. It comes down to whether each query’s output passes through the same softmax matrix. In the case of mixture-of-softmax and other nonlinear output layers, our current algorithm would not work.

Thank you for your typos and suggestions. We will fix/add them.

Thank you for your helpful review and insights.

Comparison with Carlini et al. (2024) [1]

We hope our work will be judged for both our overlapping and unique contributions, and not simply for the additional value, since we released it contemporaneously with [1]. Please note that OpenAI already patched their API when [1] came out, so we could not have conducted this research after [1]’s release. We will outline these more explicitly under a “Contributions” section.

Our overlapping contributions with [1] include

• Algorithm for recovering logits.
• Theory / algorithm for recovering the output layer (the “image”).
• Finding the embed size.

Notably, [1] claims "no immediate practical consequences" of the attack, but we show several (Secs. 4, 6, 7). Our unique contributions include

• Concrete estimate of OpenAI model size (5)
• Fast logit extraction (4)
• Model output attribution, update detection (6)
• List of additional applications (7)

Primary breakthrough

The primary breakthrough of our paper is our algorithm for finding the LM image (overlapping with [1]) and a collection of applications of the model image (not discussed in [1]).

One of these applications is the “fast logit extraction” algorithm we present in Sec. 4, which is our unique contribution. Comparing it to our logit extraction algorithms in Sec. 3 (which are equivalent to those used in [1]) we achieve a speedup from $O(v)$ API calls to $O(d)$ API calls. Note that this new algorithm can only be used after the model image is found, i.e., it is not useful for finding the model image. We compare the complexity of these algorithms in Table 1. We will make these points clearer in the paper.

LoRA updates

Thank you for the pointers for section 7. We agree the LoRA update idea is speculative and will revise the language to make this clearer as potential future work. Our other suggested applications (unargmaxable tokens, basis-aware sampling) are theoretically sound and well developed in their respective papers, since both algorithms take the model image as an input. The proposal in Sec. 7.3, recovering the softmax matrix, is a natural extension of finding the model image (this is also explored in [1], though their “recovered matrix” is equivalent to what we call the “model image”.)

Thank you for getting back to us so quickly and giving more details about your concerns. Here is some additional relevant information.

1. Yesterday COLM organizers clarified the related work policy in an email to reviewers and authors:

Related work concurrency: we follow the NeurIPS policy with regards to concurrency and related work. Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions. Authors are not expected to compare to work that appeared only a month or two before the deadline.

2. Both OpenAI and Cohere (not sure about others) were vulnerable to our method when we wrote the paper. OpenAI changed their API in response to becoming aware of the attack. Cohere's API was patched around the same time, presumably for the same reason. These responses indicate that these companies take our findings seriously. In terms of longer term impact, API providers are now aware of the vulnerabilities going forward and will design their interfaces with this in mind.

Do these additional details address your concerns?

We value your input and hope we can address your concerns:

We agree that the attack is defendable, but assert that exposing it is still valuable because many LLM APIs were vulnerable to it. Otherwise, the unknown vulnerabilities would persist. As described in Sec. 8, the defenses themselves have drawbacks, which is likely why OpenAI’s recent patch kept logit bias.

Outcomes/purpose of attack. Harmful?

Our attack reveals the LM image, which has several applications listed in the Secs. 4-7. We agree (last par., Sec. 9) that the "attack" is not egregiously harmful to LLM providers (though OpenAI likely hides model details for a reason). Providers may opt to keep these "vulnerabilities" to build trust, given the useful, benign capabilities we propose (e.g., detecting model updates.)

Distinction from [1]

Our overlapping contributions with [1] include

• Algorithm for recovering logits.
• Theory / algorithm for recovering the output layer (the “image”).
• Finding the embed size.

Notably, [1] claims "no immediate practical consequences" of the attack, but we show several (Secs. 4, 6, 7). Our unique contributions include

• Concrete estimate of OpenAI model size (5)
• Fast logit extraction (4)
• Model output attribution, update detection (6)
• List of additional applications (7)

We will outline these more explicitly in a “Contributions” section. We hope our work will be judged for both our overlapping and unique contributions, since we released it contemporaneously with [1]. Please note that OpenAI already patched their API when [1] came out, so we could not have conducted this research after [1]’s release.

Recovering last layer

We would like to clarify that our “model image” corresponds to the “last layer” recovered in [1], both of which are linear transformations of the parameters in the last layer. We will make this explicit.

Suggested application (Sec. 7).

We use Sec. 7 as “future work” to show that there are many applications (beyond the paper’s scope) aside from those detailed in Secs. 4-6. We will edit to convey this clearly.

Countermeasure experiments

We cite (first par., Sec. 8) Morris et al. (2023) who already demonstrate logprob-free logit extraction. Other countermeasures block our algorithm, so it is unclear what experiment to run.

Theory, overview of algo.

Thank you, we will more clearly highlight our algorithms in our edits. Do you have specific feedback on how we could make the theory more comprehensive?

We have to admit that the concurrent work [1] is more preferred and mature, because it 1) involves more solid theories, 2) is motivated by a concrete attack goal and corresponding threat model, and 3) is published several days earlier than this submission. I consider this work concurrent to [1] and my decision is independent to 3.

I still do not feel the properties proposed in this work pose a solid threat to LLM, but they are interesting properties to NLP community (probably worth discussing in COLM). I would suggest improving this work to form a solid attack, but considering the timing, I am not against the work to be published to COLM for the awareness of such threats in NLP community.