Online Detection for Black-Box Large Language Models with Adaptive Prompt Selection

Dear reviewer EvAs,

We thank the reviewer for the comments, and we appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided.

Q1: Clarification on the calculation for NTE and NTG

A1: Due to the exponential growth of the total combinations of N tokens with regard to N, we designed an approximation method (see Section 3.2 in our paper). At each time step, we collect C responses for prompt x, each with length N. After tokenizing these C responses, we will obtain at most CN tokens. We merge all of these CN tokens into a set and count the frequency of each token in this set to get an empirical distribution. Then both NTE and NTG metrics are computed using this empirical distribution acquired from the C*N tokens.

Q2: Regarding different decoding strategies

A2: Different decoding strategies may affect the distribution of the generated response, then for two different models, the detection task will be easier if the detection strategy is adopted such that difference in the resulting pre-change and post-change response distributions are maximized. However, we would like to emphasize again that the decoding strategy is not the focus of our work and we treat the decoding strategy as a pre-determined component of the black-box LLM models.

Q3: Regarding the performance on larger LLMs

A3: We assessed the performance of our detection algorithm in relatively large LLM, such as version update from vicuna-7b-v1.1 to vicuna-7b-v1.3 (see Figure 4b), from MiniChat-3B to MiniChat-1.5-3B (see Figure 4c), and all real-world APIs (the parameter size of the LLMs behind them is often in trillions, see Subsection 4.2). Moreover, we added a new experiment under the case of prompt injection and used GPT-4o API for an instance (see Appendix E.4). All these experiments are conducted with large-size LLMs, and the effectiveness and robustness of our detection algorithm are validated.

Q4: Regarding more real-world changes

A4: We collected data of 9 various LLM APIs, as specified in Section 4.2 in our paper (they are: gpt-4o, gpt-4, gpt-4-turbo, gpt-3.5-turbo, Cohere command-r-plus, claude-3-haiku-20240307, mistral-large-latest, jamba-instruct and j2-ultra). We collected data from these nine APIs on a daily basis from June 01 to August 31. With the collected data, we successfully detected one confirmed change and two unconfirmed changes using our detection algorithm. With regard to gpt-4 and gpt-4o that you have mentioned, we indeed did the same study but there is no evidence for a change happening during this period. Yet the success of our algorithm on other real world cases have proved the effectiveness of our detection framework.

Q5: Regarding the choice of parameter C and N

A5: We would like to clarify that C and N is a relatively flexible hyperparameter during real world usages, and the key to choose C and N configuration is to balance the performance of the algorithm under this config and the cost of the algorithm, as we mentioned in Appendix C paragraph 3. It is easy to expect that the increase in C and N will enhance the algorithm's detection power. As a result, in real world usage, what we need to do is to set a larger C and N within our affordable computation cost. And for the effectiveness evaluation of the detection algorithm in our research, we just choose a hyperparameter configuration that almost reveals real performance of our algorithm and meanwhile can be affordable to most users so that they could also use it.

Thank you again for your comments on our paper, which have guided us toward a more refined revision. We sincerely hope that we have addressed your concerns. If we have misunderstood your point, we welcome any further clarification of your concerns and hope that you may consider revising your evaluation of our work.

Dear reviewer wJQu,

We thank the reviewer for the comments, and we appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided.

Q1: Regarding the level of information and model access

A1: We focus on the black-box setting where log probabilities cannot be obtained. In fact, the top-K probabilities are not available in many cases, such as Mistral, Cohere, Claude covered in our experiments, and o1-preview most recently. Thus we focus on the black-box and generalized scenario, where the log probability is not available, or even if it is available, the user may not necessarily interact with the LLM via an API and thus only the response is available for change detection.

Q2: Regarding the concern that model update is publicly available information

A2: LLM API providers often release their new version of API with a public announcement. However, this is not always the case. In fact ,many “minor” updates are released without public notification. We reference this paper to support the relevance of detecting changes even in the absence of unauthorized parameter updates. In this paper, the authors showed an interesting fact that even in an LLM API with no official notification of version update, its performance can still vary across different times. And in our paper, we also detected changes in Jamba and GPT-4-turbo respectively, despite there being no announcement on their provider’s website. We believe this discovery underscores the value of our work.

Q3: Regarding the call for more change instances

A3: We have provided two synthetic cases and one real-world case in our paper, and each of these cases include multiple instances (such as different model versions, different watermark keys etc.). For further evaluation, we now have also added a case of prompt injection, on which our detection algorithm also has strong performance (see Appendix E.4 in the revised draft). All of the experiments under these settings are conducted with at least 3 prompts and repeated for 5 times. We also provided the delay-ARL analysis (hope this is what you mean by “True Positive Rate @ False Positive Rate”) for the case of watermark (in Figure 11) and the case of version change from opt-125m to opt-350m (in Figure 6b). For other cases where we didn’t provide the delay-ARL analysis, we provided their trajectory of detection statistics. In these trajectories, we see a surge in detection statistics after the change happens.

Q4: Regarding baseline

A4: We have implemented the “text similarity” baseline. The idea is that we convert the texts at each time step and also of history to a frequency vector, and compute the cosine similarity of the frequency vector at each time step and that of the history. When the similarity is below a threshold (selected to meet some false alarm rate constraint), the detection algorithm raises an alarm. We present the trajectory of different detection statistics on the same sequence in the newly added Figure 20, and in Appendix E.1 of the revised version, you will find that our proposed method has a much shorter delay (detect more quickly). We also perform the experiment multiple times and plot the average delay v.s. ARL (false alarm rate metric). All results can be found in the newly added Figure 19. You will see that in most cases, our cumulative type of detection statistics outperform the text similarity baseline.

Thank you once again for your comments on our paper, which have helped guide us toward a more thorough revision. We sincerely hope that we have addressed your concerns and that you may consider revising your evaluation of our work.

Dear reviewer TL1q,

We thank the reviewer for the comments, and we appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided.

Q1: Regarding the detection setting

A1: Thank you for your detailed comments on various API change scenarios. Indeed, our goal is to detect model changes from the user's perspective when interacting with a black-box LLM. And we would like to clarify that our primary focus is on designing an effective and computationally efficient online detection framework that users can flexibly apply to detect changes, rather than identifying specific types of changes or diagnosing their root causes. By developing an algorithm capable of raising alerts when any model change is detected, we aim to provide users with timely insights that may assist them in relying on LLM responses for certain tasks. While root cause diagnosis (determining the type of change) could be an interesting direction for future research, it is complex even in white-box or traditional scenarios, and especially so in black-box LLMs with countless potential causes. For instance, while we use watermarking to simulate a decoding change, our aim is not to diagnose its cause after a successful detection. Additionally, we reference this paper to support the relevance of detecting changes even in the absence of unauthorized parameter updates. In this paper, the authors showed an interesting fact that even in an LLM API with no official notification of version update, its performance can still vary across different times.

Q2: Regarding prompt injection

A2: We added new experiments on prompt injection. We conduct the prompt injection through changing the system message of GPT-4o API, and we validated the effectiveness of our algorithm in this scenario. For details, please see the newly added Figure 22 and Appendix E.4.

Q3: Regarding the necessity of online detection algorithms

A3: We would like to clarify that the example shown in Fig. 1 is purely illustrative and does not represent the actual responses we are working with. In fact, developing online change detection methods is a long-standing challenge, as many distributional changes cannot be detected by examining individual data instances alone. The simplest example is that data observations from N(0,1) and N(0.01,1) are sufficiently close and cannot be distinguished by individual points, yet cumulative sum type detection methods can effectively identify such changes quickly. Similarly, in our LLM detection setting, below we provide a side-by-side comparison of the real responses collected in our experiments, generated by different LLM models:

Prompt: The sum of 1 and 4 is:

As shown in the example above, the API change is not immediately apparent from reading the responses, as they appear quite similar. More examples like this are provided in Appendix E.2. This underscores the necessity of developing online detection methods tailored to LLM models.

Q4: Clarification on experimental settings

A4: Temperature is non-zero throughout all our experiments. In synthetic cases, we set temperature to default value 1.0, and in real-world APIs we set temperature to 1.5 for most APIs and 1.0 for Jamba and Cohere. We have added clarifications on experimental details in Appendix C paragraph 1.

To be continued ...

import openai

openai.api_key = "YOUR_API_KEY"

completion = openai.ChatCompletion.create(
    model="gpt-4-turbo-2024-04-09",
    messages=[
        {"role": "user", "content": "Write the 8 queens problem in Python."}
    ],
    temperature=0
)

print(completion.choices[0].message.content)

Dear reviewer niaS,

We thank the reviewer for the comments, and we appreciate the time you spent on the paper. Below we address the concerns and comments that you have provided.

Q1: Regarding generalizability

A1: We have added two additional experiments in the revised version: (1) Detecting prompt injection: Based on the suggestion from other reviewers, we have tested our algorithm's performance in detecting prompt injection scenarios, and it also performs effectively in this context (see Appendix E.4 in the revised draft). (2) Detecting Model Changes with Variable Temperatures: We have also included experiments to detect model changes when the temperatures of the pre- and post-change models both vary within a specified range (in contrast to our previous results, where the temperature was held constant). Our detection method continues to perform well in this scenario, illustrating its robustness, especially when responses may come from distributions that differ due to varying temperatures within the same model (see Appendix E.3 in the revised draft).

We hope these additional experiments provide further evidence of our method’s robustness and performance, alongside our previous results demonstrating detection across diverse scenarios, including watermark emergence, synthetic version updates, and real-world version updates.

Q2: Regarding adversarial attack study

A2: To demonstrate the robustness of our algorithm (which may partially address your concern), we conducted additional experiments in which the temperature of both the pre-change and post-change LLMs varied, uniformly sampled from the range [0.9, 1.0]. And in this case, our proposed detection algorithm continued to perform effectively without significant degradation, as shown in the newly added Figure 21.

Furthermore, we would like to emphasize that, to the best of our knowledge, this work is the first to formulate the online detection problem for black-box LLMs and establish a solution that demonstrates strong performance and high efficiency. While we acknowledge the validity of your concern about adversarial attacks, this falls outside the current scope of our study, though it presents an interesting direction for future research. The current detection framework could be extended to robust counterparts by incorporating distributional uncertainties introduced by adversarial attacks. For instance, with prior knowledge of the type of adversarial attack, we could frame the online detection problem as a distributionally robust change detection task, aiming to detect changes from a pre-change ambiguity set of response distributions to a non-overlapping post-change set. The sizes of these sets would reflect the strength of the adversarial attack. Nonetheless, this extension is beyond the current study’s focus.

Q3: Regarding real-time performance

A3: In our study, we highlight the real-time setting of online detection throughout the paper, and all results are obtained under the real-time detection environment. Specifically, in our real-world dataset, we collect data on a daily basis, so that we can make daily decision on whether to report a change based on the responses collected so far. In our synthetic experiments, even though we collect data before we conduct the detection algorithm, we always run the detection procedure in an online manner, which means that each decision about whether to raise alarm is completely based on the information collected by that time step.

To be continued...

Dear Reviewer niaS,

We hope this message finds you well. We are writing to follow up on our paper during the rebuttal phase. In response to your feedback, we have conducted additional experiments and made significant revisions to the paper, which we believe address your concerns.

As we near the conclusion of the discussion period, we kindly ask if you feel that our rebuttal and the updated paper adequately resolve your points. If so, we would greatly appreciate it if you could update your evaluation. Of course, if there are any remaining issues or areas needing clarification, please let us know—we would be happy to provide further details.

Thank you very much for your time and valuable input.

Best regards, Authors 13317