Thanks for your in-depth reviews. We have revised our paper based on your feedback, to address the following points:

1. We have corrected the placement of the ethical statement.
2. We have fixed typos and errors related to your questions.
3. We have reorganized Tables 1, 2, and 5 to provide a more comprehensive and fair understanding of the performance.
4. We have rewritten the methodology section for clarity.
5. We have appended an explanation of the query time complexity for LoRD.

Here are our point-by-point responses to your questions:

• Line 210-211: The victim model's response is introduced in the regularization term, which influences the optimization direction of the local model and, consequently, the selection of positive and negative models.
• Why not SimPO: We are aware of these methods, but they require an extra prerequisite of the victim model, i.e., the ability to judge responses beyond simply accomplishing the given task. This prerequisite cannot always be fulfilled, due to limitations in the victim model's capabilities and the stealthy requirement of model extraction. Additionally, they are actually more complex for the adversary, as they involve issues such as prompt design and victim model capabilities. Despite these limitations, we admit that one could potentially improve the effectiveness of LoRD by more accurately judging positive samples.
• Regularization term: Thanks again for your thoughtful insights. We agree that "regularization" is not the most appropriate term here. We used it following the tradition in RL, where the objective function is about the exploration (i.e., learning) of the local model, and the regularization term limits the local model from deviating too far from the victim model's response. This is the physical meaning of "regularization". While in RL the regularization constrains the optimized model with the last-step-optimized model, in MEA we constrain the $y^+$ with $y_{vic}$. The both are designed to ensure convergence.
• Complexity of query times: an intuitive explanation is shown in Figure 11. Formally, we need $O(V^{N\_Q})$-level query samples to represent the input data distribution. Since there is usually not just one correct answer in generative tasks, we can consider that there are at most $O(V^{N\_R})$ responses per query, which is the complexity for a token-level MLE. LoRD does not affect the query side, so the required query times to represent the whole input data distribution remains unchanged. However, ideally, LoRD can find all responses that are "isotropic" to the victim model from the search in $(V^{N\_R})$ candidates. That's the reason we refer to it as $O(1)$ complexity on the generation side. In realistic scenarios, it is impossible to find all candidates from a single "standard answer", so we relax $O(1)$ to $O(C)$ where the value $C$ quantifies the capability of the initial local model and thus should be considered as a constant.
• Watermark Resistance Experiments. Thanks for your suggestions on defenses. Regarding model-level watermarks, we have already discussed them in our potential defenses section. We believe that model-level watermarks are effective against both LoRD and MLE once the local model learns about triggers (backdoors). We have further detailed this part based on the literature you provided in our revised version. However, we face challenges in fulfilling your requirements: i) unlike content-level watermarks which has both packages and official implementations [1], model-level watermarks are still in their academia stage, making them challenging for experiments; ii) current academic studies in this field (including the two you list) have only tested their efficacy on BERT, while the effectiveness on generative and generalized NLP tasks is not clear; iii) as we mentioned in the paper, model-level watermarks will become ineffective if the query set of the adversary does not cover the backdoor triggers. This situation is common, as it is difficult for a generalized LLM to possess backdoors in each downstream field. In summary, in the raw paper we have acknowledged that model-level watermarks are effective and are a promising direction to mitigate our proposed methods, while we also discussed their limitations as well as their research states.

Based on your response, we compare LoRD with SimPO. SimPO incorporates two labels, $y\_w$ and $y\_l$, representing the winner and loser of the generated texts, respectively. Specifically, we reproduce SimPO according to Equation (6) outlined in SimPO's paper, and adopt the recommended hyper-parameter settings from SimPO's source code, which entails setting $\beta$ to 2.5 and $\gamma$ to 1.375.

It is important to note that SimPO was not originally designed for model extraction tasks but rather as a candidate for DPO, and we have made necessary modifications. We adpat SimPO for model extraction task, considering the following two implementations:

1. SimPO-I: Assigns $y^+$ as $y\_w$, and $y^-$ as $y\_l$.
2. SimPO-II: Utilizes $y\_vic$ as $y\_w$ and $y^-$ as $y\_l$.

Here is the performance comparison on WMT (de-en):

Based on the experimental results, it is evident that both SimPO methods exhibit inferior performance compared to LoRD within the same query budget. This observation indicates that SimPO may not be a highly query-efficient approach for model extraction attacks.

I appreciate the authors' additional results. However, what I expected is to prompt the victim model to decide the chosen and rejected responses, instead of directly using the chosen and rejected responses decided by LoRD to perform SimPO. I do think the responses addressed part of my concerns, and I would increase my score to a 5. I invite the authors to incorporate all the additional results and suggestions from other reviewers into the future version or submission. Thank you.

We sincerely thank you for your valuable response and constructive suggestions. Most of the experiments and revisions have been incorporated into the revised version of our paper, and we remain dedicated to addressing all concerns raised by the reviewers and will not withdraw our submission before the final decision. We kindly hope that the reviewer might reconsider the score if the explanation address the remaining concern.

In response to your concern about why we believe direct feedback from the victim model is not ideal for MEAs, we acknowledge that our previous explanation may not have been sufficient in the experiment part. To address this further, we have conducted additional experiments, as detailed below.

Settings.

We designed the prompt for obtaining feedback as follows:

For a translation task involving the conversion of the given `Text` into English, the user will provide two translation versions labeled `A` and `B`. Your task is to return the *letter corresponding to the better translation* without including any additional output.

In each training step, the local model generates two candidate responses. Using the above instruction, we determine the positive response, which is then used along with the negative response to fine-tune the local model under the SimPO loss function. We maintain the usage of hyperparameters in the previous responses.

Experiment results.

The experimental results are as follows:

In the table, T denotes the sampling temperature, and Q denotes the query times.

Analysis.

We conducted experiments with various sampling temperatures, yet the efficacy of stealing remained constrained under identical settings. This limitation may stem from the local model's lack of guidance from correct answers. When the local model generates two suboptimal responses, a direct prompting-based method is compelled to select the "winner" of two inadequate response rather than an optimal response, which we believe is the crux of the issue.

RLHF tackles this challenge by incorporating a regularization term with the initial model, LoRD addresses it through our $L_{reg}$, leveraging the victim model's response, and DPO resolves it by employing the training corpus of the reward model. unfortunately, a direct prompt-based method overlooks this point. To further investigate this problem, we increased the query number to 256, which resulted in the local model failing to converge and exhibiting poor performance.

Besides, we also observed a bias in the victim model's selection between the first and second sentences. In a series of 256 queries, the model successfully provided an answer (either A or B) 255 times. However, it chose the first sentence only 84 times, which is a mere 32.94%, significantly deviating from the expected 50%. Given that the generated sentences are randomly sampled from the local model without any significant correlation to their order, we deduce that relying on the victim model to directly generate feedback might be, at best, an unreliable approach. It may necessitate additional considerations for the design of the prompt and the capabilities of the victim model to ensure robustness.

We hope that the above empirical explanation further addresses your concerns. If not, we would be delighted to engage in further discussion after the review process, if possible.

We sincerely appreciate all of your previous feedback and suggestions. Thank you!

Thank you for your valuable feedback. We are currently conducting four experiments to address your questions, and we hope to provide the results before the rebuttal period ends.

Regarding your final question, we have only evaluated current commercial LLMs on domain-specific tasks, as shown in Figure 9.

Thank you.

Experiments beyond Domain-specific Stealing

For weakness 1, we list conducted the experiments on safety alignment extraction. Specifically, we utilized two open-source datasets for these experiments, namely SafeRLHF and DiaSafety, to assess the safety of the responses generated. We employed PerspectiveAPI to automatically evaluate the safety of the responses. The API identifies five key aspects of safety probabilities: Toxicity, Insult, Profanity, Severe Toxicity, and Threat. In these categories, a lower score indicates better safety performance. For the LoRD model, we have retained the same hyper-parameters as those used in our domain-specific experiments to ensure consistency.

DiaSafety:

SafeRLHF:

Ablation Study

We also conducted an ablation study to compare several variants of LoRD for your questions 1 and 2. We applied the same LoRD settings to the WMT (de-en) dataset and compared LoRD's performance with the variants you mentioned.

Conclusion of the Ablation Study:

1. For Question 1: Removing the term $y_{vic}$ slightly reduces the performance in the stealing task, with the decrease being less than 1 point. Threfore, as we have done in the paper, we can conclude that this term can be omitted without significantly impacting the results.
2. For Question 2: Our approach, which incorporates $y^-$ in the regularization term, yields better performance compared to replacing $y^-$ with $y^+$. This indicates that the use of $y^-$ is more effective for our design objectives.

Thanks for your detailed reading and thoughtful review. We have carefully revised the paper in response to your concerns and provide point-by-point explanations below.

The Design of Loss Functions

Our target is to design an RL-style loss function for MEA, as shown in Equation (7). It consists of two parts, $L\_{obj}$ which represents the objective function, and $L\_{reg}$ which represents the regularization term. Equation (7) aligns with LLM's alignments (Equation (6)), where $L\_{obj}$ and $L\_{reg}$ correspond to $R\_{\theta\_{phi}}$ and $D_{KL}$, respectively. It is not necessary to design a loss function that is totally the same as or derived from LLM's RLHF, because there are various RLHF methods and their variants. Besides, in RL and RLHF, many methods, such as PPO, TRPO, and SimPO, often lack rigorous formal deductions beyond intuitive design. Nevertheless, we still aim to ensure that {i)} LoRD's loss converges consistently with LLMs' alignment, and ii) it converge at all. Our responses to your detailed comments are:

• Explanation of L228: In RLHF, a reward model is typically trained to estimate the debiased reward of a sample $(x, \hat{y})$. This reward model is trained using the loss function defined in Equation (5). In LoRD, we do not train such a reward model. Instead, we "use the logarithmic proportion between the positive and negative samples as the debiased reward," following the definition in Equation (5). We provide a less formal deduction in the Appendix; however, it is important to emphasize that many related works (e.g., SimPO) did not provide rigorous justifications.

• "The scaling term is simply dropped": Based on your feedback, we conducted an ablation study and revised the paper accordingly. Intuitively, dropping the scaling term does not significantly impact the efficacy of stealing because this term, introduced in PPO, primarily scales the reward to control the speed of convergence.

• About KL divergence: We omit this term because our experiments demonstrated that it is neither efficient nor stable for model extraction tasks. An ablation study for this term is included in the paper.

• Explanation of L248: We intended to show that $\log P$ is more natural than $P$ for implementation purposes. The former expands to $\log(\text{softmax}(\text{logits}))$, with $\log\text{softmax}$ being a more fundamental operator in modern ML frameworks.

• Clarification of line 251: The term "selected tokens" refers to those tokens sampled during the generation process.

• Sigmoid function: We provide an ablation study for this term. Our results indicate that $\text{sigmoid}$ serves a similar role to the clip term mentioned in the paper. While it is not strictly necessary, we recommend including it to enhance the stability of training.

NC denotes not converged in our experiments.

We appreciate your suggestions, which have made this paper clearer and provided it with a stronger motivation.

Presentation

1. Figure 3 aims to intuitively exhibit how we select potentially positive and negative samples and why such a selection strategy is reasonable. It illustrates step 3 in Figure 2. The core idea is that we consider a generated sample as a positive sample if it has a higher increment in terms of model's familiarity after model optimization, which is also the meaning of "Locality Reinforcement".
2. Thanks for your suggestion on the intuitive explanation before details. We have added an explanation on intuition to the methodology following your advice.
3. Table 1: Following your feedback, we removed Rouge-L's Precision and Recall. We save BERTScore's all three metrics because the results can reflect why local models underperform the victim model.

Typos

1. ”ChatGPT cha (2024)”: this seems the standard formation when citing urls in the natbib format.
2. We have checked and modified all incorrect citation formats in the revised version. Thank you.

Theoretical Analysis

1. We acknowledge that Proposition 1 doesn't have a strict mathematical statement, which is why we call it a "proposition" rather than a "theorem". Despite this, we still provided an intuitive explanation (Figure 4) and in-depth analysis of four loss functions' optimization process (Appendix D.1) to support it. While it is challenging to mathematically and precisely model and compare the converging procedures of neural networks under different loss functions, we made our best efforts to provide theoretical-level analysis and explanations beyond intuition and empirical experiments.
2. "Proposition 2 does not support the proposed method". The objective of this research is not merely a new MEA method, and instead we aim to address a fundamental question that has not been investigated, i.e., whether MLE can be used to steal an RL-aligned LLM, together with its upper bound and limitations. Proposition 2 provides the answer that "yes, MLE can be used to steal LLMs and will reach the performance of the victim model when its loss function reduces to zero", where this proposition itself is one of the contributions of our paper. In addition, it is Proposition 2 that leads us to dig deep about the intrinsic strength of LoRD from Proposition 1, which lead to our analysis in Section 4.2.
3. "Lack of rigor in Section 4.2". To enhance the rigor of our analysis, we have appended an extra explanation part for Section 4.2, and cited some recent studies to support our analysis.

Experiments

• About Limited Improvements. (i) The improvements compared to MLE are not limited. We evaluated LoRD on five downstream tasks, achieving improvements of 2–3 points on QA (Table 5), 1–4 points on machine translation (Table 2), and 2 points on two out of three summarization datasets (Table 1). For the other two tasks, LoRD still outperforms MLE under smaller query numbers. The issue lies in how our results were presented: we placed the three tasks where LoRD performed worst in the main table at the beginning, and in only half of these datasets did LoRD outperform MLE. (ii) LoRD doesn't aim to outperform MLE in all scenarios. As shown in Proposition 2, LoRD and MLE can converge to the same endpoint when provided with sufficient query samples. Therefore, for tasks that are easily generalized or learned, both methods may reach the performance ceiling, resulting in comparable outcomes. The effects of query efficiency and model scale are detailed in Figures 7 and 8.
• About Task Selection. We have included safety alignment experiments, as detailed in our responses below. While we agree that safety alignment is essential and should be included in our experiments, we also emphasize that alignment is not merely about safety—it also significantly impacts task completion performance. Therefore, the domain-specific evaluations presented in this paper remain valuable for comparing the performance of the two MEA methods.

Experiments of Safety Alignments

In response to your feedback, we have conducted safety alignment experiments.

We utilized two open-source datasets for these experiments, namely SafeRLHF and DiaSafety, to assess the safety of the responses generated. We employed PerspectiveAPI to automatically evaluate the safety of the responses. The API identifies five key aspects of safety probabilities: Toxicity, Insult, Profanity, Severe Toxicity, and Threat. In these categories, a lower score indicates better safety performance. For the LoRD model, we have retained the same hyper-parameters as those used in our domain-specific experiments to ensure consistency.

DiaSafety:

SafeRLHF:

Thanks for your review. We have revised our experiments part and provide feedback to address your concerns here:

Abnormal Phenomenon on WMT (de-en) in Watermark Experiments

We acknowledge that the watermark resistance experiment results on WMT (de-en) do not align perfectly with the setting of $\lambda_1$. We suspect that this abnormality is due to the fine-tuning checkpoints on the first two points, as the performance of LoRD on WMT (de-en) exhibits significantly higher variability compared to WMT (cs-en). This suggests that these two points may not have been properly trained. We will update the relevant experiments in due course. Nonetheless, even with this variability, LoRD still outperforms MLE in terms of watermark resistance on this dataset, so the conclusion of this part of experiments remains valid.

Influence of Local Models

As in Appendix C.1.2 and Appendix C.1.3, we have already conducted experiments to investigate how different capacities of local models affect extraction performance. Specifically, in Appendix C.1.2, we explored the relationship between the scale of local models and extraction performance using OPT series models with varying parameter numbers. In Appendix C.1.3, we evaluated the MEA efficacy across different local and victim models, including commonly used models such as Phi3, OPT, Qwen2, Mistral, and Llama3. Based on your feedback, we have strengthened this section of the experiments in the main paper.

Explanation of Table 1's D2T Part

We appreciate your detailed review of our paper and your questions regarding the "outperforms" situation in Table 1's D2T part. There are two reasons for it:

1. NLG evaluation is inherently challenging and may be subject to evaluation errors. As discussed in previous literature [1][2], current metrics like BLEU have limitations. Therefore, we used both lexical- and semantic-level metrics to provide a more comprehensive and convincing evaluation, as described in Section 5.1.
2. Different metrics may focus on different aspects of evaluation. Consequently, a bad answer may obtain a higher score in some metrics, but performs much worse on others. For example, if the local model generates a short sentence that is a subset of the reference sentence, it may receive an unreasonably high BLEU score. Such a abnormal phenomena can also be observed in BERTScore (Precision) and Rouge-L (Recall) for some extraction experiments.

[1] A. R, P. Bhattacharyya, M. Sasikumar, and R. M. Shah, “Some issues in automatic evaluation of english-hindi mt: More blues for bleu,” 2006. [Online]. Available: https://api.semanticscholar.org/CorpusID:5690091

[2] A. Stent, M. Marge, and M. Singhai, “Evaluating evaluation methods for generation in the presence of variation,” in Conference on Intelligent Text Processing and Computational Linguistics, 2005. [Online]. Available: https://api.semanticscholar.org/CorpusID: 11115098

Regarding the watermark resistance experiments, we have retrained the local model with $\lambda\_1$ set to 0.0. Unfortunately, the experimental results continue to exhibit abnormalities, as illustrated in Figure 6. As elaborated in our paper, we suspect that this result arises from the disability of the regularization term when $\lambda\_1$ is set to zero, which concurrently explains the poor Rouge-L score observed in Figure 6. Consequently, for tasks necessitating the injection of substantial additional knowledge, utilizing $\lambda\_1$ diminishes the efficacy of the extraction process. From Figure 6, a reasonable range for setting $\lambda\_1$ appears to be between 0.2 and 0.6, with 0.5 serving as our default setting.

To further investigate the correlation between watermark resistance and λ1​, we have conducted additional experiments on a different dataset (e2e-nlg), which exhibits a similar tendency to WMT (cs-en).

Summary of Strengths and Contributions

We also highlight the positive feedback from reviewers regarding the strengths and contributions of the paper:

• Presentation and Organization
  • "... well-written with a clear structure and rich content, ... easy to follow." - Reviewer aNx7
• "The paper is well-written, the tables and figures are well displayed. I thank the authors for their great efforts on well organizing their manuscript." - Reviewer 5Msi
• Contribution
  • "...might be the first work to steal models by considering the alignment procedure of LLMs with RL" - Reviewer Xic1
  • "...seems promising and does try to fill in an important gap in the knowledge about model stealing attack against modern LLM" - Reviewer Bcxr
• Novelty and Impact
  • "an interesting problem with potential impact" - Reviewer Bcxr
  • "The studied problem is practical and meaningful. The motivation is good and reasonable." - Reviewer 5Msi
  • "...steal commercial models under a fully black-box threat model, making it highly practical" - Reviewer aNx7
  • "...improves the way LLMs are extracted, thereby reducing the cost of queries." - Reviewer aNx7
  • "...does not replicate possible watermarks in the victim model." - Reviewer aNx7
• Theoretical Analysis
  • "The paper provides theoretical analysis on the consistency of model performance." - Reviewer Xic1
• Experiments
  • "Extensive and comprehensive experiments..." - Reviewer Xic1
  • "I like the analysis about Figure 5 (a spectrum of almost all NLP downstream tasks under stealing), which shows some novel and interesting insights." - Reviewer 5Msi
  • "The Watermark Resistance part is interesting and reasonable." - Reviewer 5Msi
  • "Although the method is not highly effective on every task, the authors have deeply explained the reasons behind these issues." - Reviewer aNx7
• Ethics Considerations
  • "...responsibly discussed ethical concerns and provided some possible defense strategies" - Reviewer aNx7

Summary of Scores

In summary, we sincerely thank all four reviewers again for their dedicated efforts and thoughtful feedback. We have carefully addressed all the questions and concerns raised by the reviewers, and based on the feedback from three of them, we confirm that we have resolved most of the issues. The reviews and rebuttal process truly enhanced the quality of the revised version of this paper. We kindly request that the reviewers reconsider their scores if our responses, explanations, and experiments have addressed your concerns. Thank you once again for your valuable input.