Archilles' Heel in Semi-open LLMs: Hiding Bottom against Recovery Attacks

Thanks for your time and valuable comments. We are pleased that you acknowledged the novelty and significance of our focused problem, promising results and straightforward presentation. In the response below, we provide answers to your questions in order to address the concerns and increase your confidence.

Q1: Is the "attack datasets" mentioned in Figure 2 the same as the "recovery datasets" discussed later in the paper?

R1: Thank you for your insightful comments and for highlighting this inconsistency. The terms "attack datasets" and "recovery datasets" refer to the same dataset, which is used to facilitate model recovery attacks aimed at replicating victim model’s behavior. We sincerely appreciate your feedback and have revised Figure 2 in the manuscript to ensure consistent terminology is used throughout.

Q2: Could you clarify the formula and the loss function used for RD(I)?

R2: Thank you for your comment on clarifying RD. We have revised the definition and implementation details in Sections 4.2 and 5.1. Below is the detail formulation and explanation of RD:

The Recovery Difficulty (RD) quantifies the challenge of recovering the closed-source module and is defined as: $\text{RD}(I) = \mathbb{E}_{\mathbf{X},Y,{\theta_0}(I)}\left[\ell\left(f(\mathbf{X};{\theta}_0(I)), Y\right)\right]$

Where:

- $I$: The set of indices for the closed-source layers, indicating which hidden layers are kept private.

- ${\theta}_0(I)$: The initial parameters of the replacement model. Parameters for hidden layers are randomly initialized, while parameters for public layers remain unchanged.

- $\mathbf{X}$: Input features sampled to target general capabilities from the underlying distribution.

- $Y$: Labels corresponding to $\mathbf{X}$.

- $f$: The final output of the semi-open model.

- $\ell$: The loss function, where we use cross-entropy loss in this paper.

- $\mathbb{E}$: The expectation over the joint distribution of random inputs, labels, and randomly initialized

We estimate RD using 1,500 samples drawn from two diverse datasets: the MMLU benchmark and Alpaca 52k. These datasets cover tasks such as text comprehension, summarization, generation, code writing, and mathematical reasoning. Additional details are provided in Section 5.1 and Appendix B.4. Moreover, hidden layers for different closed-source sets are randomly initialized using Xavier initialization with PyTorch's default settings. In this study, the RD is averaged over three random seeds (20, 42, 1234) to ensure robustness.

Q3: Could you clarify how fully-closed and semi-open models differ in practice?

R3: Thank you for your thoughtful feedback. Fully closed models provide strong vendor control but significantly limit user customization options. For example, users must rely on fine-tuning APIs provided by vendors, which handle the fine-tuning process using vendor-controlled computational resources. In this setup, users do not have access to the model's internal parameters, restricting their ability to perform detailed customization.

In contrast, semi-open models strike a balance between customizability and robustness against recovery attacks. This approach allows vendors to retain control over proprietary components, secure revenue streams, and reduce the computational burden of fine-tuning. Meanwhile, users benefit from the flexibility to customize open-source modules offline, optimizing them for specific tasks. We hope this explanation clarifies our motivations, as outlined in our response to R1 in the general response section.

Q4: Could you explain more about the distinctions between FT-all, FT-closed, and SEM in Section 5.1?

R4: Thank you for your thoughtful comments. In the R2 of our general response, we have provided a detailed explanation of the distinctions among the three strategies. These distinctions enable a more comprehensive evaluation of SCARA's effectiveness under our proposed threat model. Additionally, we have revised the manuscript to further clarify these three strategies in our threat model. Thank again for your constructive suggestions.

Q5: Can the row and column headers in the tables be made clearer by avoiding abbreviations?

R5: Thank you for your suggestion. While we aim to maintain the table’s length and visual clarity, we understand the importance of readability. Following your suggestion, we carefully revise the manuscript to minimize abbreviations where possible and improve clarity.

Thank you for the detailed and comprehensive responses, which have clarified my concerns and provided a better understanding of SCARA.

After reviewing other feedback and the authors’ responses, I like the thoughtful revisions, which have notably improved the paper. (1) The paper effectively highlights the rise of semi-open models, where closed-source embedding models integrate with open-source modules. In your response to Reviewer KKBu, the distinction between 'semi-open pipeline' and 'semi-open model' is not crucial. What matters is the understanding and application of the concept.

(2) SCARA addresses performance challenges in semi-open models by partitioning pretrained LLMs into closed-source and open-source components. This design balances customizability with enhanced resilience to recovery attacks, benefiting both vendors and users.

(3)According to the general response, the threat model discussed is well-recognized within LLM communities. As highlighted in your paper, there is substantial research on model recovery and model extraction attacks, indicating a significant interest and concern in these areas.

(4) I agree with Reviewers d5Gh and 1rG5, the analysis of a transition layer is particularly compelling and may have broad implicatio ns for the research community. This insight helps us estimate the minimal number of hidden layers required without fine-tuning.

This paper provides a well-structured, insightful contribution of broad interest. The SCARA framework balances customizability with resilience to recovery attacks, enhancing semi-open models. I recommend to accept.

I would like to thank the authors for their effort in rebuttal and their thoughtful response, which further clarifies their practical motivation. In the initial review, I found the overall structure of the paper to be coherent and the proposed theoretical contributions to be highly innovative, which led me to assign a score of 8. However, as noted by other reviewers, the inclusion of embedding models introduced some confusion. The revised scenario of on-premises deployment is more aligned with my expectations, and I appreciate that the authors have now focused on this setting. Given that, to the best of my knowledge, there has been no prior work that addresses this issue both theoretically and empirically, I continue to strongly recommend the acceptance of this paper.

Dear Reviewer ag9Z:

We sincerely appreciate your kind and positive response. We are truly grateful for your recognition and support of our work and contributions. Furthermore, we deeply value the effort you have dedicated during the rebuttal process. Thank you once again.

Best Regards,

Authors

Thanks for your time and valuable comments. We are pleased for your positive comments. In the response below, we provide answers to your questions in order to address the concerns and increase your confidence.

Q1: Why the semi-open models are practically relevant?

R1: Thank you for your positive, constructive and valuable comments. We appreciate the reviewer’s feedback on the concern of semi-open models' practically relevant. We outlined the advantage of semi-open models in the general response section. We hope our response provides a clearer understanding of the motivations behind our approach.

Q2: Threat models are unclear.

R2： Thank you for your insightful feedback. We have detailed the threat model in general response R2 and further refined Section 3.1. Your suggestions have greatly improved the clarity of our paper.

Thanks for your kind and helpful comments and we are looking forward to discussing with you to further improve our paper!

Dear Reviewer 1rG5:

Kindly note that the author-reviewer discussion period is currently ongoing. We would greatly appreciate it if you could review our response when convenient. We earnestly request that you reconsider our manuscript and consider upgrading your score.

Regards,

Authors

Dear Reviewer 1rG5:

We sincerely thank you for raising your concerns regarding the practicality of the motivation behind the problem we addressed. In our general response 2, we have provided further clarification of our motivation and revised the manuscript accordingly. We would greatly appreciate it if you could review our response at your earliest convenience. We respectfully request that you reconsider our manuscript and kindly revise your score.

Regards,

Authors

Thanks for your kind and helpful comments and we are looking forward to discussing with you to further improve our paper! We are encouraged that you appreciated our contributions, including the theoretical proof and comprehensive experiments. In the response below, we provide answers to your questions in order to address the concerns.

Q1:Unclear motivation for semi-open models.

R1: Thank you for your constructive and valuable comments. We greatly appreciate your feedback and the opportunity to clarify our motivations, as outlined in R1 within the general response section. We hope our response provides a clearer understanding of the motivations behind our approach.

Q2: The threat model is not clear.

R2: Thank you for your constructive and valuable comments. In our general response R2, we have further clarified our threat model and provided a more detailed description of the adversary's capabilities.

Q3: Insufficient details on SCARA's implementation

R3: Thank you for your comments on SCARA's implementation. We have revised the implementation details part in Section 5.1.The key component of SCARA is the Recovery Difficulty (RD), which quantifies the difficulty of recovering the closed-source module. It is defined as:

$\text{RD}(I) = \mathbb{E}_{\mathbf{X},Y,{\theta_0}(I)}\left[\ell\left(f(\mathbf{X};{\theta}_0(I)), Y\right)\right]$

Where:

- $I$: The set of indices for the closed-source layers, indicating which hidden layers are kept private.

- ${\theta}_0(I)$: The initial parameters of the replacement model. Parameters for hidden layers are randomly initialized, while parameters for public layers remain unchanged.

- $\mathbf{X}$: Input features sampled to target general capabilities from the underlying distribution.

- $Y$: Labels corresponding to $\mathbf{X}$.

- $f$: The final output of the semi-open model.

- $\ell$: The loss function, where we use cross-entropy loss in this paper.

- $\mathbb{E}$: The expectation over the joint distribution of random inputs, labels, and randomly initialized parameters of the closed-source module. This expectation is approximated in practice.

--Approximation of RD--

- Evaluation Dataset: To estimate RD, we construct an evaluation set that represents the general capabilities of the victim model. The dataset includes 1,500 samples evenly drawn from two diverse datasets: the MMLU benchmark and Alpaca 52k. These datasets cover tasks such as text comprehension, summarization, generation, code writing, mathematical reasoning, and knowledge reasoning. For more details, please refer to Section 5.1 and Appendix B.4.

- Random Initialization of Closed-Source Parameters: To evaluate RD for different closed-source layer sets, the hidden layers are randomly initialized using Xavier initialization with PyTorch's default settings. The RD is averaged over three random seeds (20, 42, and 1234) during SCARA implementation.

--Computational Resources for RD--

- For models with up to 7 billion parameters, RD calculation and SCARA execution are performed on a system with 4×RTX 4090 GPUs, completing in approximately 8 minutes.

- For larger models like Llama2-70B, the process is carried out on 4×A100 GPUs, taking around 30 minutes.

Q4: No comprehensive comparison to existing methods:

R4: Thank you for your constructive and valuable comments on the comparison to existing methods. While the design of semi-open models has been widely studied in areas such as clustering, to the best of our knowledge, in the domain of complex tasks such as deep reasoning, knowledge-intensive operations, and problem-solving in code or math, SAP [1] is the only approach that serves as a basis for a semi-open LLM construction framework with which we can directly compare. We would sincerely welcome any suggestions for related approaches, and we would be happy to incorporate additional experiments based on your recommendations.

Q5: No analysis on potential trade-offs between customizability and security.

R5: We appreciate your feedback regarding the trade-offs between customizability and security. We have revised our manuscript to further explore this trade-off, Specifically, we examine these aspects using Llama2-7B and Phi-2. As shown in Section 5.3, we barely observe significant trade-offs in closed-source set placement but a clear trade-off in the number of hidden layers for smaller models like Phi-2.

Thanks for your kind and helpful comments and we are looking forward to discussing with you to further improve our paper!

[1] A Split-and-Privatize Framework for Large Language Model Fine-Tuning https://arxiv.org/abs/2312.15603

Dear Reviewer d5Gh:

Kindly note that the author-reviewer discussion period is currently ongoing. We would greatly appreciate it if you could review our response when convenient. We earnestly request that you reconsider our manuscript and consider upgrading your score.

Regards,

Authors

Thank you for your efforts in rebuttal. Parts of my concerns are addressed, however, the "semi-open model" or "grey-box" motivation under the LLM context is still not clear. I decide to maintain my score.

Dear Reviewer d5Gh:

We sincerely thank you for raising your concerns regarding the practicality of the motivation behind the problem we addressed. In our general response 2, we have provided further clarification of our motivation and revised the manuscript accordingly. We would greatly appreciate it if you could review our response at your earliest convenience. We respectfully request that you reconsider our manuscript and kindly revise your score.

Regards,

Authors

Thanks for your time and valuable comments. In the response below, we provide answers to your questions in order to address the concerns and increase your confidence.

Q1: Questions on the threat model.

R1: Thank you for your thoughtful comments and concerns about the threat model. We appreciate your mention of work related to stealing parts of the embedding layer in LLMs [1]. In this paper, we focus on a different type of threat model, known as model recovery or model extraction attacks. Since 2016, researchers have studied how to extract parameters and structures from black-box encoders through these attacks [2]. These typically involve querying the black-box model, collecting input-output pairs, and training a replacement model to replicate the original's behavior.

In our design, the closed-source component acts as a black-box encoder, generating hidden representations for input data. As such, we believe the closed-source component in our design is vulnerable to model recovery attacks. To address this, we investigate how hiding certain layers can help defend against such threats. Further clarification is provided in the general response R2, and we hope this addresses and alleviates your concerns.

In response to the question regarding the potential for stealing a semi-open LLM, we address this concern in Section 5.3 of our paper. Our findings suggest that poorly designed semi-open models are indeed vulnerable to recovery attacks. For instance, in our experiments, designating a later decoder layer (e.g., the 29th layer in Llama2-7B) as the closed-source component while open-sourcing the remaining layers substantially increased the model's susceptibility to recovery. Specifically, when only the 29th layer was hidden, we observed an average recovery ratio approaching 100% under recovery attacks. This indicates that the recovered model could closely replicate the victim model’s behavior across various functionalities. These results highlight that without careful design, the risk of stealing a semi-open LLM remains significant.

Q2: The main insight of the paper is trivial.

R2 Thank you for your detailed feedback. We appreciate the opportunity to clarify and expand on the contributions of our work.

First, we would like to highlight that the question of "hiding the first layer versus the last layer" is not as trivial as it may appear. Prior work, such as SAP [3], has studied a similar problem, but they open-source the bottom six layers, and keep the remaining layers closed-source. They empirically investigated the customizability of this design for downstream tasks. However, in this paper, we demonstrate that such heuristic is not optimal for balancing customizability and resilience against recovery attacks.

Second, the primary message of our theorem is not simply that "hiding earlier layers is better than hiding later layers." Instead, our theorem establishes the existence of a transition layer—a critical point in the model such that hiding layers before this transition offers strong resilience against recovery attacks, while hiding layers after it does not. The example of "hiding the first layer versus the last layer" was included for simplicity and to make the idea more accessible, but it represents just one instance of our broader result. To the best of our knowledge, this is the first work to both theoretically and empirically identify such a transition layer and rigorously prove its existence in the context of defending against recovery attacks. Our proof demonstrates the crucial role of the bottom layers in providing resilience, making them a key focus for effective defenses. Moreover, we introduced several novel techniques in our proof process that, to the best of our knowledge, have not been explored before. These techniques may have broader implications and could benefit the research community beyond this specific context.

Finally, we appreciate the reviewer pointing out related empirical observations about the redundancy of later layers and their limited role in learning higher-order interactions. While this is not directly related to our main theorem, we believe our theoretical findings provide valuable insights into understanding LLMs in scenarios beyond defending against recovery attacks. We thank the reviewer for highlighting the potential broader implications of our results.

[1] Stealing Part of a Production Language Model http://arxiv.org/abs/2403.06634

[2] Stealing Machine Learning Models via Prediction APIshttps://arxiv.org/pdf/1609.02943

[3] A Split-and-Privatize Framework for Large Language Model Fine-Tuninghttp://arxiv.org/abs/2312.15603

Q3: The actual implementation of the method is not sophisticated. It just takes this straightforward insight and turns it into a metric.

R3: Thank you for your detailed feedback. Our goal is to predict the recovery ratio, which evaluates: "If the first $N$ layers of the model are hidden, to what extent can adversaries recover the model's general capabilities?" As discussed in Section 3.2, directly calculating this metric is highly computationally intensive. This process requires constructing a large query set and performing extraction attacks, which involve fine-tuning the entire semi-open model to replicate the original model for each possible hidden module configuration.

To address this challenge, we aimed to develop a metric that avoids fine-tuning while being highly correlated with the recovery ratio. This task is non-trivial because the score function used to measure recovery (e.g., accuracy in a reading comprehension task) can be non-differentiable.

To overcome this, we used the cross-entropy loss as a differentiable surrogate metric, which is negatively correlated with the score function. Using Taylor expansion and theoretical insights from prior work, we leveraged the observation that for large neural networks relative to the dataset size, the difference between the fine-tuned parameters $\|{\theta}_{\text{FT}}(I,\mathcal{D}) - {\theta}_0(I)\|_2$ is minor. Previous research has shown that for models like single-layer ReLU networks, this difference is of the order $\mathcal{O}\left(\frac{|\mathcal{D}|}{\sqrt{N}}\right)$, where $N$ is the number of model parameters, which is significantly larger than the dataset size in large language models.

By incorporating these insights, we derived our final metric, providing a computationally efficient and theoretically grounded approach to approximate the recovery ratio.

Q4: The evaluation isn't really fair. The authors evaluate against SEM. But SEM just wants to recover the embedding and the authors are trying to show what happens if they hide the early parts of the network. This seems like an indication that this isn't a particularly realistic threat model.

R4: Thank you for your comments. As discussed in the general response section, the closed part of our model functions similarly to an encoder in our design. SEM is an attack strategy aimed at replicating the black-box encoder, which we believe aligns with our threat model and is applicable to our setting. If the adversary successfully replicates the closed-source module using SEM, it would constitute a successful model stealing, effectively compromising the entire model. We appreciate the reviewer pointing out the potential confusion regarding SEM, and we have revised the paper to provide a clearer explanation of the SEM process.

Thanks for your helpful comments and we are looking forward to discussing with you to further improve our paper!

Dear Reviewer kKBu:

Thank you for your time and comments. Below, we have provided responses to your questions to address your concerns.

Q1: Semi-Open LLMs exist: Your provided references for this are End-to-end systems that have a closed-source embedding model as one component of the pipeline. Nothing like what you are trying to attack/defend, an LLM that actually has some number of the layers open and some number of the layers closed, exists.

R1: Thank you for acknowledging the existence of end-to-end systems with semi-open pipelines that combine closed-source embedding models and open-source components. This aligns with the discussion in our paper, where the closed-source module serves as a component providing hidden representations, while the open-source module remains customizable. We use the term "model" broadly to describe these pipelines. However, if the reviewer prefers the term "semi-open pipeline" over "semi-open model," we are open to adopting this terminology, although we believe "model" effectively conveys the overarching concept. Furthermore, "semi-open models" are also referred to as "grey-box models" in references [1-4].

Q2: Threat model: Stating that the closed-source component acts as a black-box embedding layer does not actually make it an embedding layer. The SVD attack of Carlini et al. only works when there is exactly 1 layer working to take the inputs from representation space to vocabulary space. So there is no analogue here. There is still no evidence that the attack you are studying is a realistic threat.

R2: We use the term "embedding model" to broadly describe an encoder model that transforms input sentences into high-dimensional real vectors, which aligns with the role of the closed-source component in our framework. However, we acknowledge that in some contexts, "embedding model" may specifically refer to the output of the last decoder layer in large language models. To eliminate potential confusion, we have updated the paper and responses to replace "embedding model" with "encoder." There is a rich literature on stealing/extracting closed-source encoders such as [4-9].

Q3: The paper's insight is trivial: The existence of a transition layer follows immediately from the straightforward observation of compounding error at each layer. None of the techniques in the paper are novel.

R3: The observation of compounding error at each layer suggests a gradual change in the impact of individual layers in defending against recovery attacks. However, this gradual change does not directly lead to the sharp transition demonstrated in our theoretical results. Our contribution lies in rigorously establishing a theoretical analysis that proves the existence of this sharp transition, as acknowledged by all other reviewers.

[1] Risks and Opportunities of Open-Source Generative AI https://arxiv.org/pdf/2405.08597

[2] Grey-box Extraction of Natural Language Models https://proceedings.mlr.press/v139/zanella-beguelin21a.html

[3] A Comparative Analysis of White Box and Gray Box Adversarial Attacks to Natural Language Processing Systems https://www.atlantis-press.com/proceedings/iciaai-24/126004152

[4] I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences https://dl.acm.org/doi/pdf/10.1145/3595292

[5] StolenEncoder: Stealing Pre-trained Encoders in Self-supervised Learning https://arxiv.org/pdf/2201.05889

[6] Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queries 2406.10280 (arxiv.org)

[7] Refine, Discriminate and Align: Stealing Encoders via Sample-Wise Prototypes and Multi-relational Extraction https://arxiv.org/abs/2312.00855

[8] Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Services https://arxiv.org/pdf/2408.02814

[9] Sentence Embedding Encoders are Easy to Steal but Hard to Defend https://publications.cispa.de/articles/conference_contribution/Sentence_Embedding_Encoders_are_Easy_to_Steal_but_Hard_to_Defend/25287991?file=44691499

Dear Reviewer kKBu:

We sincerely thank you for raising your concerns regarding the practicality of the motivation behind the problem we addressed. In our general response 2, we have provided further clarification of our motivation and revised the manuscript accordingly. We would greatly appreciate it if you could review our response at your earliest convenience. We respectfully request that you reconsider our manuscript and kindly revise your score.

Regards,

Authors

Q2: The threat model is unclear and please provide more details.

R2: In our design, the closed-source component functions as a black-box encoder, generating hidden representations for input data. The open-source component then uses these representations and is fine-tuned for specific downstream tasks. However, since 2016, researchers have studied how to extract parameters and structures from black-box models in attacks known as model recovery or model extraction attacks [17]. These attacks typically involve querying the black-box encoder, collecting input-output pairs, and training a replacement model that replicates the behavior of the original.

We adopt a common threat model [18-19], where we assume the adversary can query the semi-open model, access its final outputs, and retrieve the representations generated by the closed-source module. Additionally, as described in [20-21], the adversary is assumed to know the architecture of the closed-source module but does not have direct access to its parameters. By leveraging full access to the open-source components, the adversary can fine-tune a replacement model based on the known architecture of the closed-source module, using the retrieved representations or final outputs as training labels. We examine three recovery strategies under this threat model:

- FT-all: The adversary fine-tunes both the replacement model for the closed-source module and the open-source module together, using the input and final output of the semi-open model.

- FT-closed: The adversary fine-tunes only the replacement model of the closed-source module, keeping the parameters of the open-source module unchanged, using the input and final output.

- SEM: The adversary fine-tunes the replacement model of the closed-source module using the input and the representations generated by the original closed-source model, without involving the open-source module.

We have revised the paragraph begining with "semi-open model recovery attack" in Section 3.1 to provide more details on threat model.

Summary of the Paper Revisions

The main updates are summarized as follows:

1. Page 1, Sec. 1: Add related works on semi-open models and their applications.
2. Page 3, Figure 2: Correct the description of the attack datasets
3. Page 3, Sec. 3.1: Provide more detailed threat model of the Semi-open Model Recovery Attack.
4. Page 5, Sec. 4.2: Supplement the definition of Recovery Difficulty (RD) with additional details.
5. Page 6, Sec. 5.1: Add more details on SCARA's implementation， and a brief description of the attack dataset sizes.
6. Page 9, Sec. 5.3: Rewrite the analysis of the trade-off between customizability and resilience in SCARA.
7. Page 9, Figure 6(a)(b): Revise the figure and caption to better illustrate the trade-offs between customizability and resilience to recovery attack.
8. Page 28, Appendix B.9: Add a discussion on how resilience transitions vary across specific capabilities.

References

[1] https://openai.com/index/introducing-vision-to-the-fine-tuning-api/

[2] https://ai.meta.com/resources/models-and-libraries/

[3] https://platform.openai.com/docs/guides/embeddings/embedding-models

[4] https://cohere.com/embeddingsh

[5] https://cloud.google.com/vertex-aih

[6] https://www.llamaindex.ai/h

[7] https://haystack.deepset.ai/

[8] https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag

[9] Vector Search with OpenAI Embeddings: Lucene Is All You Need https://arxiv.org/pdf/2308.14963

[10] Unsupervised Anomaly Detection in Multi-Topic Short-Text Corporahttps://cnrs.hal.science/hal-04471726/file/EACL_2023_ait-saada.pd

[11] Detection of Hate Speech using BERT and Hate Speech Word Embedding with Deep Model https://www.tandfonline.com/doi/full/10.1080/08839514.2023.2166719

[12] Performance Optimization in the LLM World 2024 https://dl.acm.org/doi/10.1145/3629527.3651436

[13] How Open Source Machine Learning Software Shapes AI https://dl.acm.org/doi/10.1145/3514094.3534167

[14] 2024 AI Predictions | NVIDIA Blog

[15] Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)

https://ieeexplore.ieee.org/document/8466590/?arnumber=8466590

[16] https://www.preprints.org/manuscript/202307.2142/v2

[17] Stealing Machine Learning Models via Prediction APIs https://arxiv.org/pdf/1609.02943

[18] I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences https://dl.acm.org/doi/full/10.1145/3595292

[19] Can't Hide Behind the API: Stealing Black-Box Commercial Embedding Models http://arxiv.org/abs/2406.09355

[20] Stealing Part of a Production Language Model http://arxiv.org/abs/2403.06634

[21] Grey-box Extraction of Natural Language Models https://proceedings.mlr.press/v139/zanella-beguelin21a/zanella-beguelin21a.pdf

Multiple reviewers made it clear that the authors are studying a threat model that is not well motivated, and the conclusions of the method are basically trivial. In response, the authors have pivoted their paper to focus on a setting where the original model developers hosts some layers are on the TEE that are closed, and some layers outside the TEE that can be finetuned. However, TEE memory is numbered in the MBs, not GBs, and is not even large enough to fit a single layer of any model worth stealing. The setting the authors are considering simply does not exist in the real world, because it doesn't make any sense to only open some layers of the model. Because the problem of "what LLM layers should be open and which should be closed to prevent model stealing during finetuning" has no practical relevance, it has not been studied by any prior work. The authors cite Carlini 2024 to support their case that the problem is worth studying, but Carlini 2024 does not operate in this threat model. Without any prior work to measure their method against, we have to just evaluate the method on its merits. And the conclusions of the method are trivial: just evaluate the model to see how many of the first few layers you can reasonable close, and close those layers. If anyone cared about this problem, this is the natural thing to do, and as another reviewer has pointed out, this is what embedding models already do.

In summary: the paper proposes a trivial solution to an unrealistic threat model. I recommend a reject.

We sincerely thank you for your time and valuable comments. In the response below, we provide answers in order to address the concerns.

Q1. TEE memory is numbered in the MBs, not GBs, and is not even large enough to fit a single layer of any model worth stealing

Thank you for your constructive comments. In order to address your concerns, we provide further clarification regarding TEEs. While individual TEE hardware typically has limited memory, certain recent implementations allow for the extension of secure memory. For instance, Microsoft Azure recently introduced the DCesv5 series VMs [1] (released on October 25, 2024), which leverage Intel’s Trust Domain Extensions (TDX) [2]. TDX is a hardware-based TEE that facilitates the deployment of trust domains (TDs) and can be scaled to support more than 8GB of secure memory [3]. This capacity is sufficient to protect several layers in models with 70B parameters or larger (e.g., a single layer of Llama2-70B requires approximately 1.75G of memory under float16 precision), but is insufficient to prtect the entire model (e.g., Llama2-70B requires more than 140G under float16 precision) .

Besides from the capacity limits, [4-8] also state that using TEE to protect all parameters of an entire model is not a practical solution. As noted in [8], "Attempting to shield the complete DNN model within a TEE (shielding-whole-model) could result in a 50x reduction in the model’s speed". Since TEEs are CPU-based, they do not provide the same level of efficiency for fine-tuning and customization on private data as GPUs. This limitation results in a trade-off, where security is prioritized at the cost of meeting users’ customization needs.

We have revised the introduction of our manuscript to further clarify the capacity of TEE. We hope this revision addresses your concern.

Q2. The setting the authors are considering simply does not exist in the real world, because it doesn't make any sense to only open some layers of the model.

Thank you for your valuable feedback regarding the practicality of the setting we consider. We appreciate the opportunity to provide further examples that support the setting we have explored.

First, we would like to emphasize that model asset protection in private deployments on users' local servers is a real-world challenge, which has been studied since 2018 [4-8]. For instance, a paper published in MobiSys in 2020 [4] states: "Due to the limited memory of the edge device’s TEE, we partition model layers into more sensitive layers (to be executed inside the device’s TEE) and a set of layers to be executed in the untrusted part of the operating system." This scenario aligns with our own, wherein certain layers are partitioned to be executed within the TEE, while the remaining layers are executed outside the trusted environment. The key difference is that the study [4] primarily focuses on defending against membership inference attacks (MIA), whereas our work focuses on mitigating risks related to model theft.

Second, we consider fine-tuning open-sourced layers to allow users to customize them using their local, private data. In our design, we employ the SCARA to identify and isolate a few layers within the TEE to secure the model, while allowing the remaining layers to be accessible for customization. As discussed in Section 5.3 of our manuscript, hiding more layers reduces the model’s customizability for downstream tasks but does not significantly affect the resilience provided by securing only the first few layers identified by SCARA. Meanwhile, protecting only those layers identified by SCARA offers customizability comparable to that of a fully open-source model. Therefore, the semi-open model constructed using SCARA strikes a balance between model security and customization, effectively addressing the trade-off between security and customization in on-premises deployments [9].

[1] https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dcesv5-series?tabs=sizebasic

[2] https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html

[3] https://www.intel.com/content/www/us/en/developer/articles/technical/tdx-performance-isolated-partitioned-vms.html

[4] Darknetz: towards model privacy at the edge using trusted execution environments http://arxiv.org/abs/2004.05703

[5] No privacy left outside: On the (in-) security of tee-shielded dnn partition for on-device ml. https://arxiv.org/pdf/2310.07152

[6] Confidential Inference via Ternary Model Partitioning. https://arxiv.org/abs/1807.00969

[7] Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. https://arxiv.org/abs/1806.03287

[8] TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Models. https://arxiv.org/pdf/2411.09945

[9] Securing AI Model Weights https://www.rand.org/pubs/research_reports/RRA2849-1.html