MPC-Minimized Secure LLM Inference

Thank you for your time and the encouraging feedback!

In Section 5.1, authors do not explain the criteria or threshold of the frozen fraction f.
According to Weakness 1, if the faction f is too large or too small, what will it happen?

We selected the layer freezing configuration (LF=0.5) based on an ablation study we performed on the MTBench benchmark (see Appendix K and Table 5 in the paper). The results show that score remains relatively consistent up to LF=0.5 (13 layers frozen out of 26). However, beyond this point, there is an almost linear decline in score as fewer layers are fine-tuned.
Based on these observations and a strong performance of this configuration on other benchmarks, we chose to freeze half the layers (LF=0.5) across our experiments to strike an effective balance between accuracy and secure inference cost. We’ll explicitly clarify this in the evaluation section.

Since only 2PC-Dealer supports GPU acceleration, the comparison to other MPC approaches with CPU only looks unfair due to the hardware difference. If we want to do apple-to-apple comparison, it is better to make 2PC-Dealer run with CPU only, too.

We compared with the 2PC-Dealer framework with GPU acceleration because we wanted to demonstrate Marill’s improvements over the most efficient configuration of this framework. It is important to note that the communication improvements over GPU will be the same over CPU, and communication is typically the bottleneck for 2PC-Dealer secure inference.
Additionally, the runtime improvements on 2PC-Dealer and 3PC framework are similar, even though the former is run on GPU and the latter on CPU, indicating that Marill’s performance improvements are consistent across hardware configurations.

In the real-world enterprise-level LLMs, the system usually takes millions of requests at a short time. So, instead of 2-PC or 3-PC setting, have you considered the extension of this work to n-PC setting or distributed setting?

In MPC, adding more parties typically makes the computation slower, unless you introduce a trust assumption for the additional parties. This is why 2PC+Dealer and 3PC settings are more efficient than 2PC – they have a weaker security guarantee due to this trust assumption.
To horizontally scale secure inference, in 2PC, if you have $n$ servers, you can run $n$ parallel queries by distributing them across those servers.
3PC and 2PC-Dealer require an additional helper party other than the client and the server, and if this helper party can horizontally scale the same way, then these settings can also scale similarly.

Can you provide more information about the difference between head-pruning and proposed head-merging.

Both head-pruning and head-merging are similar in that they aim to reduce the number of heads, but the key difference is that head-merging achieves that while preserving all the parameters of the foundational model. In contrast, head-pruning simply removes certain heads from the model along with their parameters.
This distinction allows head-merging to achieve better ML performance than head-pruning (see section 6.4 in the paper).
Furthermore, while head-merging is not useful for plaintext inference because the FLOPs remain the same, we show in Table 1b that it offers similar secure inference performance as head pruning due to the unique performance characteristics of MPC.

The paper's innovation is limited. The proposed method does not include improvements to the MPC protocol itself but focuses on different partitioning and head merging methods in the fine-tuning network structure.

Thank you for the feedback! We agree that our work does not focus on modifying MPC backends, and we want to emphasize that this is intentional and central to our contribution. Our goal is to make changes to the machine learning (ML) pipeline that outputs the inference models so that it is significantly cheaper to perform secure inference on these models. A key strength of our approach is that by making MPC-tailored changes to the ML itself, we are able to offer secure inference improvements irrespective of the underlying MPC protocol.

Within the context of secure inference, all three of our techniques are novel:

• Layer freezing and LoRA: while these techniques were originally introduced to accelerate fine-tuning, our work is the first to leverage these techniques in the secure inference setting. We identify how to adapt these techniques to both preserve security and improve efficiency, and present a novel application of these ideas.
• Head merging: this technique is entirely novel and uniquely suited to secure inference. It relies on the unique performance characteristics of MPC to improve secure inference cost while preserving FLOPs; note that the plaintext inference cost will stay the same.

Regarding the Layer Freezing method in the partitioning approach, is there any reference basis for selecting how many layers to freeze for different LLMs to achieve a balance between accuracy and timeliness?

We selected the layer freezing configuration (LF=0.5) based on an ablation study we performed on the MTBench benchmark (see Appendix K and Table 5 in the paper). The results show that score remains relatively consistent up to LF=0.5 (13 layers frozen out of 26). However, beyond this point, there is an almost linear decline in score as fewer layers are fine-tuned.
Based on these observations and a strong performance of this configuration on other benchmarks, we chose to freeze half the layers (LF=0.5) across our experiments to strike an effective balance between accuracy and secure inference cost. We’ll explicitly clarify this in the evaluation section.

2-Insufficient comparison of accuracy in experimental results. The paper does not compare the accuracy of other methods that combine MPC and LLM. Please supplement with more relevant experiments.

Thank you for raising this concern. To the best of our knowledge, the prior works that combine MPC and LLM primarily fall in the MPC-friendly approximations category discussed in the related work section of the paper.
Marill is complementary to these works as it makes high-level changes to the architecture, as opposed to the underlying operations. We also show in section 6.3 that these approximations can be combined with Marill to yield further performance improvements.
We did not include most prior works in our experiments because they rely on neural architecture search (NAS), where the model architecture is dependent on private training data. This dependency introduces additional information leakage beyond the standard secure inference guarantees, which conflicts with the strict security requirements we aim to preserve. In contrast, Marill’s architecture is statically configured, independent of the training data, and maintains the same security properties as standard secure inference. We’ll clarify this explicitly in the related work section.

If there are specific works we may have overlooked that do not fall into the MPC-friendly approximations or NAS categories, we would be happy to review and consider them for inclusion.

Considering public models first posses the question: what does it mean to "fine-tune on private data"? who owns this data? this makes the most sense only if the data is held by a collection of parties, since if it's held by only one party, what is preventing this party from performing the fine-tuning themselves? However, in this case, the fine-tuning would this an MPC protocol among who? Who runs the secure computation? In particular, at the end of the fine-tuning the model is "private" (well, part of it), so this probably means shared among at least two parties... who is the client then who performs the query at inference time?

Thank you for your feedback! It seems that these questions arise from a misunderstanding about our work’s objective. We will clarify the objective more explicitly in the introduction, and hope that the following clarifications will address your concerns.

Marill operates in a setting where the foundational model is open-source and the fine-tuning data (which is sensitive or proprietary) is known entirely by a single service provider.

In this setting:

• Fine-tuning is performed locally on the service provider, with no involvement of MPC (see Fig. 1 caption).
• The goal is to leverage the open-source model to achieve better ML performance than a model that is trained on the private data alone. This aligns with the standard workflows of leveraging advanced foundational models.

The resulting model can then be served to any client through a secure inference API.
Using traditional secure inference, the service provider will pay the secure inference cost of the entire foundational model.
With Marill, only a part of the foundational model needs to evaluated within MPC during secure inference without compromising security, and this results in the secure inference performance improvements we report.

The claimed proof in the appendix is very high level and feels more like a "checkbox" rather that anything that is supposed to shed any light. In particular, there are several details unspecified which an actual proof would need to deal with. For instance, the functionality is inherently defined over real values, but all of the protocols cited in the paper operate over fixed point values. There's no way then to instantiate the functionality as written. Furthermore, most of the cited protocols use probabilistic truncation for fixed-point arithmetic, which is not provably secure.

We hope that the above clarification on Marill’s setting helps address some of your concerns about the rigor of the proof.
Besides the issues you’ve highlighted (addressed below), we are happy to hear any additional concerns you may have.

We agree that we should clarify that the ideal functionality $F_A$ runs the fixed-point converted model because that is what all prior works support. Each work also has a slightly different way to perform fixed-point approximations of real values and we can also make that clear. So we will basically prove the following statement:

“Given a secure inference protocol $P$ that securely realizes an ideal functionality $F_A$, where $F$ applies $P$’s fixed-point approximations to the model architecture $A$, Marill, by making black-box calls to $P$ also securely realizes $F_A$”

Given this change, there are secure inference protocols that securely realize some $F_A$ and we can clarify that this proof only applies to those that do.

While it is true that most of the cited protocols use probabilistic truncation for fixed-point arithmetic, we note that Marill does not introduce any changes to the underlying arithmetic operations as it only makes black-box calls to a secure inference protocol. Thus, the security guarantees remain consistent with those established in prior works. We will explicitly state this limitation and reference the potential vulnerability [Li et al.].

Why do you cite [73] as the SOTA for 2PC+Dealer? Earlier in the text you refer to Sigma as such SOTA (which makes sense).

Thank you for your question! We cite CrypTen [73] as a SOTA secure inference framework for 2PC-Dealer because it remains widely adopted in both academia ([1,2,3,4]) and industry. This is largely due to its user-friendly frontend, which seamlessly integrates with PyTorch, enabling secure inference directly on PyTorch code.
At the same time, we also refer to SIGMA as a SOTA secure inference protocol for 2PC-Dealer because it offers the most efficient backend for secure inference.

Marill is complementary to both CrypTen and SIGMA and improves their performance. We report improvements on CrypTen in section 6.1, and Marill’s LF=0.5 + HM=4 configuration improves SIGMA’s pre-filling runtime and communication efficiency by $3.2\times$ and $3.3\times$, respectively. We will include these results in the paper for additional clarity and evidence of Marill’s improvement.

[1] SecFormer. ACL 2024. [2] SAL-ViT. ICCV’23. [3] RNA-ViT. ICCAD’23. [4] MPCFORMER. ICLR 2023.