CryptoMoE: Privacy-Preserving and Scalable Mixture of Experts Inference via Balanced Expert Routing

We thank Reviewer d2x4 for your valuable and detailed feedback! Below we list our responses to each of the comments:

[To Q1 & W1: Comparison with BOLT (BSGS), Bumblebee, Nexus and CipherGPT on batch MatMul]

Our batch MatMul packing strategy is fully compatible with the Baby-Step Giant-Step (BSGS) algorithm. As analyzed in Appendix E, we provide a detailed complexity analysis for integrating BSGS with our batch packing method. Specifically, BSGS combined with batch packing reduces the number of required rotations from $O(n\sqrt{td_1 d_2/N})$ to $O(\sqrt{ntd_1 d_2/N})$, yielding a $\sqrt{n} \times$ reduction in rotation complexity. In our implementation, we consistently apply the BSGS optimization. For clarity, Figure 6 in our paper reports results without BSGS, but Appendix E covers both settings.

In the table below, we compare our batch MatMul protocol against existing approaches, including BOLT [r24], NEXUS [r37], and CipherGPT (VOLE-based) [r44]. All evaluations are conducted on a machine equipped with an Intel Xeon Platinum 8468 CPU (48 cores @ 2.1GHz, 64 threads) under a LAN setting. Given ciphertext matrices $\\{A_i\\}\_{i=0}^{n-1} \in \mathbb{Z}^{t \times d_1}$ and plaintext weights $\\{B_i\\}_{i=0}^{n-1} \in \mathbb{Z}^{d_1 \times d_2}$, we compute $C_i = A_i \times B_i$ for $n = 64$, $t = 24$, $d_1 = d_2 = 2048$. These dimensions are derived from the DeepSeekMoE-2.8B/16.4B models.

We follow the HE parameter settings used in Bumblebee, namely: $N = 8192$, $\log q = 114$, and $\log p = 64$, where $N$ is the polynomial degree, and $q$, $p$ are the ciphertext and plaintext moduli, respectively. For secret sharing, we adopt a 64-bit ring.

The communication cost and computational latency for CipherGPT are estimated following their paper, as their code is not open-source. It is important to note that the MatMul protocols proposed by CipherGPT and Nexus are optimized for scenarios involving different ciphertext input matrices $A_i$ multiplied by the same plaintext weight matrix $B$. However, in the batch MatMul of the MoE block, different experts process distinct ciphertext tokens with distinct weight matrices. This means we cannot leverage the batch amortizable properties that CipherGPT and Nexus offer, resulting in their method incurring higher latency.

The results show that our batch matrix multiplication protocol, specifically optimized for MoE layers, performs significantly better than the methods discussed above.

[To Q2 & W2 & Limitation 1: Concerns about performance and reproducibility] We clarify that the reported inference time and communication cost in Table 3 are amortized per-token costs, i.e., the total cost divided by the number of input tokens. When considering this, our results are consistent with prior works such as BOLT [r24] and Bumblebee [r31]. We include a table below summarizing the per-token cost across BOLT, Bumblebee and NEXUS for clarity.

We would like to emphasize that prior works such as BOLT and Bumblebee focus on supporting dense models (e.g., BERT, LLaMA-1). In contrast, our core contribution lies in proposing the first framework that supports privacy-preserving inference for MoE models. For other components, such as RMSNorm and attention, we directly adopt the protocols from prior works BOLT and Bumblebee. Specifically, for matrix multiplications in linear layers, we leverage BOLT’s protocol; for nonlinear operations, including RMSNorm, Softmax, and SiLU, we employ the protocols proposed in Bumblebee. We acknowledge that these aspects are not the primary focus of our work. In addition, the communication overhead of MoE models is relatively higher than that of dense models, primarily due to the additional communication introduced by the dispatch and combine protocols. However, MoE models reduce the overall computation per expert, leading to lower latency and thus offering better efficiency in terms of inference time.

As for our HE/MPC parameter configuration, we follow the settings in Bumblebee: we use $N = 8192$, $\log q = 114$, and $\log p = 64$, where $N$ is the polynomial degree, and $q$, $p$ are the ciphertext and plaintext moduli, respectively. For secret sharing, we adopt a 64-bit ring. We will include these details in the revised version to improve clarity.

Regarding code availability, we understand that an open-source release is vital for independent verification and fostering adoption. Our current code framework is built upon Secretflow SPU, and we are actively in the process of organizing and refining it for public release. We are fully committed to releasing a reference implementation of our code once our paper is accepted.

[To Limitation 2: Assumptions on the Adversarial Model and Multi-Party Scalability]

Our work focuses on the two-party setting under the semi-honest threat model, which is a common and practical assumption in privacy-preserving machine learning. Extending the framework to a malicious adversarial model or to a multi-party setting falls outside the current scope of our study. Nevertheless, both directions are important and promising. As noted in BOLT, designing protocols that are secure against malicious adversaries is far from trivial. Similarly, supporting multi-party computation would require replacing the underlying MPC primitives we currently employ, which we also consider a valuable avenue for future work. We will add a discussion of these points in the final version to enhance completeness.

[To Paper Formatting Concerns]

Thank you for your careful review and helpful suggestions.

Regarding the notation inconsistency between "top-t" and "top-k", we clarify that our intention was to use "top-k" consistently to refer to the general protocol for selecting the top-$k$ elements (e.g., $\Pi_{\text{top}k}$). The occurrence of "top-t" in line 55 was meant to emphasize that each expert receives $t$ tokens during dispatch. We will standardize the notation to "top-k" throughout the paper and explicitly state $k = t$ in relevant contexts to eliminate ambiguity.

For Figure 8, the reported speedup factors (e.g., 3×, 2×) refer to the performance gain of our proposed method compared to the dense baseline. The figure caption will be revised to use the plural form "settings" in the final version.

We will revise the final version to improve the consistency and clarity of notation and figures to avoid misunderstandings.

References

• [r24, r31, r37, r44] represent the 24th, 31th, 37th, 44th references cited in our original submission.

We sincerely appreciate the detailed advice from reviewer d2x4! We take all your suggestions into account in our final version. Please let us know if you have any further questions. If you feel that the above improvements help clear up your doubts and further improve our work, you can kindly consider a re-evaluation, thank you!

We thank Reviewer 8zVq for your support and for the professional and detailed feedback! Below is our response.

[To W1: Compatibility with BOLT BSGS]

Our batch MatMul packing strategy is compatible with the Baby-Step Giant-Step (BSGS) algorithm. As detailed in Appendix E of our paper, we provide a complexity analysis of integrating BSGS with our packing. Specifically, BSGS combined with batch packing reduces the number of required rotations from $O(n\sqrt{td_1 d_2/N})$ to $O(\sqrt{ntd_1 d_2/N})$, resulting in a $\sqrt{n} \times$ improvement in rotation complexity.

[To W2: Cost of repacking for batch MatMul packing]

We would like to clarify that both the input and output of our dispatch and combine protocols are in secret share (SS) form, not HE ciphertext. As shown in Algorithms 1 and 2 in Appendix C, under the SS form, the repacking operations can be performed locally and in plaintext, incurring negligible cost. Thus, our batch MatMul format does not introduce extra packing cost in the HE computation.

[To W3: Setup details for end-to-end evaluation]

All components of our Transformer model—including RMSNorm, attention, and MoE blocks—are evaluated securely under our HE/MPC hybrid framework, ensuring end-to-end privacy. The reported inference time and communication cost in Table 3 are amortized per-token costs, i.e., total cost divided by the number of input tokens. When considering this, our results are consistent with prior works such as BOLT [r24] and Bumblebee [r31]. We include a table below summarizing the per-token cost across BOLT and Bumblebee for clarity.

We would like to emphasize that prior works such as BOLT and Bumblebee focus on supporting dense models (e.g., BERT, LLaMA-1). In contrast, our core contribution lies in proposing the first framework that supports privacy-preserving inference for MoE models. For other components, such as RMSNorm and attention, we directly adopt the protocols from prior works BOLT and Bumblebee. Specifically, for matrix multiplications in linear layers, we leverage BOLT’s protocol, and for nonlinear operations, including RMSNorm, Softmax, and SiLU, we employ the protocols proposed in Bumblebee. We acknowledge that these aspects are not the primary focus of our work.

Furthermore, as shown in Figure 8, when the total number of input tokens is relatively small (e.g., typically under 256 tokens), the MoE layer becomes the dominant bottleneck in both LAN and WAN settings. This observation further highlights our key contribution: enabling efficient private inference for the MoE module.

We will incorporate this clarification in the revised version to avoid potential confusion. We hope this addresses your concerns and provides a clearer understanding of the scope and focus of our design and performance contributions.

[To Typos]

Thank you for pointing out that "Threat Model" was misspelled as "Thread Model". We will correct this in our final version.

References

• [r24, r31] represent the 24th, 31th references cited in our original submission.

Last, thanks so much for helping us improve this work through your professional perspective!

Thank you very much for your support and constructive comments, which have greatly helped us improve our work! See below for the answers to your questions and comments.

[To Q1 & W2: Details on the dense baseline implementation]

We provide a detailed implementation of the dense baseline below, which complements the brief description in Section 3 of our paper. The dense baseline evaluates all experts for each token, thereby fully hiding the routing information. The core steps are as follows:

• Routing and Score Matrix Construction: After the gate routing step, we obtain the expert indices and scores $K, W \in \mathbb{Z}^{m \times k}$, where $m$ is the number of tokens and $k$ is the number of selected experts per token. We then convert $K$ into a one-hot representation as follows: $K_{\text{onehot}} = \text{onehot}(K) \in \mathbb{Z}^{m \times k \times n}$, where $n$ is the total number of experts. Within the $n$-dimension of $K_{\text{onehot}}$, only the entries corresponding to the selected expert indices are set to 1, and all others are 0. We then compute the expert score matrix: $S = W \times K_{\text{onehot}} \in \mathbb{Z}^{m \times n}$, where the multiplication reduces along the $k$-dimension, and $m$ remains the parallel axis. Each row $S_i$ reflects the expert scores for the $i$-th token: the positions corresponding to selected experts contain their routing scores, while the rest are 0.
• Expert Computation: Each expert $E_i$ computes on all tokens: $y_{E_i} = E_i(x) \in \mathbb{Z}^{m \times d}$, aggregating all expert outputs gives: $y_E \in \mathbb{Z}^{n \times m \times d}$.
• Combination: Finally, we compute the output of the MoE layer as: $y = S \times y_E \in \mathbb{Z}^{m \times d}$, where the multiplication reduces over the expert dimension $n$, and $m$ remains the parallel dimension. This operation essentially computes a weighted sum over expert outputs for each token.

The corresponding privacy-preserving inference algorithm is described below:

Algorithm: Dense Baseline Input: Secret-shared token embeddings $[\\![x]\\!] \in \mathbb{Z}^{m \times d}$, with $m$ tokens and $d$-dimensional embeddings; $k$ is the number of selected experts. Output: Secret shares of MoE output $[\\![y]\\!] \in \mathbb{Z}^{m \times d}$.

1. Use $\Pi_{\text{matmul}}$, $\Pi_{\text{softmax}}$, and $\Pi_{\text{topk}}$ to perform secure routing and obtain $[\\![K]\\!], [\\![W]\\!] \in \mathbb{Z}^{m \times k}$.
2. Compute outputs for all experts: $[\\![y_{E_i}]\\!] = E_i([\\![x]\\!]) \in \mathbb{Z}^{m \times d}$ for $i \in [0, n-1]$; reshape to $[\\![y_E]\\!] \in \mathbb{Z}^{n \times m \times d}$.
3. Apply $\Pi_{\text{onehot}}([\\![K]\\!], c=n)$ to obtain one-hot routing matrix $[\\![\text{onehot}(K)]\\!]^B \in \mathbb{Z}^{m \times k \times n}$.
4. Compute expert score matrix: $[\\![S]\\!] \leftarrow \Pi_{\text{matmul}}([\\![W]\\!], [\\![\text{onehot}(K)]\\!]^B) \in \mathbb{Z}^{m \times n}$.
5. Compute MoE output: $[\\![y]\\!] \leftarrow \Pi_{\text{matmul}}([\\![S]\\!], [\\![y_E]\\!]) \in \mathbb{Z}^{m \times d}$.

[To Q2: Why CryptoMoE outperforms the dense baseline]

As shown in the ablation study in Figure 8, CryptoMoE significantly reduces the cost of expert computation compared to the dense baseline. In the dense baseline, each expert processes all $m$ tokens, leading to a high computational cost. In contrast, $\text{CryptoMoE}\_{t=2.0}$ reduces this to $2mk/n$ tokens per expert. Taking DeepSeekMoE-2.8B/16.4B as an example, with $m=128$, $k=6$, and $n=64$, each expert processes only $2 \times 128 \times 6 / 64 = 24$ tokens, whereas the dense baseline requires computing all $128$ tokens. CryptoMoE saves over $500\\%$ of the expert computation cost, which is the main bottleneck in the MoE block. Although CryptoMoE introduces additional operations such as Top-K and one-hot encoding, these account for only 18% of the total LAN latency, as shown in Figure 8. Therefore, the savings in the expert layers far outweigh the cost in $\Pi_{\text{dispatch}}$ and $\Pi_{\text{combine}}$.

[To W1: Dispatch and combination protocols are designed using existing building blocks.]

Our $\Pi_{\text{dispatch}}$ and $\Pi_{\text{combine}}$ protocols are built upon well-established cryptographic primitives [r46]. Our key contributions lie in the token manipulation strategy designed specifically for secure MoE inference, which enables secure token-to-expert assignment and result aggregation without leaking any sensitive information. The underlying cryptographic primitives, such as comparison protocols, are not our focus.

To highlight the novelty and effectiveness of our $\Pi_{\text{dispatch}}$ and $\Pi_{\text{combine}}$ protocols, we compare them with the protocol used in CipherPrune [r22]. Prior works [r22, r24] rely on MPC-based sorting that directly swaps high-dimensional token embeddings, resulting in significant communication and latency cost. This challenge is especially pronounced in MoE blocks, where sorting is required in both the dispatch and combine phases for each expert.

In contrast, CryptoMoE eliminates the need for direct manipulation of embeddings by decoupling token indices from embeddings. This design significantly reduces the communication cost. The table below compares CryptoMoE and CipherPrune on a single MoE block of DeepSeekMoE under a LAN setting, where $m$, $n$, $k$, $t$, and $d$ represent the number of input tokens, number of experts, number of selected experts, dispatched tokens per expert, and embedding dimension, respectively. The evaluation is conducted using the following parameters: $m=128$, $n=64$, $k=6$, $t=12$, $d=2048$.

More detailed analysis is provided in Appendix D in our paper.

References

• [r22, r24, r46] represent the 22th, 24th, 46th references cited in our original submission.

We sincerely appreciate the professional, detailed advice from reviewer NVGY! We hope this response fully addresses your concerns.

Thank you very much for your strong support and the thorough and creative comments! We also appreciate your evaluation of our work as “a very interesting research problem.” Please find our responses to your questions below.

[To Q1: Details on the framework’s communication flow]

Our framework adopts the hybrid HE/MPC scheme, a widely used approach in recent secure ML research [r24, r25, r26, r27, r28]. In this scheme, Homomorphic Encryption (HE) is used to efficiently handle linear layers, while Multi-Party Computation (MPC)—including secret sharing (SS) and Oblivious Transfer (OT)—is employed for non-linear operations. This hybrid design involves communication in the following two main cases:

• HE–SS Conversion: Communication arises at the interface between linear and non-linear layers, where HE ciphertexts must be converted to/from SS form.
• MPC Protocol Execution: Communication is inherent to executing MPC protocols for non-linear functions. In particular, Oblivious Transfer (OT), a core building block for non-linear computation, requires interactive communication between the client and the server.

Figures 4 and 5 in our manuscript illustrate how HE and MPC protocols are integrated into our dispatch and combine steps. During expert computation, the SiLU activation is performed via MPC, while the surrounding linear matrix multiplications are handled by HE.

Your analysis of the dense baseline is correct. The dense baseline involves substantial communication cost because each expert processes all $m$ tokens, producing a large volume of intermediate results. These results then require conversion between HE and SS representations, causing significant HE ciphertext transmission. Additionally, the dense computation increases both linear and non-linear processing and communication costs.

[To Q2 & W1: HE parameter settings]

We follow the configuration of Bumblebee for HE parameters and set $N=8192$, $\log q=114$, and $\log p=64$, where $N$ is the polynomial degree, and $q$, $p$ are the ciphertext and plaintext moduli, respectively. For secret sharing, we adopt a 64-bit ring.

[To Q3: Explanation on the dispatch protocol]

The primary function of secret sharing in our protocol is to ensure data privacy: neither the client nor the server can access the full data independently. As shown in Figure 4, we process $m$ tokens of hidden dimension $d$, resulting in data of shape $m \times d$. This data is converted into secret shares to maintain privacy. Throughout our protocol, all intermediate results are represented in secret-shared form.

Regarding your suggestion to pre-split tokens into $m$ shares for weighted-sum computation at each expert—this is a very insightful idea. However, due to two critical constraints:

• Each expert applies a SiLU non-linear activation, which makes pre-aggregation of experts' weights infeasible.
• The expert weights must also remain private, and thus cannot be revealed or used to compute a public weighted sum.

As a result, we must design a secure $\Pi_{\text{dispatch}}$ and $\Pi_{\text{combine}}$ protocol based on secret sharing. Your idea would be well-suited to purely linear pipelines, where pre-aggregation of weights can be applied in a reparameterization-style approach as in [a1].

We hope this clarifies the necessity and role of secret-shared tokens in our protocol.

[To W2: General HE computation protocol]

For linear layer computations, we adopt the MatMul protocol described in Section 4.1.1 of BOLT [r24]. Furthermore, for matrix multiplication within the MoE block, we apply an additional batch optimization on top of the BOLT protocol. We will incorporate a more detailed explanation of the BOLT-based HE protocol in the appendix—thank you for the suggestion!

[To formatting concerns]

Thank you for pointing out the formatting of $\bf{1}\\{\mathcal{P}\\}$ in the first paragraph of Section 2. We use $\bf{1}\\{\mathcal{P}\\}$ to denote an indicator function, which returns 1 when condition $\mathcal{P}$ is true and 0 otherwise. For instance, ${\bf{1}}\\{x == 0\\}$ evaluates to 1 if $x = 0$, and 0 otherwise. We will revise the notation for better clarity and avoid confusion in the final version.

References

• [r24, r25, r26, r27, r28] correspond to references 24 through 28 in our original submission.
• [a1] Ding, Xiaohan, et al. RepVGG: Making VGG-style ConvNets Great Again. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

Thank you again for your strong support and the valuable comments, which have strengthened and made our paper more robust! We will add all of these discussions to our final version.