Thank you so much for your encouragement and recognition of our theoretical contribution! We're happy that our research has raised your interest. We will further refine the detail in our work according to suggestions.

Clarification on "perturbed parameters and embeddings".

The "perturbed embedding" $h^{t+,-}$ refers to "parameter-perturbed embedding", which is obtained by perturbing the client-side parameters during the forward pass.

For server-side updates, we take $h^t$ as input when perturbing the server-side parameters to obtain ZO gradient estimates. Meanwhile, $h^{t+,-}$ is used on server-side to calculate $\delta^t$(Eq. 6), which will be sent back for client-side gradient estimation and model updates. That being said, in our algorithm design, both client and server follow the same unified ZO update protocol.

This also helps explain our statement that “$\delta^t$ obtained from the perturbed embeddings”, since $\delta^t$ is obtained by using $h^{t+,-}$ as input. This information is used for client-side updates and does not require perturbing the server-side parameters.

Hope our explanation addresses your question. Thank you again for your careful review!

Thank you so much for your thoughtful feedback and for recognizing the strengths of our theoretical work. Below, we address your main concern regarding the necessity of incorporating ZOO with our unbalanced update strategy.

On the Role of ZOO in Addressing Straggler Issues:

1. Theoretical dependency of ZOO in solving straggler issue. Our straggler resilience result is derived from the sublinear speed-up convergence rate presented in Corollaries 4.4 and 4.8. Achieving this sublinear speed-up w.r.t the unbalanced update factor $\tau$ requires a coordinated cutting strategy, where the dimension split satisfies $d_c = \sqrt{\frac{d}{\tau}}$ and $d_s = d - \sqrt{\frac{d}{\tau}}$. And this conclusion is only validated under ZOO, since only the convergence rate of ZOO is dependent on dimension $d$. This implies that the mitigation of straggler issue is not merely due to multiple updates, but is tied to the incorporation of ZOO.
2. Practical suitability of ZOO for straggler settings. ZOO also offers practical advantages in straggler scenarios, particularly for memory-constrained edge devices. Compared to FOO, ZOO can significantly reduce memory usage[17, 18]. This aligns well with the realities of resource-constrained edge devices, where memory and computation overhead can introduce time delay that further exacerbates the straggler problem[a,b,c].

More interesting findings uncovered by incorporating ZOO with Unbalanced Update:

1. ZOO enables theoretical interpretation between cutting strategy and unbalanced update. A key novelty of our work lies in providing a theoretical explanation for how the choice of splitting layer coordinates with unbalanced updates $\tau$ affects convergence rate. Compared to FOO, the convergence rate of ZOO is related to the model dimensionality $d$, which helps us link the convergence performance to the cutting layer position. Specifically, adjusting the splitting layer allows us to optimize convergence behavior, which provides a principled guideline for choosing the cutting layer in practice. In contrast, such a theoretical connection between the splitting layer and convergence behavior is difficult to uncover in the FOO setting without the dimension dependency of ZOO convergence rate.
2. Dimension-Free ZOO achieved by Unbalanced Updates. Our work is able to eliminate the dimension dependence in ZOO by employing unbalanced updates. Based on Corollaries 4.4 and 4.8, by appropriately scaling the unbalanced update factor $\tau$ to the same order as the model dimension dimension $d$, the convergence rate becomes independent of $d$. This directly addresses a key limitation of traditional ZOO methods, where convergence rates typically deteriorate with increasing dimension.

Regarding your question about whether our algorithm design is just simply combines ZOO with Unbalanced updates.

Firstly, we would like to clarify that our algorithm does not merely apply conventional ZOO within SFL, but rather introduces a tailored design that explicitly addresses the straggler issue by reducing synchronization delays through a novel Eager forward transmission scheme.

In traditional ZOO, updates often require waiting for all perturbations to be computed before proceeding. We refer to it as the “Lazy” forward. In contrast, our “Eager” forward method follows a sequential update process that effectively reduces waiting time. Specifically, during each forward pass on the client, parameter-perturbed embeddings are immediately transmitted to the server as they are computed, without waiting for the full set of parameter-perturbed embeddings ($h, h^+, h^-$) to complete. For instance, the unperturbed embedding $h$ is first computed and sent to the server, which can immediately begin performing $\tau$ local updates with $h$ as input, while the clients continue computing.

To have a more direct observation about our eager methods in minimizing waiting time, we conduct experiment using OPT1.3B on SST-2, and calculate the required time using different forwarding manner. As shown in Table 4, where we set $\tau=10$, our eager scheme reduces the overall forward time by nearly half. This confirms the effectiveness of our eager update manner in minimizing synchronization delays by utilizing the overlapping computation time.

Table 4 Required Time for Different Forward Manner

Zeroth-order optimization usually imposes multiple rounds of perturbations and hence increase the duration of each round. How is this problem solved in this work?

Towards this question, we believe that our Eager Forward design directly addresses this issue by performing sequential transmission of parameter-perturbed embeddings. This eliminates the need to wait for all perturbations to complete before the server can begin updates, thus significantly reducing time delay caused by multiple perturbations.

Moreover, from a theoretical perspective, multiple perturbations lead to faster convergence rates w.r.t $p$ [18]. Although applying multiple perturbations can introduce time delay for each local round, it can accelerate global convergence, thereby helping to mitigate the straggler issue to some extent.

Regarding non-iid issue:

In this paper, we do not focus on the non-IID issue. Non-IID data is often used to model data heterogeneity, which is indeed an important challenge in federated learning. However, our focus is on the straggler problem, which stems from system heterogeneity [8-14]. This refers to disparities in computational capabilities and network latency among clients, as we stated in the first paragraph of our paper.

We introduce variable time delays across all the clients which is a widely adopted approach to simulate the impact of stragglers in distributed learning system. This method is well-supported in prior literature [8–14]. To establish a more realistic model that reflects the randomness of clients' time delay, we consider an exponential distribution model, which has been widely used to capture the computation delay for distributed clusters [d,e,f].

As we acknowledge the importance of data heterogeneity in federated learning. To evaluate the robustness of our method under non-IID data, we incorporate a high data heterogeneity setup with shard=2.

As shown in Table 5, although all methods experience some performance degradation under non-IID settings, our proposed algorithm continuously outperforms the baseline GAS[8], which is specifically designed for non-IID scenarios. This suggests that our method is not only effective in mitigating straggler effects but also exhibits robustness to data heterogeneity.

Table 5 Impact of data heterogeneity on SFL performance

Questions about performance boost:

To avoid any misunderstanding, we would like to first clarify that, as stated in Line 276, we reimplemented all baselines using ZOO to ensure a fair comparison across methods.

To address your concern, we conducted an ablation study on the SST-2 dataset using OPT-1.3B. Specifically, we implemented MU-Split in both its FO and ZO versions. This simplified setup includes only one client and one server, isolating the impact of the unbalanced update strategy. As shown in Table 6,7, we observe consistent performance improvement in both FO and ZO versions, indicating that the accuracy gain primarily stems from the unbalanced server update schedule. This further confirms the effectiveness of our proposed unbalanced update strategy in enhancing convergence and accuracy.

Table 6 Best accuracy after 100 rounds of MU-Split(OPT-1.3B, Zeroth Order)

Table 7 Best accuracy after 100 rounds of MU-Split(OPT-1.3B, First Order)

a. “Thinking Forward: Memory-Efficient Federated Finetuning of Language Models”

b. “Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Training”

c. “Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting”

d. “Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity”

e. "Speeding up distributed machine learning using codes."

f. "Coded computation over heterogeneous clusters.”

We really appreciate the time and effort you dedicated to reviewing our work and providing such valuable insights! To facilitate a direct comparison, we have added Table 3, which includes the convergence rates and communication complexity of our proposed MU-SplitFed alongside the SplitFed algorithms in [2] and other FedZO methods.

Table3 Convergence/Communication Complexity Comparison Result

Weakness 1:

We initially avoided a direct rate comparison with [2] because their analysis is grounded in first-order optimization (FOO), while our work is based on zeroth-order optimization (ZOO). Given the fundamentally different gradient estimation paradigms, we considered the comparisons less direct. However, we agree that contrasting them will offer more valuable insights, and Table 3 now addresses this.

While [2] provides an important theoretical foundation for SFL, we note two key limitations in their analysis:

1. Conflict in the Strongly Convex Case: The results in [2] for the strongly convex setting rely on the assumption of bounded gradients. However, this assumption contradicts strongly convex, making the conclusions theoretically questionable.
2. Loose Bounds and Strong Assumption in the Non-Convex Case: Although [2] does present convergence results for the non-convex case, it relies on stronger assumptions (e.g., bounded gradients), whereas we use bounded variance. Despite using milder assumptions, our analysis achieves a sublinear speed-up w.r.t $\tau$, attributed to our use of more advanced techniques such as properties of martingale difference sequence and tighter equality-based estimations. Interestingly, [2] allows for multiple local updates in each round that should intuitively accelerate convergence. However, their theoretical bounds do not reflect this benefit, suggesting room for tightening.

In contrast, our analysis reveals that both increasing $\tau$ and involving more clients contribute to convergence rate. This reflects a more refined understanding of the interplay between local computation and global coordination in SFL.

Lastly, thank you for catching our inaccurate phrasing in Lines 51–52. We will revise that section to more precisely reflect the limitations of [2] in the updated version of our paper.

Weakness 2:

We have now included a detailed comparison of both convergence rates and communication complexities between our proposed method and existing ZOO approaches in federated learning, including one of the earliest works combining ZO with FL [a], and DecomFL [18]. The results are also presented in Table 3.

From this comparison, we observe that our algorithm achieves a convergence rate comparable to those of existing FedZO methods. The key distinction lies in how acceleration is achieved:

While FedZO and DecomFL rely on increasing the number of local update steps $K$ to accelerate convergence, our method introduces a novel acceleration mechanism via our unbalanced update strategy parameterized by $\tau$.

To isolate the effect of $\tau$, we consider the setting where $K=1$ across all methods. In this scenario, our algorithm exhibits a sublinear speed-up w.r.t $\tau$, highlighting the role of unbalanced updates in improving convergence. Importantly, our method does not impose any restrictive assumptions that would hinder generalization. This allows our framework to naturally extend to $K > 1$, enabling multiple local updates and potentially further enhancing convergence.

In summary, this comparison highlights the advantage of our framework in exploiting the unbalanced update ratio $\tau$ to improve convergence, while remaining compatible with and extensible to broader settings explored in the FedZO literature.

Weakness 3:

Regarding your increased communication concern (i):

We would like to clarify two key facts:

1. Per-round backward communication is reduced in MU-SplitFed. In our proposed algorithm, during the forward pass, the client transmits $H_m^t=h^t, h^{t+},h^{t-}$, which involves three vectors. However, in the backward pass, the client only transmits a scalar value $\delta_{c,m}^t$ as defined in Eq. (6). In contrast, in SplitFed-V1 [2], the backward pass requires transmitting the the partial gradient w.r.t. $h^t$. As such, although our forward pass involves slightly more communication, the backward communication is significantly lighter, making the total communication per round comparable.
2. Our total communication complexity is theoretically lower compared to [2]. As shown in Table1, our method exhibits a lower overall communication complexity despite slightly higher per-round communication. This is due to our unbalanced update strategy, which plays an important role in reducing the number of global communication rounds. In comparison, the method in [2] incurs a bad communication complexity even fail to reveal the accelerating of $K$ and $M$. Thus, our approach achieves a reduction in total communication cost over the entire training process.

Regarding your question(ii) about client local update steps:

1. Simplifying the analysis with $K=1$. To make our unbalanced update strategy easier to understand and analyze, we set $K=1$ in our current framework. However, our method is not limited to this setting. It can be naturally extended to support $K>1$. In such cases, during each local $K=j$ round, the client and server would still apply our proposed unbalanced update strategy. After completing $n$ local rounds, the client and server would perform global aggregation via FedServer. This generalization preserves the structure of our method while achieving accelerated convergence.
2. No increase in communication complexity. Even with $K=1$, our method does not suffer from increased communication cost. As shown in Table 3, the overall communication complexity remains comparable to other FedZO methods. This is because our server-side update parameter $\tau$ plays a similar role to $K$ in reducing the frequency of global communication. Furthermore, extending our method to support $K>1$ is expected to further improve both convergence rate and communication efficiency.
3. Lower communication cost for model aggregation. We understand that your concern is because global communication costs are comparably higher than local communication costs. However, our algorithm is designed to substantially reduce the communication cost of model aggregation. In particular, it allows us to allocate only a small portion of the model to the client (typically we choose $L_c=2$). This is supported by our theoretical analysis: by tuning $\tau$, we can effectively minimize the client-side model size. Additionally, experimental results indicates that our unbalanced update strategy also lead to performance boost(see Supplementary Table 6, 7). In other words, our approach enables the use of a smaller client-side model while still maintaining competitive accuracy, thereby reducing the overall cost of global aggregation. In contrast, Han et al. [2] suggest that a larger portion of the model should be put on client to achieve decent accuracy, which inherently requires higher global communication cost.

Response to Questions:

Q1: Thank you for pointing out this inconsistency, and we apologize for the misleading description. The reason we originally referred to concatenation is that certain SFL designs [8,b] use concatenation of activation vectors instead of aggregation. However, we acknowledge that our current implementation follows the aggregation-based approach. To accurately represent our design, we will revise the sentence as follows in the updated version: “The straggler issue only delays the global aggregation in traditional FL. In contrast, in SFL, it also impacts local updates due to SFL's relay-based update mechanism. This motivates us to revisit straggler solutions tailored for SFL.”

Q2: Thank you for your comment. To achieve a tighter bound, we apply an equality in the derivation. However, since the final result is expressed as “less than or equal to,” using the inequality is technically not incorrect.

Q3: $f_\lambda^t$ and $f_\lambda^{t, i}$ are not equivalent. Specifically, $f_\lambda^t$ denotes the loss function at the t-th global training round, while $f_\lambda^{t, i}$ denotes the loss function at the t-th global training round and i-th server iteration. We will clarify this notation more explicitly in the revised version.

Q4: Thank you for catching this typo, we appreciate your careful reading. We will correct it in the final version.

Q5: Thank you for your comment. We would like to clarify that we did not apply the equation you mentioned in deriving the first inequality of Equation (32). Instead, our analysis follows the martingale-based approach under conditional expectation, consistent with the second situation in Lemma 4 of SCAFFOLD paper[c]. However, we acknowledge that a scaling factor of 2 was omitted in front of the inequality, and we will correct this in the revised version.

We will follow your suggestions to refine our work in our future version.

Supplement references:

a. “Communication-efficient stochastic zeroth-order optimization for federated learning”

b. “SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustments”

c. “SCAFFOLD: Stochastic Controlled Averaging for Federated Learning”

1. Which algorithm does MU-SplitFed correspond to in SplitFed? SplitFedv1 or SplitFedv2?

2. Does this paper include the first order methods with unbalanced updates in SplitFed for now? If it does, this is an important contribution for me.

3. I am not convinced that there is no $O(\eta^2)$ terms in the bound in Theorem 4.7, which is common in FedAvg. See Yang et al. (2021); Karimireddy et al (2020). I only see the $O(\eta)$ terms in the bound.

   Yang et al., Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning, ICLR, 2021.

   Karimireddy et al., SCAFFOLD: Stochastic Controlled Averaging for Federated Learning, ICML, 2020.

4. It seems that it uses $\eta_g = \frac{1}{\sqrt{M}}$ in Corollary 4.8. For comparison, if the function of $\eta_g$ is the same for all algorithms, I recommend using the same $\eta_g$ for all the algorithms. This is because $\eta_g$ makes a big difference to the bounds. As shown in Cheng et al. (2024)'s Table 1 (the reproduced bound of Yang et al. (2021)), when $\eta_g \to \infty$, FedAvg is even immune to the heterogeneity.

5. The claims in the rebuttal for $f_\lambda^t$ and $f_\lambda^{t,i}$ are not uniform. Note that function $f$ is related to the model parameters and data points; when the data or parameters change, the function also changes.

Note that the questions with respect to the bounds are mostly from Table 1. I do not met the bounds (that I expected) with the form like Cheng et al. (2024). These bounds seem unfamiliar to me, which makes me a little puzzled.

Cheng et al., Momentum Benefits Non-Iid Federated Learning Simply and Provably, ICLR, 2024.

At last, some notations are not displayed well in openreview, which can be adjusted by adding \.

The responses seem reasonable. I realize that I may not be as familiar with the existing works in split learning as I thought I was.

The concern about the tightness of the bounds (i.e., $O(\eta^2)$ terms) still exists. This is mainly because the original SplitFedv1 (or SFL_v1) (in Thapa et al, (2022)) is equivalent to FedAvg in the absence of model partitioning. So, in my opinion, the bounds of SplitFed may be quite similar to those of FedAvg. However, this does not seem to be the case, based on the existing works (Han et al., 2024) and this paper.

As a consequence, I decide to increase the rating to 4, and decrease the confidence to 3, to avoid the rejection of this paper due to my unfamiliarity with the existing works in split learning.

Some suggestions are given:

1. I hope the authors can consider the concern more deeply in the revised version. Some intuitive discussion is still required.
2. I also hope the authors consider the form of Koloskova et al. (2020); Cheng et al. (2024) in the revised version. As you have said, they consider the other factors (beyond the training rounds, local update steps, and the number of clients), which is more professional and convincing for me.
3. In addition, try to avoid using "communication round" in split learning, as communication also exists in the stage of local updates. See Thapa et al, (2022); no "communication round" is used. One alternative can be "training round".
4. At last, I hope the authors revise the paper as promised.

Thapa et al., SplitFed When Federated Learning Meets Split Learning, AAAI, 2022.

Han et al., Convergence Analysis of Split Federated Learning on Heterogeneous Data, NeurIPS, 2024.

Koloskova et al., A Unified Theory of Decentralized SGD with Changing Topology and Local Updates, ICML, 2020.

Cheng et al., Momentum Benefits Non-Iid Federated Learning Simply and Provably, ICLR, 2024.

Response to Reviewer's Novelty Concerns:

We believe that our work identifies and addresses fundamental challenges in Split Federated Learning (SFL) that have not been properly analyzed in prior work.

The Critical Challenge in SFL:

The goal of Split Federated Learning (SFL) is to reduce memory usage on resource-constrained edge devices by dividing the model between the client and the server. However, this model partitioning introduces additional communication overhead during each local update round. As a result, SFL faces a fundamentally different straggler problem: unlike traditional Federated Learning (FL), where stragglers only delay global aggregation, stragglers in SFL block every local iteration, significantly hindering the training progress.

Existing work on addressing straggler issue in SFL remains limited[8,9,16]. Most approaches borrow straggler mitigation strategies from FL, primarily focusing on minimizing server waiting time. However, under the relay-based update mechanism in SFL, this strategy proves less effective, yielding only marginal improvements. These limitations highlight the need to revisit the straggler problem in SFL.

Our Novel Solution - MU-SplitFed:

Acknowledging the limitations of current work, we propose MU-SplitFed that combines unbalanced updates with ZOO to address straggler issue from a fundamentally different perspective-by effectively reducing the number of global communication rounds. For further improvement, we introduce an eager forward method that minimizes server waiting time in each local round. The effectiveness of this method in reducing time delay has been empirically validated in Table 4.

Contribution I: Solving Straggler Issues in SFL

1. Improved Convergence with Sublinear Speedup: Our MU-SplitFed provides sublinear speed-up regarding unbalanced update steps, parameterized by $τ.$ Specifically, we improve the convergence rate from $O(\frac{\sqrt{d}}{\sqrt{T}})$ to $O(\frac{\sqrt{d}}{\sqrt{\tau T}})$, without relying on strong assumptions such as bounded gradients.
2. Theoretical Solution to Straggler Problem: We provide direct theoretical insight into how our unbalanced strategy effectively solves the straggler issue under Corollary 4.8, addressing the fundamental communication bottleneck in SFL.
3. Practical Effectiveness: Our extensive experimental results demonstrate the advantage of our proposed framework in both small models and LLMs. As indicated in supplementary Table 2, our proposed unbalanced update can reduce more than half of the communication rounds, demonstrating the effectiveness of our algorithm in practical application.

Contribution II: Additional Theoretical Insights

1. Principled Model Splitting Strategy: Our work reveals important theoretical interpretation regarding the choice of cutting position, which coordinates with our unbalanced update strategy. Our theory indicates that for a larger choice of $τ$, more layers should be placed on the server side. The dependency between the convergence rate of ZOO and parameter dimension $d$ allows us to build this connection, enabling our theory to provide guidance for hyperparameter selection and making our algorithm more explainable.
2. Dimension-Free ZOO: Our unbalanced update strategy can achieve dimension-free ZOO without further assumptions. Based on Corollaries 4.4 and 4.8, by appropriately scaling the unbalanced update factor $τ$ to the same order as the model dimension $d$, the convergence rate becomes independent of $d$. This directly addresses a key limitation of traditional ZOO methods, where convergence rates typically deteriorate with increasing dimension.

In summary, MU-SplitFed introduces a fundamentally new perspective on handling stragglers in SFL. We believe our contributions, both theoretical and empirical, establish a strong case for novelty and impact in this underexplored space.

Regarding your doubt of “straggler scenario are limited”

We need to clarify that introducing time delay is a widely adopted approach to simulate the impact of stragglers in distributed learning. This method is well-supported in prior literature [8–14]. To establish a more realistic model that reflects the randomness of clients' time delay, we consider an exponential distribution model, which has been widely used to capture the computation delay for distributed clusters

Here are more works that use this methodologies:

a. “Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity”

b. "Speeding up distributed machine learning using codes."

c. "Coded computation over heterogeneous clusters.”

Regarding your doubt about lack of comparison with other SOTA methods

Thank you for your suggestion. However, we must clarify that related work addressing straggler issues specifically in the Split Federated Learning setting is extremely limited. We have identified only three recent works tackling straggler issues in SFL [8,9,16] and we provide detailed justification for our comparison choices below. [8]: is compared in our manuscript [16]: This paper proposes a ring-structured SFL framework, which fundamentally differs from the standard SFL architecture. We believe that comparing algorithms under different ML system structures would not provide meaningful insights.

[9]: While this work addresses straggler issues in SFL, it has significant limitations that make it less effective than our approach. The paper proposes an adaptive splitting strategy that dynamically adjusts layer allocation between clients and server. Specifically, the model is split into parts A, B, and C, where B consists of m+n layers. The client computes A+m layers (while still need to contain A+B), the server computes n+C layers (while still need to contain B+C), and m and n are dynamically adjusted based on client computational capacity.

However, we identify two fundamental limitations of this method:

1. Additional computational overhead: The dynamic layer adjustment requires continuous monitoring and reallocation, introducing extra computation time on the server side.
2. Limited improvement potential: The effectiveness is constrained by the fixed size of the overlap region B. Since B's size is fixed, the potential improvement is inherently bounded and comparatively small.

In contrast, our MU-SplitFed addresses these limitations through a fundamentally different approach: we directly reducing communication rounds rather than merely minimizing client computation time. Our key insight is that the computation disparity between client and server is considerably large in SFL. Instead of optimizing the already limited client-side computation time, our framework intelligently utilizes the server's idle time by applying multiple updates strategy. As demonstrated in Table 2, our method achieves more than 50% reduction in communication rounds, which is significantly more effective for overall training time reduction.

If you are aware of other SOTA methods for straggler issue in SFL, please feel free to recommend them to us. We would be happy to conduct additional experiments for further comparison.

Regarding your request about “The experiments also should be conducted on more complex dataset and LLMs.”

Thank you for your suggestion. In response, we have conducted and included additional experiments across a variety of datasets and model architectures, as detailed below:

1. LLM Benchmarks with OPT-125M (Table 1). We evaluated our proposed MU-SplitFed against SplitFed_V1 [2] using the OPT-125M model on multiple LLM benchmarks, including SST-2, CB, WSC, WIC, RTE, and BoolQ. With $L_c=2$ and $\tau=2$, MU-SplitFed consistently outperforms SplitFed_V1 across all tasks, demonstrating its effectiveness under various language understanding challenges.
2. Ablation Study with OPT-1.3B (Table 2). We conducted an ablation study on the SST-2 dataset using OPT-1.3B to assess the impact of $\tau$ and $L_c$ in reducing the communication rounds. According to the result, for a large choice of $\tau$, less portion of the model should be assigned to client side. This aligns with our theoretical analysis on the interplay between unbalanced update steps and splitting strategy.
3. Image Classification under Data Heterogeneity (Table 5). To evaluate the performance of our proposed algorithm under data heterogeneity, we added experiments on CIFAR-10 and FashionMNIST under non-IID setting with high heterogeneity (shard=2). The result shows that MU-SplitFed outperforms the baseline, indicating the robustness of our designed algorithm under data heterogeneity.
4. Ablation Study on Update Strategy (OPT-1.3B, SST-2). To investigate the reason of performance boost, we compared both FO and ZO versions of MU-Split on the SST-2 dataset. Specifically, we implemented MU-Split in both its FO and ZO versions. As shown in Table 6,7, we observe consistent performance improvement in both FO and ZO versions, indicating that the accuracy gain primarily stems from the unbalanced server update schedule. This further confirms the effectiveness of our proposed unbalanced update strategy in enhancing both the convergence and accuracy.

Table 1 Best accuracy after 1500 rounds (OPT125M, Zeroth Order)

Table 2 Required rounds to reach 85% accuracy of MU-Split (OPT-1.3B, Zeroth Order)