You Only Communicate Once: One-shot Federated Low-Rank Adaptation of MLLM

We sincerely appreciate Reviewer Cz1u for the insightful comments and constructive questions. We have provided comprehensive responses to clarify these points and look forward to further dialogue and feedback.

Q1: The paper seems to present conflicting conclusions. (Q1-1) Figure 4 indicates that the conflict among LoRA A is smaller than that of LoRA B. Does this mean that A learns shared knowledge, while B learns client-specific features? (Q1-2) Why is it that after PCA in Figure 5, the learned A appears more similar, while B appears less so, leading to the conclusion that "LoRA B learns shared knowledge, while LoRA A captures client-specific features"?

Thank you for raising these questions. We address Q1-1 and Q1-2 separately in A1-1 and A1-2, providing point-by-point responses.

A1-1: We begin by clarifying the definition of conflict rate, emphasizing that its value cannot directly indicate the type of knowledge. Under unconstrained conditions, the conflict rate is often shaped by multiple factors acting together. We then explain why LoRA B tends to capture shared knowledge, whereas LoRA A is more inclined to reflect client‑specific knowledge.

1. Conflict rate $\not\Rightarrow$ Knowledge type

Let $k_i$ denote the number of client signs consistent on parameter $i$, with $M = \operatorname{card}(\mathbf{\Delta W})$ representing the number of parameters and $N$ the total number of clients. The overall conflict rate $C$ and the average pairwise conflict rate $\bar{\rho}$ are defined as follows (see lines 112-127):

Both metrics rely only on the multiset ${k_i}_{i=1}^{M}$, failing to capture parameter semantics and being easily affected by initialization, noise, or sign imbalance. Hence, the conflict rate alone cannot determine the type of knowledge learned.

2. Knowledge type of LoRA B and A

During training, we impose a common directional constraint on every client to align the LoRA $B$ matrix with the pre‑trained weight direction. Without this constraint, LoRA updates are dictated entirely by local gradients: the consistent component across clients forms shared knowledge, while differences constitute specific knowledge.

With the constraint, each client’s LoRA $B$ converges toward the same global direction, explicitly injecting a common component and thus favouring the capture of shared patterns.
Meanwhile, client individuality is maintained—indeed emphasized—through an $\ell_1$‑norm sparsity regularizer applied to the LoRA $A$ matrix.

In short: LoRA B learns more shared knowledge due to explicit global directional guidance, whereas LoRA A, lacking such guidance and relying on local data, becomes increasingly biased toward client‑specific knowledge with more training epochs.

A1-2: In the PCA visualisation of Figure 5, the features of LoRA $A$ from different clients are more scattered, whereas those of LoRA $B$ are more clustered. This supports our claim that LoRA $A$ leans towards client‑specific knowledge, while LoRA $B$ leans towards shared knowledge.

It is worth noting that in Figure 5, the regions separated by the dashed line correspond to two different data partitioning configurations. The PCA visualization assesses feature similarity based on the degree of dispersion and aggregation: the more aggregated the features, the higher their similarity, and thus the more likely they reflect shared knowledge. Therefore, it is reasonable to conclude that LoRA B tends to capture shared knowledge.

Q2: Some phenomena in this work are not well explained. For example, in lines 132-133, "Constraining both LoRA A and B introduces excessive restrictions, which in turn increases the direction conflicts of LoRA A across clients." Why does this happen?

A2: Thank you for the insightful comment. In the OFL setting, imposing identical directional constraints on both LoRA matrices ($A$ and $B$) overly restricts the adaptation space, thereby weakening the model’s ability to handle non‑IID data. Under such restrictions, clients are forced to update within the same narrow subspace. This limitation is particularly severe for the $A$ matrix, which is responsible for core feature projection and is highly sensitive to local data distributions. Consequently, gradient updates on $A$ are more likely to diverge across clients, further amplifying directional inconsistency. We will incorporate this clarification in the final version.

Q3: In lines 184-185, "The regularization terms encourage LoRA B to learn shared knowledge, while LoRA A captures client-specific features." How is this achieved? Why does the sparse regularization help A learn specific knowledge?

A3: Thank you for your insightful question. In this study, we impose a unified directional constraint on the LoRA $B$ matrix, aligning it with the direction of the pre-trained weights. This encourages all clients to share a common set of output basis vectors, thereby naturally capturing cross-client shared patterns.

Meanwhile, we apply $\ell_1$‑norm sparsity regularization to the LoRA $A$ matrix, which drives each client to retain only a small number of non-zero coefficients that are most representative of its local data. Due to the heterogeneity in data distributions, the activated sets differ across clients, enabling the model to effectively capture personalized features.

The effectiveness of this design is supported by the feature distribution visualization (Figure 5) and the cosine similarity analysis (Figure 8).

Q4: In Figure 4, what are the results when only using SVD without constraints? Since the effect in Figure 4(d) may be attributed to SVD, it would be better to include results that use only SVD.

A4: Thank you for the suggestion. Since figures cannot be included in the rebuttal, we provide a quantitative explanation instead. In the original paper, YOCO$_{\text{init}}$ (lines 161 and 214) in Tables 1 and 2 corresponds to the SVD‑only result, while Figure 4(d) shows YOCO$_{\text{initB}}$, which additionally incorporates the directional constraint on LoRA $B$.

Across most scenarios, YOCO$_{initB}$ consistently outperforms YOCO$_{init}$, with the largest gain of 2.23 points observed at $p=80$% on the CrisisMMD dataset. This confirms that combining SVD with a directional constraint on LoRA $B$ yields better performance than using SVD alone. We will also provide a visualization of the conflict matrix for the SVD‑only case in the final version to further substantiate this conclusion.

Q5: The statement in line 8, "meaning they never attain true one-shot communication," seems a bit too absolute. According to the authors' discussion in Section 2, there are indeed some works that achieve "true one-shot communication."

A5: Thank you for pointing this out. The full sentence on line 8 of the original manuscript is:

“However, existing adaptive ensemble OFL methods still need more than one round of communication, because correcting heterogeneity‑induced local bias relies on aggregated global supervision, meaning they never attain true one‑shot communication.”

The conclusion “they never attain true one‑shot communication” rests on two premises:

1. “existing adaptive ensemble OFL methods,” and
2. “correcting heterogeneity‑induced local bias relies on aggregated global supervision.”

We sincerely appreciate the reviewer’s attention to the precision of our wording and fully agree with this rigorous academic attitude. In the final version, we will revise the statement to “meaning they still do not achieve true one‑shot communication” to convey the intended meaning more accurately.

Q6: I have some doubts about using pre-trained models as “implicit global supervision” to correct local training biases. Since the dataset used for the pre-trained model may differ from the data used in this FL setting, is it really reasonable to use this model as “implicit global supervision” to correct local training biases?

A6: Thank you for raising this concern. Due to space limitations, please refer to our response A2 to Reviewer P4k5.

Q7: There is some confusion in Equation (5). For example, what does $Z_i$ represent? How is $P$ obtained?

A7: We apologize for the confusion. Here, $Z_i$ denotes the coordinate vector of the $i$‑th client in the top‑$k$ principal components, representing the position of client $i$’s LoRA $A$ in the principal component space.

During PCA, we obtain both the principal component coordinates $Z$ and a loading matrix $P \in \mathbb{R}^{d\times k}$. In the paper, this is written as $P^{\top} \in \mathbb{R}^{k\times d}$—the transpose of the loading matrix—used to reconstruct the original parameter space from the global coordinates $Z_g$.

Here, $Z_g$ is the weighted average of all $Z_i$ vectors, with weights determined by each client’s local data size $|D_i|$. We will add this clarification in the final version to improve clarity and completeness.

Q8: Why is LoRA applied only to the last layer's q_proj?

A8: Thank you for raising this point. The choice of LoRA layers—including their number and positions—falls under parameter selection for LoRA fine‑tuning, whereas this paper focuses on mitigating local training bias under one‑shot communication. While distinct, both are critical for MLLMs with OFL; we leave layer selection strategies to future work.

Our experiments show that fine‑tuning even one layer achieves strong performance, sometimes surpassing multi‑round baselines. Evaluations across different single‑layer positions revealed only minor differences, with slight variations among $q_{proj}$, $k_{proj}$, $v_{proj}$, and $o_{proj}$, as shown in Figure 7, with $q_{proj}$ and $v_{proj}$ performing slightly better overall.

We would like to thank Reviewer HPxi for the thoughtful feedback and valuable questions. We have addressed these concerns with detailed responses and welcome further discussion and suggestions.

Q1: The principal component-weighted aggregation shows inconsistent improvements across different datasets (e.g., Hateful-Memes and CrisisMMD), raising questions about its generalizability.

A1: Thank you for pointing this out. The inconsistent improvements of principal component-weighted aggregation across Hateful-Memes and CrisisMMD mainly stem from task difficulty and the choice of aggregation strategy.

As shown in Table 3 of the supplementary material, YOCO adjusts its aggregation method to dataset characteristics. CrisisMMD, unlike Hateful-Memes, includes more categories with pronounced modality gaps (e.g., I:-T:), making the task more challenging. In such cases, the ensemble approach (line 212 of the main text, where "Ensemble" aggregates LoRA votes from all clients) significantly outperforms methods that merge all client updates into a single model. Consequently, when principal component‑weighted aggregation compresses all client weights into a single set, its performance falls short of the ensemble strategy. This explains why, in CrisisMMD, the principal component‑weighted aggregation sometimes demonstrates less pronounced advantages than in Hateful‑Memes.

Q2: In the "Empirical Motivation" section, the authors claim that the training loss under directional constraints indicates effective implicit global supervision without disrupting the LoRA training process. However, Figure 3 only demonstrates lower training loss. Could the authors clarify the logic why lower training loss necessarily indicates better supervision? It appears that unconstrained training might potentially achieve better results under perspective of training loss.

A2: We sincerely apologize for any confusion caused by the previous ambiguous phrasing. We now provide a clear explanation of the purpose and insights of Figure 3, followed by a discussion from the perspective of LoRA fine‑tuning to highlight the role of directional guidance in achieving effective adaptation.

1. Purpose of Figure 3

Figure 3 presents the training‑loss curves under either a directional or magnitude constraint. Our objective is to identify which constraint does not disrupt the original training process and can thus serve as a preliminary selection criterion. The subsequent subsection on conflict matrix analysis then explains why the direction constraint provides an effective implicit global supervision.

2. Insights from Figure 3

• Training stability.  Directional constraints converge rapidly and soon reach the same loss level as the unconstrained baseline, unlike magnitude constraints.

• Interference with optimization.  Magnitude constraints keep the loss consistently higher, suggesting stronger distortion of the optimization trajectory, whereas directional constraints cause only a mild detour with minimal gradient perturbation.

• Quality of supervision.  Directional guidance is gradual and readily integrated, enabling the model to align with pre‑trained directions while retaining local learning flexibility. By contrast, magnitude constraints act as a rigid clamp that often obstructs convergence.

Together, these observations indicate that directional supervision guides without disruption, whereas magnitude supervision tends to constrain too aggressively.

3. Role of Directional Guidance in LoRA Fine‑Tuning

As LoRA updates are restricted to a low‑rank subspace, alignment with the pre‑trained direction—so as to approximate full fine‑tuning—is essential for effective adaptation. Prior studies support this view:

• [a] LoRA‑GA: Uses SVD‑based initialization to preserve pre‑trained directions, aligning gradients with full fine‑tuning.
• [b] LoRA‑Pro: Promotes gradient‑direction alignment at each step, further narrowing the gap.
• [c] FRLoRA: Applies similar SVD initialization in federated settings, achieving consistent improvements.

Hence, leveraging pre‑trained weight directions as a lightweight yet effective form of implicit global supervision naturally extends these approaches.

Conclusion: Our claim is not that lower training loss alone guarantees better performance. Instead, we show that directional constraints (i) ensure stable training, (ii) minimally disturb LoRA’s local updates, and (iii) embody a proven mechanism of global guidance, as validated by [a–c].

References

• [a] Wang, Shaowen, Linxi Yu, and Jian Li. LoRA-GA: Low-Rank Adaptation with Gradient Approximation. NeurIPS 2024.
• [b] Wang, Zhengbo, et al. LoRA-Pro: Are Low-Rank Adapters Properly Optimized? ICLR 2025.
• [c] Yan, Yunlu, et al. Federated Residual Low-Rank Adaptation of Large Language Models. ICLR 2025.

Q3: 1.Please use $\ell_1$-norm instead of "L1 norm" for consistency with standard mathematical notation.

A3: Thank you for your careful review. We will correct this in the final version.

We sincerely appreciate Reviewer 6tEs for the insightful comments and questions. We have provided comprehensive responses to clarify these points and look forward to continued discussion and feedback.

Q1: Lemma 3.1 provides intuition but lacks rigorous theoretical guarantees about convergence or optimality.

A1: Thank you for this suggestion. Based on Lemma 3.1, we give theoretical guarantees on convergence and optimality. Owing to space limits, only key results are shown; full proofs will appear in the final version.

Background

• Problem: One‑round federated optimization $T = 1$ with LoRA parameters $W$ (backbone frozen). Data may be IID or non‑IID.
• Challenge: Sign conflicts in client gradients, measured by conflict rate $C$ and dispersion $\zeta$.
• Goal: Establish convergence of global loss $\mathcal{L}(W)$.

Core Assumptions

1. Smoothness & Convexity $\mathcal{L}(W)$ is $L$-smooth and $\lambda$-strongly convex with unique minimizer $W^*$.

2. Gradient Noise Per‑client gradient variance $\le \sigma_L^2$.

3. Sign Conflict

   • Dispersion $\zeta$: $\tfrac{1}{M}\sum_i (C_i - C)^2 \le \zeta^2$.
   • Alignment $\gamma$: Local gradients align with the global one ($\gamma \in (0,1]$; larger $\gamma$ = better).

Convergence Results

(i) IID Data (Theorem 3.3)

Bound:

Key terms:

• $\Delta_0 = \mathcal{L}(W_0) - \mathcal{L}(W^*)$ (initial loss gap)
• $K = \tfrac{4C(1-C)\mu^2}{N}$ (conflict‑induced variance)
• $\sigma_{\text{eff}}^{2} = \sigma^2 + 4C(1-C)\mu^2 + 4\zeta^2\mu^2$ (sampling $\sigma^2$ + conflict + dispersion)

Insight: Conflicts ($C>0$) and dispersion ($\zeta>0$) inflate both $K$ and $\sigma_{\text{eff}}^{2}$, slowing convergence. The residual error still shrinks as $\mathcal{O}(1/N)$.

(ii) Non‑IID Data (Theorem 3.4)

Bound:

Key terms:

• $\eta = \tfrac{1}{L + \lambda^{-1}K_{\text{niid}}}$ (learning rate)
• $K_{\text{niid}} = \tfrac{4C(1-C)\mu^2 + d^2}{N}$ (conflict + heterogeneity)
• $\sigma_{\text{niid}}^{2} = \sigma^2 + d^2 + 4C(1-C)\mu^2 + 4\zeta^2\mu^2$
• $d^2 = \tfrac{1}{M}\sum_i \tfrac{1}{N}\sum_n (\mu_{n,i}-\mu_i)^2$ (client drift)

Insight: Contraction depends on alignment $\gamma$ and conflict $C$. Heterogeneity $d^2$ and dispersion $\zeta$ dominate the residual error.

(iii) Asymptotic Optimality ($N \to \infty$)

Residual error:

Interpretation: Sampling noise ($\sigma^2$) and conflict ($C$) vanish, but heterogeneity ($d^2$) and dispersion ($\zeta$) remain the bottlenecks.

Q2: How does YOCO scale to hundreds or thousands of clients? Any analysis on conflict‑matrix scalability?

A2: Thank you for pointing this out. Based on the adopted benchmark, we evaluated with 10–30 clients. Our YOCO method is not limited to this range: due to GPU constraints, we also tested with 50 clients and analyzed the scalability of conflict matrix computation.

Note: V2_6 = MiniCPM‑V‑2_6int4; V2_5 = MiniCPM‑Llama3‑V‑2_5int4.

As shown in the above table, YOCO continues to demonstrate its effectiveness at larger scales compared with basic one‑shot baselines and multi‑round FL.

Scalability Analysis

Notation: $G$-number of client clusters (non‑IID groups). $p_i$-probability that parameter $i$ is positive.

Bottom line: Whether IID or moderately non‑IID, increasing clients always reduces both conflict and variance at $O(1/N)$. YOCO’s constraint further cuts the $C$‑linked variance, so its benefits persist—even with hundreds or thousands of clients.

We will include these additional experiments and scalability analyses in the final version.

Q3: Do YOCO’s gains come from its innovations or from the inherent advantages of large foundation models in one‑shot FL, as shown in [d] One Communication Round is All It Needs for Federated Fine‑Tuning Foundation Models?

A3: Thank you for your thoughtful question.

Reference [d] shows that in one-shot FL, a single communication round can match multi-round FL for large foundation models, but this relies on several assumptions. From [d] and its 2025 ICLR reviews, these include:

1. Pre-training and fine-tuning datasets are closely aligned.
2. Only FedAvg is used for federated optimization.
3. Random splits yield relatively homogeneous data from one or two sources.

To this end, we first present the results of [d] in our evaluation setting, confirming that one-shot performance falls short of multi-round training. We then analyze the advantages of YOCO’s innovations based on the aforementioned assumptions.

Evaluation of [d] in Our Setting

Because [d] uses a text-based LLM unsuited to our multimodal evaluation, we instead adopted a multimodal LLM. The results below show that one-shot performance falls short of multi-round training (More in Tables 1–2).

YOCO

1. Distribution Mismatch

• Setting: The pre-training and fine-tuning distributions differ (e.g., MLLM better suited for QA tasks, while downstream tasks involve binary classification on Hateful Memes or multi-class classification on CrisisMMD).
• Observation: As shown in Tables 1–2 of the paper, the MLLM+OFL baseline underperforms multi-round FL, with a performance gap up to 6.5 points.
• YOCO Result: Under the I:5–T:5 setting on CrisisMMD, YOCO achieves performance comparable to multi-round FL, outperforming the OFL ensemble baseline by 3.71 points and the best non-ensemble baseline by 5.35 points — at only ~0.03% of the communication cost.

2. Beyond FedAvg

• Setting: When using adaptive optimizers such as FedAdam and FedAdagrad instead of FedAvg.
• Observation: On CrisisMMD, one-shot FL performance drops notably, likely because these dynamic optimizers are less robust to the increased local training bias inherent in one-shot settings.
• YOCO Result: YOCO, specifically designed to mitigate local training bias, maintains strong performance despite these conditions.

3. Complex Data Heterogeneity

• Setting: A total of 12 heterogeneous scenarios, including 9 multimodal heterogeneous cases.
• Observation: In cross-model cases with stronger distribution shifts (e.g., I:–T:), OFL one-shot baselines degrade sharply compared to multi-round FL.
• YOCO Result: YOCO continues to deliver substantial improvements, with particularly strong results on CrisisMMD.

Q4: Uses only last-layer LoRA fine-tuning, which the authors acknowledge is suboptimal.

A4: Thank you for raising this point. It is important to distinguish two separate optimization directions here. First, the choice of parameter configurations—such as the number of LoRA fine‑tuned layers and their automatic selection strategy. Second, and the central focus of our work, is how to effectively reduce local training bias under truly one‑shot communication. While both are challenging and critical for MLLMs with OFL, our current study concentrates on mitigating local training bias, and we leave parameter selection strategies for future exploration.

Our experiments further show that fine‑tuning a single LoRA layer already yields strong performance, in some cases surpassing baselines that train all layers over multiple rounds. We also tested different layer positions; the overall performance differences were small. Within the same layer, choosing among q_proj, k_proj, v_proj, or o_proj led to only minor variations, as illustrated in the right panel of Figure 7.

We thank Reviewer P4k5 for the valuable comments and questions. We have provided detailed responses to address these concerns and look forward to further discussion and feedback.

Q1: In Section 4, I cannot see the proposed method is strongly related to multimodal settings. I wonder whether this approach is also suitable for the single-modality setting.

A1: Thank you for pointing this out. We propose the YOCO method in a multimodal setting primarily for two reasons:

1. The chosen benchmark emphasizes multimodal heterogeneity, which intensifies data distribution differences across clients and exacerbates the local training bias issue in OFL (see lines 32–33 of the paper and FuseFL [24]).
2. Compared to LLMs, MLLMs offer broader applicability in real-world scenarios.

To further evaluate the generalizability of YOCO in a unimodal setting, we additionally conducted the following experiments:

As shown in the table above, YOCO consistently outperforms all one‑shot baselines under the unimodal setting, and even surpasses multi‑round FL in the text modality of CrisisMMD. This demonstrates that YOCO remains effective in unimodal scenarios, highlighting its strong generalization capability.

Q2: This work relies on an important assumption, i.e., pre-trained weights provide sufficient global knowledge to guide local updates. However, in federated learning, clients' private data may not be consistent to the pre-trained data, leading to unreasonable local updates.

A2: Thank you for raising this concern. To clarify:

1. Rationale from the original paper

   The original paper concludes that the direction of the pre-trained weights (rather than the pre‑trained weights themselves, which include both magnitude and direction) can serve as a global supervisory signal to guide local updates in LoRA (see lines 47–49 and Section 3). This is further supported by our evidence in Figure 4 and Tables 1–4.

2. Support from prior work

   This perspective is reinforced by several recent works. As LoRA updates are restricted to a low‑rank subspace, alignment with the pre‑trained direction—so as to approximate full fine‑tuning—is essential for effective adaptation. Specifically:

   • [a] Theoretically shows that initializing LoRA’s A and B matrices with the SVD of pre-trained weights—scaled to preserve direction—helps align the initial gradient with that of full fine-tuning.
   • [b] Extends this approach by aligning gradients at every step.
   • [c] Applies a similar SVD-based initialization in a federated learning context.

3. Our conclusion

   Taken together, these works demonstrate that aligning LoRA updates with the direction of pre‑trained weights helps narrow the gap to full fine‑tuning and enhances performance. Building on this principle, our YOCO constrains the updates of LoRA B in OFL using the direction of the pre‑trained weights, thereby effectively mitigating local training bias.

References

• [a] Wang, Shaowen, Linxi Yu, and Jian Li. LoRA-GA: Low-Rank Adaptation with Gradient Approximation. NeurIPS 2024.
• [b] Wang, Zhengbo, et al. LoRA-Pro: Are Low-Rank Adapters Properly Optimized? ICLR 2025.
• [c] Yan, Yunlu, et al. Federated Residual Low-Rank Adaptation of Large Language Models. ICLR 2025.