MokA: Multimodal Low-Rank Adaptation for MLLMs

We sincerely thank the reviewer for the positive feedback on our work. We also appreciate the helpful suggestions regarding the efficiency evaluation, the discussion of the cross-attention module, and other valuable advice. In response, we will include related experiments and discussions in the revised manuscript to make the paper clearer.

1. Efficiency evaluation

In the previous implementation of Multiple-LoRA, we did not merge the multiple LoRA heads during inference, to keep the implementation modular easily extensible to other LoRA variants. As a result, each token passes through multiple A and B matrices independently, bringing additional cost. This could be the reason that Multiple-LoRA has a longer latency in the previous experiments.

Thank you for pointing this out. We improve the implementation of Multiple-LoRA, merging multiple LoRA heads during inference. In addition, we also add additional efficiency metrics, including FLOPs and GPU memory usage, to better evaluate the efficiency. The following table shows results under Vision-Language case as an example.

Based on the results, Multiple-LoRA-merged during inference has a similar computational cost. MokA obtains noticeable performance improvement with only modest extra resource consumption. We will add these experiments and discussion to the revised manuscript.

2. Discussion about cross-attention module

Here we will provide a more detailed discussion about our cross-attention module.

1. Cross-attention only applies to prompt tokens

   In the original MokA, cross-attention is motivated by the fact that text modality typically conveys the question or task description, while the audio and visual modalities provide environment information, in the prompt. Cross-attention is applied to explicitly enhance the interaction between the task (text token in the prompt) and environment (non-text token in the prompt).

   Therefore, the cross-attention is only applied to the prompt parts. Concretely, during training, only the prompt tokens (excluding response tokens) are considered in the cross-attention module. During inference, cross-attention only operates on the prefilling stage, where the model processes the entire prompt sequence and generates the first output token. The decoding stage is not applied.

   Accordingly, MokA does not greatly break the causal nature when generating response. It only enhances the comprehension of prompt information.

2. Query and positional encoding

   Yes, Query is from audio/vision tokens. Key and Value are from text tokens. Regarding positional encoding, we retain the original positional encoding of the LLMs, and do not introduce any new encoding specifically for the cross-attention for efficiency.

3. No additional parameters

   Yes, this module does not introduce any new parameters, and only introduces new computes. Before passing into the cross-attention module, tokens of each modality are processed by their own low-rank matrices $A$. The low-rank matrices $A$ of each modality can be approximately regarded as the linear projection in the standard attention in this case. In addition, projections of key and value are shared in this case (since key and value are both text tokens that are processed by $A^t$). And this kind of projection sharing strategy has been shown to be effective for improving attention efficiency [1]. Therefore, we do not introduce other linear projections in the cross-attention module to be more efficient. Thank you for this helpful advice, and we will add the related discussion to the revised manuscript.

[1] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.

3. MLLMs can generate audio/vision tokens

At present, our work focuses on scenarios where the MLLMs generate text tokens while conditioning on audio/vision/text inputs. MLLMs that can generate audio/vision tokens are indeed an important and exciting direction for future research. We sincerely thank the reviewer for inspiring us to consider this promising direction.

4. Discussion about Multiple-LoRA & MokA

Here we first provide the differences between LoRA, Multiple-LoRA, and MokA w/o CA. Suppose the number of modalities is $N$:

• LoRA: uses one matrix A and one matrix B, both shared by all modalities.
• Multiple-LoRA: uses $N$ matrices A and $N$ matrices B. But all matrices (both A and B) are still fully shared across modalities.
• MokA w/o CA: uses $N$ modality-specific matrices A but one shared matrix B. Here, the matrices A adapt each modality independently, while the shared matrix B maps the separated unimodal subspaces back into a common latent space.

Multiple-LoRA simply expands standard LoRA’s capacity by adding multiple heads but still treats all modalities the same—ignoring their unique characteristics. However, just increasing capacity while fully sharing parameters across all modalities does not solve the specific challenges of multimodal scenarios. It still does not consider both unimodal adaptation and multimodal adaptation.

In contrast, design of MokA w/o CA is multimodal-aware. It explicitly ensures better unimodal adaptation. MokA w/o CA explicitly captures unimodal information by introducing modality-specific matrices A. It ensures that each modality’s unique information is individually captured before merging. This simple but crucial design helps better utilize unimodal adaptation. This explains why MokA w/o CA outperforms Multiple-LoRA, even with fewer total parameters.

5. Applied weights of LoRA/MokA

In experiments, we apply LoRA or MokA modules to important weights within each transformer block. Specifically, we insert them into:

• q_proj, k_proj, v_proj, o_proj (the self-attention projection weights),
• gate_proj, up_proj, and down_proj (the MLP feed-forward layers).

6. Typos

We thank the reviewer for pointing out the typos. We have carefully revised the manuscript to correct these errors. Thank you for helping us improve the clarity and presentation of the paper.

Dear reviewer, we would like to know if our responses have successfully resolved your concerns. We are willing to respond to any further questions.

We are deeply grateful to the reviewer’s positive feedback and helpful comments. Based on suggestions, we have added new ablation studies, comparisons, and evaluations, which will be included in the revised manuscript.

1. Ablation studies about rank

Thank you for the valuable suggestion. In our experiments, we did not specifically tune the rank hyperparameter, as our primary focus is to provide a multimodal-aware PEFT method that is rank-agnostic. Based on this suggestion, we conducted ablation studies with different ranks, and the results are shown in the following table. Experiments are conducted based on LLaMA2. These results indicate that simply increasing the rank does not always lead to performance gains, which aligns with the observation in the original LoRA paper [1]: “We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.”

More importantly, MokA consistently outperforms the LoRA baseline across different rank settings, demonstrating its scalability. We will include this discussion and the new results in the revised manuscript. Thank you again for highlighting this point.

[1] LoRA: Low-Rank Adaptation of Large Language Models, ICLR, 2022.

2. Comparison to related methods

We thank the reviewer for pointing out the related method, Omni-SMoLA.

Compared to MokA, Omni-SMoLA adopts a more complex design with noticeably more parameters. For example, in an AVL (Audio-Visual-Language) scenario, Omni-SMoLA employs separate SMoLA modules for each modality (visual, audio, and text), in addition to an extra multimodal SMoLA. Each SMoLA module internally integrates multiple LoRA branches.

In contrast, MokA only uses a few low-rank A matrices (one per modality) and a single shared low-rank B matrix, without stacking multiple independent LoRA blocks. Therefore, Omni-SMoLA can have several times more parameters than MokA.

Concretely, in experiments, we set each SMoLA to have two LoRA branches. Therefore, in the AVL case, Omni-SMoLA has four SMoLA modules (audio, visual, text and multimodal), and each SMoLA has two LoRA branches. Then the total number of low-rank matrices (both A and B) is 16, which is much more than MokA. The VL case is similar.

The comparison results in the following table show that MokA achieves comparable or even better performance than Omni-SMoLA, while maintaining a simpler and more lightweight design. Experiments are conducted based on LLaMA2.

We will add this comparison and discussion to the revised manuscript.

3. Evaluation on MME reasoning benchmark

We thank the reviewer for this helpful suggestion to evaluate performance on the MME reasoning benchmark to assess potential catastrophic forgetting of the LLM’s pretrained knowledge. Following this advice, we conduct an evaluation on the MME reasoning benchmark. Experiments are conducted based on LLaMA2.

On the MME reasoning benchmark, the LoRA baseline achieves an accuracy of 16.78%, while MokA reaches 18.01%, indicating that MokA has stronger capacity in this aspect.

It should be noted that both LoRA and MokA do not obtain high scores on this benchmark, as our work focuses only on the supervised fine-tuning (SFT) stage and does not include additional designs specifically aimed at enhancing reasoning ability. Nevertheless, under the same setting, MokA achieves consistently better results, demonstrating its effectiveness.

We will include the new MME reasoning results and discussion in the revised manuscript.

4. Training setting

Yes, all the compared methods under the same scenarios are conducted under the same settings to ensure fairness.

Dear reviewer, we would like to know whether our responses addressed your concern. We are happy to answer any further questions and comments.

We greatly appreciate the reviewer for the constructive and detailed comments, which help us improve the quality, clarity and presentation of the manuscript. We will add these experiments and analysis to the revised manuscript.

1. Ablation studies about rank

Based on the suggestions, we explore the effect of varying the rank. Experiments are conducted based on LLaMA2. As shown in the following table, simply increasing the rank does not necessarily lead to consistent performance gains. This observation aligns with the finding reported in the original LoRA paper [1]: “We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.”

More importantly, our observation in Figure 1 of the manuscript shows that the optimization of all-modality-shared LoRA parameters is overly influenced by text tokens, resulting in non-text tokens being less effectively utilized. Therefore, even with an increased rank, the parameters remain fully shared across modalities, which still structurally prevents independent utilization of unimodal information. The issue is not just about capacity, but more about whether the module structure can effectively capture modality-specific characteristics and cross-modal interactions.

Based on this, MokA is designed to explicitly introduce modality-specific parameters and cross-modal interaction, addressing this structural limitation that increasing rank alone cannot resolve. We also conduct experiments of MokA with different ranks. It consistently outperforms the LoRA baseline across different rank settings.

[1] LoRA: Low-Rank Adaptation of Large Language Models, ICLR, 2022.

2. First token generation process

Here, “at the first generation step during inference” refers specifically to the prefilling stage, where the model processes the entire prompt sequence and generates the first output token.

Thank you for pointing this out, and we will revise the related parts of the manuscript to make it more readable.

3. Discussion about multiple-LoRA & MokA w/o CA

To clarify, the following explains the differences between LoRA, Multiple-LoRA, and MokA w/o CA. Suppose the number of modalities is $N$:

• LoRA: uses one matrix A and one matrix B, both shared by all modalities.
• Multiple-LoRA: uses $N$ matrices A and $N$ matrices B. But all matrices (both A and B) are still fully shared across modalities. There are no multiple modality-specific matrices. The multiple heads simply increase capacity without distinguishing modality-specific characteristics.
• MokA w/o CA: uses $N$ modality-specific matrices A and one shared matrix B. Here, the matrices A adapt each modality independently, while the shared matrix B maps the separated unimodal subspaces back into a common latent space.

Multiple-LoRA can be viewed as simply expanding the capacity of standard LoRA by adding multiple heads. However, it still treats all modalities identically, i.e. all parameters are shared across all modalities. There are no modality-specific matrices in this method. In this fully shared case, as discussed above, optimization of parameters is overly influenced by text tokens, which results in non-text tokens being less effectively utilized. Therefore, simply increasing the number of LoRA heads does not provide a significant benefit in addressing the multimodal challenge.

By contrast, MokA w/o CA explicitly captures unimodal information by introducing modality-specific matrices A. This design better captures unique modality characteristics before merging them back with the shared B matrix.

Therefore, the better performance of MokA w/o CA comes from its more effective use of modality-specific information.

4. Discussion about "Uni LoRA + MM LoRA" and "Uni LoRA + MM LoRA + Gate"

As mentioned in Sec 2.1, one of the core motivations for our work is that existing LoRA variants for multimodal adaptation often overlook the need to jointly address both unimodal adaptation and cross-modal adaptation. Our MokA method is proposed to address this limitation.

Since most LoRA variants do not explicitly consider multimodal characteristics, we further introduce two more relevant and competitive baselines to provide a more comprehensive comparison.

Concretely, these two baselines, Uni LoRA + M LoRA and Uni LoRA + MM LoRA + Gate, are straightforward designs that aim to address both unimodal and multimodal adaptation:

• “Uni LoRA + MM LoRA” directly uses unimodal LoRA to fit unimodal adaptation, and multimodal LoRA to fit multimodal adaptation.
• “Uni LoRA + MM LoRA + Gate” further adds a gating mechanism to adaptively integrate the unimodal and multimodal branches.

These baselines demonstrate that introducing a multimodal-aware design—i.e., accounting for both unimodal and cross-modal adaptation—can indeed lead to improved performance, confirming that jointly considering both aspects is necessary.

However, these variants rely on simply stacking separate LoRA modules and thus require substantially more parameters.

In contrast, MokA achieves both unimodal and multimodal adaptation more neatly and efficiently through its modality-specific A matrices and flexible cross-attention module, achieving better performance with a simpler and more cohesive structure.

5. Discussion about the simplicity of LoRA

Thank you for pointing this out. Indeed, MokA makes the side branch separate from the main LLM parameters and thus it can no longer be merged back. However, multimodal scenarios are inherently more complex, and it is often inevitable to introduce additional cost to better capture multimodal characteristics.

We have aimed to keep the structure of MokA as neat and efficient as possible. Based on our efficiency evaluation in terms of FLOPs, GPU memory usage, and average forward time per sample, MokA introduces only modest additional resource consumption while achieving significant performance gains. Therefore, we believe MokA is practical with noticeably better performance.

6. Enhancing Fig. 4

Thank you for this suggestion. We will revise Fig.4 to make the arrows/lines clearer and more organized, improving its clarity.

Thanks for your response.

After the discussion, this paper is complete and sound to me. I am therefore happy to raise my score.

Meanwhile, since I have not been closely tracing the recent literature on this topic, for novelty and comparative experiments, I would suggest the AC to give more weight to the opinions of other reviews.

We sincerely thank the reviewer for these valuable suggestions, which have greatly strengthened our work. We will add these additional experiments and related discussion to the revised manuscript.

1. Efficiency evaluation & 3+ modality extension

We thank you for the valuable suggestions about efficiency evaluation. As suggested, we analyze and compare LoRA and MokA from computational cost, GPU memory usage, and extension under 3+ modality cases. Experiments are conducted based on LLaMA2.

1. On Computational Cost:

   Compared to LoRA, MokA introduces additional A matrices and a cross-attention module. However, it should be noted that the additional computational cost of MokA only comes from the cross-attention module, which is flexible and can be replaced by a more efficient strategy if needed.

   Concretely, in LoRA, each token is processed through one A matrix and one B matrix. Similarly, in MokA w/o cross-attention, each token passes through one modality-specific A matrix and one shared B matrix. This design maintains the same computational cost as LoRA while enabling modality awareness. In addition, as discussed in Sec. 4.4 and shown in the following table, MokA w/o cross-attention already brings notable performance improvements over LoRA without additional computational burden.

   ---

   	MUSIC-AVQA	POPE	AIR-Bench
   LoRA	73.41	70.28	31.75
   MokA w/o cross-attention	74.85	73.57	33.25
   MokA	75.71	74.23	39.64

   ---

   As per the cross-attention module, it is employed to explicitly strengthen the interaction between text and non-text tokens, thereby facilitating cross-modal adaptation. Alternative strategies, like more efficient cross-modal interaction methods, can also be considered as mentioned in the manuscript. Here we provide one variant, MokA w/ naive interaction, which performs a simple, uniform mapping from text tokens to non-text tokens without employing any attention mechanism. And this more efficient variant also achieves improvement.

   ---

   	MUSIC-AVQA	AVE	MME_${percep}$	MMBench	POPE	SEED-Bench
   LoRA	73.41	69.84	908.52	50.64	70.28	39.71
   MokA	75.71	74.68	1025.86	52.74	74.23	40.45
   MokA w/ naive interaction	75.04	73.18	996.73	51.49	73.52	40.17

   ---

2. On GPU Memory Usage:

   MokA’s additional GPU memory usage mainly comes from having modality-specific A matrices. In practice, the A matrices are lightweight by design (following LoRA’s low-rank parameterization), so the increase of memory usage is small and remains within acceptable limits. We will provide more information in the following parts.

3. 3+ Modality Extension:

   Based on the suggestion, we extend our experiments to scenarios with more than three modalities. Specifically, we consider a four-modality setting involving audio, visual, point cloud, and language data, and evaluate it on the MCUB-3 benchmark. In this 3+ modality case, LoRA has an accuracy of 37.41%. Our MokA has an accuracy of 45.58%, indicating its scalability and effectiveness under 3+ modality cases. We also provide the efficiency comparison in the following parts.

4. On Comprehensive Efficiency Analysis:

   Following the reviewer’s valuable suggestion, we have added a more detailed efficiency analysis comparing FLOPs, GPU memory usage, and average forward time per sample (proportional to training time). For clarity, we report these metrics for the two-modality (VL), three-modality (AVL), and newly added four-modality (AVPL) settings. As shown in the tables, MokA’s extra computational and memory cost remains limited and acceptable for typical multimodal scenarios, which aligns with the above analysis. Overall, we obtain significant performance gains with only modest extra resource consumption.

   ---

   VL (MME_${percep}$)	FLOPs	Memory Usage	Avg. forward time/sample
   LoRA	1526.25 GFLOPs (1.000x)	27315.75 MB (1.000x)	1.75 s (1.000x)
   MokA	1539.55 GFLOPs (1.009x)	27329.78 MB (1.001x)	1.87 s (1.069x)

   ---

   AVL (MUSIC-AVQA)	FLOPs	Memory Usage	Avg. forward time/sample
   LoRA	12116.72 GFLOPs (1.000x)	27961.74 MB (1.000x)	5.53 s (1.000x)
   MokA	12375.90 GFLOPs (1.021x)	27996.49 MB (1.001x)	6.27 s (1.134x)

   ---

   AVPL (MCUB-3)	FLOPs	Memory Usage	Avg. forward time/sample
   LoRA	12620.51 GFLOPs (1.000x)	28329.56 MB (1.000x)	1.64 s (1.000x)
   MokA	12778.88 GFLOPs (1.013x)	28377.37 MB (1.002x)	1.99 s (1.213x)

   ---

2. Uni-modal adaptation in Eq. (13)

Specifically, in Eq. (13), the unimodal adaptation component is formulated as $$ BA^a\mathbf{x}^a; BA^v\mathbf{x}^v; BA^t\mathbf{x}^t $$, where $A^a$, $A^v$, $A^t$ are modality-specific matrices and $B$ is a shared projection matrix. Input of modality $i$ first undergoes a unique modality-specific transformation through its respective $A^i$ matrix. And then, the subsequent shared $B$ matrix applies an identical mapping to all modalities. Therefore, the modality-specific characteristics introduced by $A^i$ and $\mathbf{x}^i$ are preserved, since $B$ is identical to all modality-specific terms and it does not explicitly mix information across modalities.

Therefore, we consider this component an approximation of unimodal adaptation, since the distinct modality-specific paths remain independent. In the revised manuscript, we will refine the related analysis.

3. Audio-visual interaction

In the original MokA, only the attention between text and non-text tokens is considered. It is motivated by the fact that text modality typically conveys the question or task description, while the audio and visual modalities provide environment information, in the instruction. Therefore, cross-attention is applied to explicitly enhance the interaction between the task (text token) and environment (non-text token).

As suggested, it is also worth exploring whether further interactions between the scene modalities themselves—i.e., audio and visual—can be beneficial. To this end, we conduct additional ablation studies. Experiments are conducted based on LLaMA2. The table reports the results for audio-visual attention with both audio as query and video as query. The results show that introducing additional audio-visual attention can bring gains but is not very noticeable. The reason could be that the original cross-attention in MokA has effectively captured task-relevant information within the audio and visual inputs. As a result, the potential benefit from further enhancing explicit audio-visual interactions is relatively limited. We will include this new ablation and the corresponding discussion in the revised manuscript.

4. More attention strategies

We thank the reviewer for this valuable suggestion regarding the flexibility of the cross-attention module. As mentioned above, our cross-attention design primarily aims to explicitly strengthen cross-modal interactions, and it is flexible and can be replaced by different strategies if needed. Based on the suggestion, we further explore bi-directional attention and dynamic attention as alternative designs. Experiments are conducted based on LLaMA2.

The results show that these alternative designs also achieve improvements over the LoRA baseline, further demonstrating the flexibility and effectiveness of the cross-modal interaction module. We appreciate the reviewer’s suggestion, which has strengthened the paper by motivating us to include these informative comparisons.

Dear reviewer, we would like to know if our responses have successfully tackled your concerns. We are open to answering any additional questions that you may have.