Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation

Hi, Reviewer 2FJn:

Thank you for recognizing TopLoRA’s superior accuracy, and for your detailed and insightful feedback. It appears that your main concerns relate to the empirical support for our motivation and how our method compares to other baseline methods. Please find our point-by-point responses below.

Q1: "lack experimental support for the motivation."

R1: As noted in Eq.(1) of our paper, the standard LoRA formulation $BA=Q_BPQ_A$projects token representations into a fixed low-dimensional subspace spanned by $Q_A$. This inherently limits the model’s ability to capture token-specific variations: any input component orthogonal to this subspace is irrecoverably lost. Consequently, when the rank is small—as is often necessary for efficiency—the amount of discarded token-specific information increases.

To further quantify this limitation, we compute the mutual information (MI) between input tokens $X$ and output representations $Y$ using the InfoNCE lower bound [1], a standard technique in representation learning. Each input-output pair $(X_i,Y_i)$ is treated as positive, with others as negatives. A higher MI indicates better preservation of token-specific information. This method allows us to provide a tractable, quantitative comparison of how much token-level information is retained.

Due to the image upload limitations during rebuttal, we summarize a subset of the results on LLaMA-3-8B (Math) in the table below and will include full visualizations and analysis in the revised paper.

These results show that TopLoRA retains significantly more mutual information than standard LoRA, even at smaller ranks, confirming its stronger ability to capture token-specific features. Together with the consistent performance improvements shown in our experiments, this provides strong support for our central claim: TopLoRA’s dynamic modulation enhances the expressiveness of LoRA.

[1] A. van den Oord et al., Representation Learning with Contrastive Predictive Coding, 2018.

Q2: "each (LoRA) module has its own set of independent weights without weight sharing across tokens."

R2: We believe there may be a misunderstanding. When we state that different tokens share the same LoRA weights, we mean that within a given LoRA layer (or module), each input token $X$ (from a sequence of multiple tokens) is processed using the same set of LoRA weights. This can be formally expressed as $Y=(W+BA)X$, where the same LoRA weights $BA$ are applied uniformly to all tokens in the sequence. To introduce token-specific modulation, we propose generating a diagonal matrix $\Sigma_X$ based on the input token $X$, which is used to adjust the LoRA weights for each token in the form $B\Sigma_X A$.

Q3: "This raises the question of whether the performance improvements are due to the proposed design or merely the increased number of parameters."

R3: Due to its improved structure, TopLoRA introduces approximately 50% more parameters than LoRA at the same rank. However, the performance gains it achieves significantly surpass those obtained by merely increasing the rank of LoRA. As noted in line 227, "these gains surpass those achieved by increasing LoRA’s rank by a factor of four". For example, on Mathematical Reasoning tasks with Gemma-7B, LoRA (r=32) requires 19.3M trainable parameters to achieve an accuracy of 72.71, whereas TopLoRA (r=8) achieves a higher accuracy of 73.11 with only 6.88M parameters.

To ensure a fair comparison, we evaluated TopLoRA with ranks of 5, 10, and 20, and compared it with LoRA at ranks of 8, 16, and 32. In these settings, both methods use a similar number of parameters. As shown in the table below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The Accuracy Gain column indicates the performance improvement of TopLoRA over LoRA under comparable parameter budgets. TopLoRA achieves a 1–3% boost in accuracy with a similar number of parameters. Notably, increasing the LoRA rank by a factor of four yields only a modest 1–2% improvement. Therefore, we can confidently attribute TopLoRA’s performance gains to its structural design, rather than the approximately 50% increase in parameter count.

On the other hand, inspired by Reviewer SWr9, we explored sharing a common set of learnable projectors across different layers for weight modulation. The results show that a small number of shared projectors—approximately 8—is sufficient to achieve performance comparable to the original TopLoRA. This strategy offers an effective means of further reducing the number of trainable parameters in TopLoRA without sacrificing accuracy. Please kindly check Q5 in our response to Reviewer SWr9 for experimental results.

Q4: "Additional computational cost during inference."

R4: As described in Appendix C, we explicitly acknowledge this limitation: TopLoRA introduces token-dependent LoRA weights that cannot be merged back into the pretrained model after fine-tuning. As a result, this design incurs additional computational overhead during inference and may lead to increased latency.

Similar limitations are observed in methods such as MoELoRA and HydraLoRA ([19,25] in our paper), which also adopt dynamic adaptation mechanisms to enhance expressiveness at the cost of higher inference complexity. Likewise, TopLoRA is best suited for scenarios where merging LoRA weights is not required; for example, in multi-task settings involving multiple LoRA modules.

Q5: "Comparison with HiRA, KronA and AdaLoRA."

R5: As suggested, we added comparisons with HiRA, KronA, and AdaLoRA. HiRA and KronA enhance LoRA by using the Hadamard and Kronecker products, respectively, to increase the rank of LoRA weights. AdaLoRA, on the other hand, dynamically adjusts the LoRA rank across layers during fine-tuning.

In the comparison, we primarily set the LoRA rank to 16. It is important to note that KronA does not involve a rank hyperparameter, and its number of trainable parameters is fixed and cannot be adjusted. Due to their structural modifications, HiRA and KronA require larger learning rates to perform well. Therefore, we tuned the learning rate among {1e-4, 3e-4, 1e-3, 3e-3, 1e-2} and report their best accuracy. AdaLoRA is configured with an initial rank of 24 and a target rank of 16. For TopLoRA, we used a rank of 10 to ensure comparable parameter counts with the baselines.

As shown below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The Accuracy Gain column reflects the performance improvement over standard LoRA. KronA, likely due to its extremely low parameter count, shows worse performance than LoRA. HiRA and AdaLoRA provide moderate improvements, but the gains are less pronounced compared to TopLoRA.

It is also worth noting that TopLoRA's improvements vary across datasets and models. For instance, Qwen2.5-14B already achieves strong performance on commonsense reasoning tasks, so the additional gain from TopLoRA is relatively modest (0.42%). However, this still exceeds the improvement from doubling the LoRA rank, which only yields a 0.18% increase (as shown in the table in Q3).

Q6: "This paper lacks a related work section."

R6: The related work is currently presented in Section 5 to maintain a coherent narrative, allowing readers to first gain a complete understanding of our method before engaging with comparisons to prior approaches. We will carefully consider relocating the related work to Section 2 in the revised version to improve clarity and alignment with standard conventions.

Q7: "There is an error in line 40: a unitary matrix must be square."

R7: Thank you for pointing out the error. The correct statement is that both $Q_A$ and $Q_B$ are semi-unitary matrices; that is, $Q_A Q_A^\top=I$ and $Q_B^\top Q_B=I$.

Hi, Reviewer SWr9:

Thank you for your positive assessment of the clarity of our writing and the comprehensiveness of our experiments. Regarding your questions on comparisons with LoRA+ and the integration with MoELoRA, we have conducted substantial additional analysis to strengthen our response. Please find our point-by-point responses below.

Q1: "Comparison with LoRA+."

R1: Following your suggestion, we have added comparisons with LoRA+.

The core idea of LoRA+ is to apply a larger learning rate to the $B$ matrix, i.e., $\beta = \eta_B/\eta_A >1$. We tuned $\beta$ over the range {2, 4, 8, 16}, and reported the best-performing result. The rank of both LoRA and LoRA+ is set to 16, while for TopLoRA, we used a rank of 10 to ensure comparable parameter counts across methods.

As shown below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The Accuracy Gain column reflects the performance improvement over standard LoRA. With a similar number of parameters, TopLoRA consistently achieves higher accuracy than LoRA+.

Importantly, our method is orthogonal to LoRA+. Specifically, we can also apply a larger learning rate to the $B$ matrix in TopLoRA—a variant we refer to as TopLoRA+, which further improves accuracy by approximately 1%. These results will be included in the revised version of the paper.

Q2: "Combination of MoELoRA and TopLoRA."

R2: Based on your suggestions, we conducted a more in-depth investigation of MoELoRA. First, we added experiments using MoELoRA with four fully activated experts, each implemented as a rank-4 LoRA module. This configuration results in a parameter count comparable to that of a rank-16 TopLoRA.

As shown below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The Accuracy Gain column reflects the performance improvement over LoRA ($r=16$). The results show that TopLoRA significantly outperforms MoELoRA under a similar parameter budget.

Furthermore, we replaced each expert in MoELoRA with a rank-4 TopLoRA module, creating a variant we refer to as MoETopLoRA. Our findings indicate that the performance gains from MoELoRA and TopLoRA are largely additive, resulting in further improvements. The corresponding content will be included in the revised version of the paper.

Q3: "Parameter count on Figure 2."

R3: Thank you for the suggestion. We will add an additional vertical axis in Figure 2 to indicate the number of parameters. As image updates are not allowed during the rebuttal stage, we are currently unable to present the revised figure. However, this is a relatively straightforward modification, and we will incorporate it in the revised version.

Q4: "the only way for me to make sense of this statement (L51) is to relate it to the rank again."

R4: Below is our response to the question raised by the reviewer in the Questions section.

First, we believe that the reviewer’s understanding is correct, our proposed method is not completely rank-independent. As shown in Equation (1), the LoRA weights are expressed as $𝐵𝐴=Q_B P Q_A$. The rank determines the dimensions of the input space ($Q_A$) and output space ($Q_B$); increasing the rank expands both, thereby enhancing the expressive capacity of LoRA.

In this paper, we argue that beyond increasing the rank, LoRA’s expressiveness can also be improved by enriching the mapping from the input space to the output space. In standard LoRA, a fixed projection matrix $P$ is applied uniformly to all tokens $X$. We propose generating a diagonal matrix $\Sigma_X$ based on the input token $X$ to modulate $P$, thereby introducing token-wise adaptation and improving expressive capacity of LoRA.

We realize that the reviewer’s question likely stems from the phrasing "beyond the LoRA rank..." in Line 51. A more precise statement would be "beyond increasing the LoRA rank..." which is consistent with our framing in the title: "Beyond Higher Rank".

Q5: "It is possible to share the overhead between layers by attributing a single operation for the whole model."

R5: Thank you for this insightful suggestion, it was truly inspiring for us. Indeed, the learnable projector in TopLoRA can be reconsidered from the perspective of sharing across layers.

In the original design of TopLoRA, each layer learns a distinct projector to modulate the LoRA weights based on the input token. However, if we temporarily ignore the layer-wise structure, input tokens from different layers can be grouped together and share a common set of projectors for weight modulation.

Motivated by this idea, we conducted experiments on LLaMA-3-8B with a rank of 8, sharing a fixed number of learnable projectors $N\in\\{1,2,4,8,16\\}$ across different layers. As shown below, we report the average accuracy on mathematical and commonsense reasoning benchmarks. The Accuracy Gain column indicates the performance improvement relative to standard LoRA. The results show that a small number of shared projectors—approximately 8—is sufficient to achieve performance comparable to the original TopLoRA. This strategy offers an effective means of further reducing the number of trainable parameters in TopLoRA without sacrificing accuracy. The revised version of the paper will include these discussions and results.

Once again, we sincerely thank the reviewer for this valuable comment, which has meaningfully improved our work.

Hi Reviewer 6dXU:

Thank you for your recognition of our motivation, methodological clarity, and extensive experiments. We notice that the concerns mainly relate to the additional parameter cost of TopLoRA and its conceptual distinction from attention. We have carefully addressed both points through further clarification and empirical evidence. Below, we provide responses to each point individually.

Q1: "Unfair comparison between LoRA and TopLoRA."

R1: Due to the improved structure, TopLoRA introduces approximately 50% more parameters than LoRA at the same rank. However, the performance gains it achieves significantly surpass those obtained by merely increasing the rank of LoRA. As noted in line 227, "these gains surpass those achieved by increasing LoRA’s rank by a factor of four". For example, on Mathematical Reasoning tasks with Gemma-7B, LoRA (r=32) requires 19.3M trainable parameters to achieve an accuracy of 72.71, whereas TopLoRA (r=8) achieves a higher accuracy of 73.11 with only 6.88M parameters.

To ensure a fair comparison, we evaluated TopLoRA (rank=5/10/20) and compared it with LoRA (rank=8/16/32). In these settings, they use a similar number of parameters. As shown below, we report the average accuracy on mathematical and commonsense reasoning tasks. The Accuracy Gain column indicates the performance gain of TopLoRA over LoRA under comparable parameter budgets. TopLoRA achieves a 1–3% boost in accuracy with a similar number of parameters. Notably, increasing the LoRA rank by a factor of four yields only a modest 1–2% improvement.

On the other hand, inspired by Reviewer SWr9, we explored sharing a common set of learnable projectors across different layers for weight modulation. The results show that a small number of shared projectors—approximately 8—is sufficient to achieve performance comparable to the original TopLoRA. This strategy offers an effective means of further reducing the number of trainable parameters in TopLoRA without sacrificing accuracy. Please kindly check Q5 in our response to Reviewer SWr9 for experimental results.

Q2: "If we compare TopLoRA to stronger LoRA variants, the improvement is smaller or even marginal."

R2: Compared to stronger LoRA variants, our approach still achieves substantial improvements.

Our paper benchmarks several state-of-the-art methods, including MELoRA, DoRA, and HydraLoRA. The results show that TopLoRA achieves higher accuracy while using fewer parameters. Please kindly check Tables 1/2/3 in the paper and compare TopLoRA (r=8) with DoRA (r= 16), MoELoRA (r=16), and HydraLoRA (r=8), respectively.

We also include comparisons with other strong baselines here, such as the higher rank methods HiRA and KronA, and the dynamic rank adjustment method AdaLoRA. Note that KronA does not have a rank hyperparameter, and its number of trainable parameters is fixed and non-adjustable. Due to their architectural modifications, HiRA and KronA require larger learning rates to perform well. Accordingly, we tuned the learning rate over {1e-4, 3e-4, 1e-3, 3e-3, 1e-2} and report the best results. AdaLoRA is configured with an initial rank of 24 and a target rank of 16. TopLoRA uses a rank of 10 to maintain similar parameter counts as the baselines.

As shown below, the Accuracy Gain column reflects performance gains relative to standard LoRA. KronA shows inferior accuracy compared to LoRA, likely due to its extremely small number of trainable parameters. Nevertheless, TopLoRA consistently outperforms all of these state-of-the-art methods under comparable parameter budgets.

Importantly, as shown in the table in R1, doubling or even quadrupling the LoRA rank does not lead to accuracy gains comparable to those achieved by TopLoRA—gains that these baselines fail to match. The improvements delivered by TopLoRA are substantial, underscoring the effectiveness of its architectural design.

Q3: "The original Q,K,V mapping is capturing token-wise information, so the attached updated LoRA layers are capable of capturing token-wise information as well ... In such cases, the methodological contribution of TopLoRA is limited to me."

R3: This is an interesting question​​, and we appreciate the opportunity to clarify the distinction between token interactions (handled by attention) and token-specific processing (our focus within LoRA layers), as well as why TopLoRA represents a methodological innovation.

​1. Roles of Attention vs. LoRA: The QKV mappings in attention are designed to capture inter-token dependencies (how tokens influence one another). In contrast, LoRA layers operate prior to attention, transforming individual token representations. The ability of LoRA to preserve token-specific information is therefore orthogonal to the function of attention.

2. ​​Limitation of Standard LoRA​​: As noted in Eq.(1) in our paper, LoRA weights $BA=Q_BPQ_A$​ projects inputs into a fixed subspace (spanned by $Q_A$). This means that token information orthogonal to $Q_A$ is irrecoverably lost in the transformation. Increasing rank (more $Q_A$ columns) expands this subspace but remains static across tokens, and it cannot adapt to token-specific features.

3. TopLoRA’s Innovation​​: Instead of expanding rank, TopLoRA generates token-dependent diagonal matrices $\Sigma_X$ to dynamically modulate the input-output mapping per token. This token-aware reweighting enriches the expressiveness of LoRA by allowing the transformation to adapt to each token’s unique features.

​4. Why This Matters for Attention​​: Better token-specific transformations (via TopLoRA) lead to higher-quality QKV representations. Attention can only model inter-token relationships effectively if the underlying QKV mappings already capture rich, discriminative token features.

​​In summary​​, TopLoRA is not just "another channel" for token-wise information but a fundamental enhancement to LoRA’s capacity to preserve token-specific features—a prerequisite for attention to function optimally. This is a distinct contribution beyond static low-rank adaptations.

Q4: "Evidence that support the claim 'Existing methods fail to capture token-specific information'"

R4: As noted in Eq.(1) of our paper, the standard LoRA formulation $BA=Q_BPQ_A$projects token representations into a fixed low-dimensional subspace spanned by $Q_A$. This inherently limits the model’s ability to capture token-specific variations: any input component orthogonal to this subspace is irrecoverably lost. Consequently, when the rank is small—as is often necessary for efficiency—the amount of discarded token-specific information increases.

To further quantify this limitation, we compute the mutual information (MI) between input tokens $X$ and output representations $Y$ using the InfoNCE lower bound [1], a standard technique in representation learning. Each input-output pair $(X_i,Y_i)$ is treated as positive, with others as negatives. A higher MI indicates better preservation of token-specific information. This method allows us to provide a tractable, quantitative comparison of how much token-level information is retained.

Due to the image upload limitations during rebuttal, we summarize a subset of the results on LLaMA-3-8B (Math) in the table below and will include full visualizations and analysis in the revised paper.

These results show that TopLoRA retains significantly more mutual information than standard LoRA, even at smaller ranks, confirming its stronger ability to capture token-specific features. Together with the consistent performance improvements shown in our experiments, this provides strong support for our central claim: TopLoRA’s dynamic modulation enhances the expressiveness of LoRA.

[1] A. van den Oord et al., Representation Learning with Contrastive Predictive Coding, 2018.

Hi, Reviewer 2KRw:

Thank you for your positive assessment of the clarity and novelty of our paper. In response to the concerns regarding comparisons on vision tasks and evaluations against MoELoRA, we have conducted substantial supplementary experiments and analyses to strengthen our findings. Please find our point-by-point responses below.

Q1: "Comparison with MoELoRA (and other methods)."

R1: As suggested, we conducted a more in-depth investigation of MoELoRA. Specifically, we added experiments using MoELoRA with four fully activated experts, each implemented as a rank-4 LoRA module. This configuration yields a total parameter count comparable to that of a rank-16 TopLoRA.

As shown below, we report the average accuracy of different models on mathematical and commonsense reasoning benchmarks. The Accuracy Gain column reflects the performance improvement over LoRA ($r=16$). The results show that TopLoRA significantly outperforms MoELoRA under a similar parameter budget.

Furthermore, we replaced each expert in MoELoRA with a rank-4 TopLoRA module, creating a variant we refer to as MoETopLoRA. Our findings indicate that the performance gains from MoELoRA and TopLoRA are largely additive, resulting in further improvements.

In addition, we have included more relevant baseline comparisons, such as the rank-increasing methods HiRA and KronA, the dynamic rank adjustment method AdaLoRA, and the optimization-enhanced method LoRA+. For further details, please refer to Q1 in our response to Reviewer SWr9 and Q5 in our response to Reviewer 2FJn. Thank you again for your thoughtful feedback.

Q2: "The role (of Exp and RMSNorm) needs further explanation. Also, does the order in which they are applied matter?"

R2: To clarify this, we first highlight three key points:

1. TopLoRA extends LoRA by introducing a diagonal matrix $\Sigma_X$ in the form of $B\Sigma_XA$. $\Sigma_X$ is generated by a learnable projector based on the input token $X$. Notably, LoRA is a special case where $\Sigma_X$ is the identity matrix.
2. When the standard deviation of the parameters in $\Sigma_X$ is small (i.e., the values are tightly concentrated), $\Sigma_X$ becomes nearly identical across different tokens, limiting its ability to modulate token-specific weights.
3. If an element in $\Sigma_X$ is zero or close to zero, it is equivalent to setting the column of B or the row of A to zero. If there are a large number of values distributed near zero, it may reduce the expressive power of TopLoRA.

Given that the standard deviation of the projector output is typically small (around 0.05), we first apply RMSNorm to scale $\Sigma_X$, removing the effects of both input token variance and projector weight variance.

Additionally, since the projector output tends to have a mean close to zero, we use an exponential (Exp) transformation to shift the distribution of $\Sigma_X$ from around 0 to around 1.

Note that RMSNorm does not include a mean subtraction step (or implicitly assumes zero mean). Therefore, if we apply the Exp transformation before normalization, we must use LayerNorm instead of RMSNorm to account for the shift in mean. Continuing to use RMSNorm after Exp would fail to properly normalize the standard deviation of $\Sigma_X$.

We also added experimental results to compare the effects of different orders of applying the exponential function (Exp) and normalization methods (RMSNorm and LayerNorm). Based on Table 4 in the paper, we introduced two additional settings where Exp is applied first, followed by either RMSNorm or LayerNorm. The results show that there is no significant difference between RMSNorm(Exp($\cdot$)) and simply applying Exp($\cdot$) alone. In contrast, LayerNorm(Exp($\cdot$)) achieves performance close to the default setting, Exp(RMSNorm($\cdot$)).

Q3: "How does TopLoRA perform on vision tasks?"

R3: As suggested, we have included fine-tuning results of TopLoRA on an image classification task. Specifically, we fine-tuned a pretrained ViT model (google/vit-base-patch16-224-in21k) on the CIFAR-100 dataset for 60 epochs. The learning rate was tuned over {1e-4, 3e-4, 1e-3, 3e-3}, with the final value set to 3e-4. The rank was varied among 8, 16, 32, and 64. Both LoRA and TopLoRA were applied to the query, value, and classifier weights. As shown below, TopLoRA achieves comparable accuracy while reducing the number of trainable parameters by approximately 6×.