Provable Meta-Learning with Low-Rank Adaptations

Dear Reviewer qeBR, thank you for your positive feedback regarding the clarity of our presentation and the originality of our contributions. We address your remaining questions below.

(Q1-2) Could you explain...commentary from the authors... Is it possible... claim of mine

(A1-2) Thank you for raising this important point.

As you note, the structure of real-world adaptations is not well understood. We analyze the symmetric case to avoid complications in the analysis that do not primarily relate to our goal of studying meta-learning with low-rank adaptation. The LoRA fine-tuning approach we analyze is closely related to problems in matrix sensing, where assuming the ground truth matrix is symmetric can lead to a cleaner analysis. While it is not immediately clear how to extend the guarantees of our proposed LoRA-ML objective to the asymmetric case (please see our response (A4) for details on technical challenges), we emphasize that even in the case of asymmetric adapters, standard retraining provably fails.

If the ground truth adapters $R_t^\ast = U_t^\ast V_t^\ast$ are asymmetric, the analog of Eqn. (12) shows that standard retraining (SR) recovers $\hat{A}_{SR} = A^\ast + \frac{1}{T} \sum _{t=1} ^T R_t^\ast$. For any $T$, this estimate will deviate from $A^\ast$ by an error of large rank under a mild condition:

Theorem: For finite $T$, assume the distribution over adapters $R_t^\ast$ is absolutely continuous with respect to the Lebesgue measure and that the ambient dimension $d$ is large enough so $kT \leq d$. Then with probability 1, $rank(\hat{A}_{\text{SR}} - A^\ast) = kT$

Thus for finite $T$, an analog of Proposition 1 still holds: a rank-$k(T+1)$ adapter is required to exactly fit to a test task, even though recovering $A^\ast$ would have allowed us to adapt using only a rank-$k$ adapter.

For large $T$, we emphasize that in general $\hat{A}_{SR}$ does not converge to $A^*$. The error term $\frac{1}{T} \sum_{t=1}^T R_t^\ast$ is the average of the adapters. If we assume that $R_t^\*$ are drawn i.i.d. from a fixed distribution with mean $M_R$, then this term converges almost surely to $M_R$ as $T \to \infty$:

Theorem: Consider ground truth adapters $R_t^*$ which are drawn from some arbitrary distribution with mean $M_R$. Then, $\lim_{T \rightarrow \infty} \hat{A}_{SR} = A^\ast + M_R$

Even this limit, SR cannot recover $A^\ast$ unless $M_R = 0$. We emphasize that this $0$-mean scenario is highly restrictive. To be effective in realistic applications, an algorithm must succeed under general task distributions.

We now clarify our assumption that $U_t^\ast$ is generated according to a $0$-mean distribution in our symmetric setting, even though we have just argued against requiring such assumptions. This choice was for notational and analytical convenience, as all of our guarantees for LoRA-ML in Section 3.2 hold deterministically for any $\\{U_t^*\\}_{t=1}^T$ as long as the diversity condition in Remark 1 is satisfied. We use the assumed distribution over $U_t^*$ only to quantify the error of SR in an average sense in Theorem 1 and Proposition 2. These assumptions do not restrict the generality of our results on the performance of LoRA-ML, as long as the diversity condition holds.

We will update the revised paper to include the above discussion and clarify (i) why we assumed symmetric adapters, (ii) why standard retraining still fails in the asymmetric case, and (iii) why the distributional parameters over $U_t^\ast$ do not affect our performance guarantees for LoRA-ML.

(Q3) Remark 1 is not suited ... number of tasks?

(A3) Thank you for this comment. To clarify, we make the technical assumption of independent adapters in Remark 1 to naturally formalize the notion of task diversity for our theoretical setting. This is analogous to prior works [1-4], which consider the problem of recovering a shared linear representation across tasks. To guarantee recovery, they require that the tasks represent a set of independent vectors on this shared subspace.

We note that it is difficult to make claims about the structure of idealized adapters for real data. When adapting deep networks via LoRA across multiple layers, it is also unclear which properties of these adapters meaningfully capture task diversity. Linear independence was natural for our setting, but a more complex notion is likely required for deep networks. Further, in low diversity settings, the diminishing marginal benefit of meta-learning relative to standard retraining is expected. If the tasks are entirely unrelated, or conversely nearly identical, then meta-learning is not the appropriate tool: in the former case, no shared structure exists to exploit, and in the latter, modeling task-specific variation is unnecessary.

We agree with the reviewer that the characterization of task diversity deserves further discussion, particularly in relation to the relative performance of meta-learning versus standard retraining. We will update the paper to include these additional insights, clarifying both the importance of our task diversity assumptions and their implications for performance.

(Q4) Can you comment a bit more in detail about the technical challenges in the paper?

(A4) Thank you for this comment. We are happy to discuss the technical challenges of our analysis and theoretical framework as a whole.

Firstly, we note that our framework differs fundamentally from standard meta-learning analyses focused on recovering a shared linear representation across tasks [1-4]. In those settings, parameter recovery reduces to a subspace recovery problem. In contrast, our framework involves recovering $A^\ast$ from low-rank perturbations $A^\ast + U_t^\ast U_t^{\ast \top}$. Here, the measurements are additive and non-linear (algebraically) in $A^\ast$, making the recovery problem fundamentally different.

The first technical challenge we faced was understanding the solution set of our LoRA-ML objective in Eqn. (14). Remarkably, we show in Theorem 3 that if (i) $T \geq 3$ and (ii) there exist distinct adapters $U_s^\ast$, $U_t^\ast$, $U_r^\ast$ with independent column spaces, then the only solutions minima of Eqn. (14) are the exact ground truth parameters. This is counterintuitive to prior meta-learning analyses, which require that the number of tasks scale with the intrinsic dimension of each task to ensure recovery. However, in our case, no matter the ground truth adapter rank $k$, we can always guarantee recovery given the above assumptions.

The crux of the proof relies on using the set of equalities $U_1^\ast U_1^{\ast \top} = U_1 U_1^{ \top} + U_2^\ast U_2^{\ast \top} - U_2 U_2^{\top} = U_1 U_1^{ \top} + U_3^\ast U_3^{\ast \top} - U_3 U_3^{\top}$ along with the assumption that the combined columns of $U_1^\ast$, $U_2^\ast$, and $U_3^\ast$ are linearly independent to show that $im(U_1) = im(U_1^\ast)$, where $im$ denotes the matrix image. We show that for any $\\{U_i,U_i^\ast\\}_{i=1}^3$ which satisfy both of the above conditions, we must have $im(U_1) = im(U_1^\ast)$. The rest of the proof then quickly follows.

The second challenge we faced related to the landscape of our proposed objective in Eqn. (14). In Theorem 4, we show with $T=2$ tasks, our objective Eqn. (14) has a strict saddle property, meaning that even though it is non-convex, every second-order stationary point (SOSP) is in fact a global minimizer.

This proof is more involved, but we discuss the key steps along with the barrier we face going beyond $T=2$. We note that for a fixed $A$, the objective in Eqn. (14) reduces to separate matrix factorization objectives, but joint minimization leads to challenging interactions between the variables, which form the technical complications. When $T=2$, the analysis reduces to relating the matrix $U_1 U_1^\top-U_2U_2^\top$ to its ground truth counterpart $U_1^\ast(U_1^\ast)^\top-U_2^\ast(U_2^\ast)^\top$. For larger $T$ we cannot make this reduction. The proof follows by contradiction, where we assume the existence of an SOSP $(A,U)$ of Eqn. (14), which does not achieve the minimum function value. For such an $(A,U)$, we prove that $U_1U_1^\top-U_2U_2^\top$ shares an eigenbasis with $U_1^\ast(U_1^\ast)^\top-U_2^\ast(U_2^\ast)^\top$, but $U_1U_1^\top-U_2U_2^\top$ is rank deficient whereas $U_1^\ast(U_1^\ast)^\top-U_2^\ast(U_2^\ast)^\top$ has rank-$2k$ due to the diversity assumption. Thus, there exists a non-zero eigenvector of $U_1^\ast(U_1^\ast)^\top-U_2^\ast(U_2^\ast)^\top$ which belongs to the nullspace of $U_1U_1^\top-U_2U_2^\top$. We use this eigenvector to construct a negative direction of the Hessian, contradicting the fact that $(A,U)$ was an SOSP.

We note that our analysis relied on many subtle properties of the problem, which followed from the assumption of only two tasks. Through numerical experiments, we found strong evidence that our objective in Eqn. (14) can exhibit spurious local minima in some degenerate cases when $T>2$ (please see Appendix E). This sharp transition from $T=2$ to $T=3$ underscores the unique complexity of the meta-learning landscape we consider and highlights the broader challenges of our analysis.

Concluding Remark:

We hope that our responses have adequately addressed your concerns regarding our theoretical assumptions and their practicality, as well as the technical challenges we encountered. If so, we kindly request you to consider revising your score. We are committed to addressing any additional questions you may have during the discussion phase.

[1] Saunshi, N., et al. "A sample complexity separation between non-convex and convex meta-learning." ICML 2020.

[2] Collins, L., et al. "Maml and anil provably learn representations." ICML 2022.

[3] Thekumparampil, K., et al. "Statistically and computationally efficient linear meta-representation learning." NeurIPS 2021.

[4] Du, S., et al. "Few-shot learning via learning the representation, provably." ICLR 2021.

Dear Reviewer JxEW, thank you for recognizing the strength of our analysis as well as the organization of our presentation. We address your questions below.

(Q1) Unfair theoretical setting between standard retraining and meta-learning...

(A1) Thank you for this question which touches on a central difference in the way meta-learning models the different tasks relative to learning them in aggregate, which we refer to as standard retraining (SR).

We consider the problem of preparing a model for efficient adaptation to an unseen, downstream task, using a set of accessible representative tasks drawn from the same task distribution. Meta-learning is a well-established framework for this problem, as it explicitly aims to learn model parameters that can be "easily" adapted. A key aspect of any meta-learning method is defining this "ease" of adaptability. In MAML, for example, the adaptation mechanism is chosen to be a few steps of gradient descent on the task-specific loss. As you mention, this adaptation does not introduce additional parameters, as MAML simply uses a different training algorithm on the same parameter vector compared to standard retraining, which minimizes the average task loss directly.

We focus on parameter-efficient fine-tuning (PEFT) adaptation methods, as these are standard for fine-tuning large-scale models. A common PEFT paradigm is to freeze the given model weights and introduce a small set of additional trainable parameters, like LoRA adapters. To incorporate PEFT with meta-learning, we must include these additional parameters to form the inner meta-learning loss, just as MAML devises its inner loss as the result of applying a fixed number of gradient steps on the task-specific loss. Therefore, when we compare our PEFT-based meta-learning framework to SR, we are comparing two fundamentally different modeling strategies: (i) our approach, which models each task as a PEFT adaptation of a shared model, and (ii) SR, which learns a single parameter by aggregating all task data. Because PEFT methods can rely on additional task-specific parameters for fine-tuning, the fine-tuning adaptation procedure and the parameterization are inseparable.

However, to compare our framework to a method which models all tasks in the aggregate but also includes the same number of trainable parameters as our proposed method, we analyze the following method, which we denote SR+. Consider our theoretical setting where we are given $T$ tasks, where each task $t$ is equipped with the infinite-sample loss function $\mathcal{L}_t^\ast$ defined by the shared parameter $A^\ast \in \mathbb{R}^{d \times d}$ along with the task-specific rank-$k$ adapter with factor $U_t^\ast \in \mathbb{R}^{d \times k}$:

$

\mathcal{L}_t^\ast (A) = \|| A - A^\ast - U_t^\ast U_t^{\ast \top} \||_F^2

$

We define the SR+ method to minimize $\sum_t \mathcal{L}_t^\ast$ just like SR, except it uses the parameterization $A + U U^\top$, where $A \in \mathbb{R}^{d \times d}$ aims to recover the ground truth matrix $A^\ast$, while $U \in \mathbb{R}^{d \times kT}$ represents a rank $kT$ adaptation applied to each task. Formally, SR+ minimizes:

$$\mathcal{L}_{SR+} (A,U) = \sum _{t=1} ^T \|| A + U U^\top - A^\ast - U_t^\ast U_t^{\ast \top} \||_F^2$$

Thus, SR+ uses the same number of parameters as LoRA-ML but still models the tasks in aggregate like SR. We then directly show that it is impossible to recover $A^\ast$ by minimizing this SR+ objective.

Theorem: Let $\hat{U}$ be any $d \times kT$ matrix. Then for the matrix $\hat{A} = -\hat{U} \hat{U}^\top + A^\ast + \frac{1}{T} \sum_{t=1}^T U_t^\ast U_t^{\ast \top}$, we have that $(\hat{A}, \hat{U})$ globally minimizes $\mathcal{L}_{\text{SR+}}$

Thus $A^\ast$ is completely unidentifiable by the SR+ loss, as any value of $\hat{U}$ corresponds to a global minimizer, whereas the only value that leads to recovery $\hat{A} = A^\ast$ is when $\hat{U}$ contains the properly scaled columns of the ground truth factors $U_t^\ast$. However, this is impossible to recover, since any of the infinite values of $\hat{U} \in \mathbb{R}^{d \times kT}$ can result in global minimization. This is in contrast to our LoRA-ML objective, where we show that global minimization always leads to exact recovery of the ground truth parameter $A^\ast$, with a rank $2k$ error only when $T=2$.

We hope this clarifies the rationale behind our setup and addresses your concern about a fair comparison between our method and standard retraining. We are happy to provide further details if helpful.

(Q2) Unclear definition and distinction between meta-learning and multi-task learning...rather than demonstrating a true meta-learning strategy.

(A2) Thank you for this comment. We clarify the interpretation of our method meta-learning rather than standard multi-task learning.

A key feature of meta-learning is the bi-level optimization structure, where the outer objective minimizes a sum of task-specific losses over shared parameters, where each task-specific loss is itself a minimization problem over a task-specific adaptation. This precisely underlies our derivation: in the linear case in Equation (13), the inner problem minimizes the loss for each task after fine-tuning with the task-specific adapter $U_t$, and the outer problem optimizes the shared base model parameter $A$ to improve post-adaptation performance.

Because the LoRA adaptation introduces new parameters to update, the entire meta loss can be rewritten in Eqn. (14) as the joint minimization over $A$ and $\\{ U_t\\}_{t=1}^T$. While this resembles a multi-task learning objective, it is the outcome of a meta-learning formulation with a specific adaptation rather than simply a multi-task loss.

As you mention, in prior model agnostic meta-learning methods such as MAML or Reptile, the inner adaptation runs full gradient steps over all model parameters. Our method aligns with model-based meta-learning, where the focus is on structuring the model (e.g., via PEFT modules) to enable fast adaptation. This explains why our meta-objective can be equivalently expressed in the form of Eqn. (14), since the inner problem introduces new parameters to update with respect to the outer problem.

(Q3) Limited theoretical scope..

(A3) We recognize that our theoretical contributions only directly apply to adapting linear models with LoRA. While this setting allows us to derive precise and interpretable guarantees within the new paradigm of meta-learning via PEFT-based adaptation, we acknowledge that these results do not directly extend to the non-linear or deep networks where we ultimately aim to apply these ideas.

To address the practical setting, we include a set of empirical experiments across vision and language tasks using deep networks, demonstrating that the key principles established theoretically, such as improved adaptation via retraining with our proposed framework, continue to hold in practice. We agree that extending our framework to more complex models is a natural and interesting next step for future work.

(Q4) Counterintuitive result in Theorem 3...

(A4) Thank you for this comment.

Theorem 3 states that, given infinite samples per task, access to 3 independent tasks guarantees exact recovery of the ground truth parameters, no matter the rank of the ground truth adapters. This contrasts with prior meta-learning analyses [1-4] that consider linear representation learning settings, where the number of tasks needed for ground truth parameter recovery depends on the intrinsic task dimension. This difference arises since these previously analyzed settings correspond to subspace recovery problems, but our setting requires estimating a matrix $A^\ast$ from low-rank perturbations $A^\ast + U_t^\ast U_t^{\ast \top}$. For more details on the technical challenges of our setting, please see our response (A4) to Reviewer qeBR.

In terms of the empirical observations that highlight the importance of task-diversity rather than just the number of tasks, we note that this is in fact supported by our result in Theorem 3 which shows that recovery fails with many dependent tasks but succeeds with as few as three independent ones.

Regarding practical task construction, we first note that the result of Theorem 3 is specific to LoRA-based meta-learning for linear models. For other PEFT strategies or model architectures, 3 tasks may not be enough to guarantee exact parameter recovery. Further, in practical settings, we only have access to tasks with a finite number of samples, whereas Theorem 3 considers the infinite sample case. With only finite samples, each task does not give an exact measurement of the true parameters, so increasing the number of tasks can effectively denoise the parameter estimate by providing additional measurements of the true parameters. Thus, it is likely beneficial to construct a larger set of diverse, high-quality tasks in practice.

We will add this discussion relating the result in Theorem 3 to the empirical studies you mention as well as its implications for practical task construction to the revised paper.

Concluding Remark

We hope that our responses have adequately addressed your concerns regarding the theoretical characterization of our framework and the guarantees we provide, along with how they relate to practical insights. If so, we kindly request you to consider revising your score. We are committed to addressing any additional questions you may have during the discussion phase.

[1] Saunshi, N., et al. ``A sample complexity separation between non-convex and convex meta-learning." ICML 2020.

[2] Collins, L., et al. ``Maml and anil provably learn representations." ICML 2022.

[3] Thekumparampil, K., et al. ``Statistically and computationally efficient linear meta-representation learning." NeurIPS 2021.

[4] Du, S., et al. ``Few-shot learning via learning the representation, provably." ICLR 2021.

Dear Reviewer bDJi, thank you for highlighting the novelty and strength of our theoretical contribution, as well as the clarity of our formulation. We answer your remaining questions below.

(Q1) The theoretical results are derived under a linear regression setting with Gaussian noise...The theoretical guarantees do not extend to these more complex settings.

(A1): Thank you for raising this important point regarding the applicability of our theoretical guarantees to more complex model classes and a broader range of loss functions. Our goal is to develop a theoretically principled framework for preparing a model for efficient test-time adaptation using PEFT, and as you correctly note, our theoretical analysis focuses on model adaptation in the setting of linear models with squared $\ell_2$ error. This setting enabled strong theoretical guarantees, yet still gave rise to several counterintuitive phenomena and posed non-trivial technical challenges (please see our response (A4) to Reviewer qeBR for further discussion of these challenges). We agree that extending this framework to encompass more expressive architectures and diverse objectives, particularly those beyond regression, is a valuable direction for future work, especially as our empirical results demonstrate that the principles extend effectively to non-linear models and classification tasks.

(Q2) The proposed framework requires solving a nested optimization problem...the paper does not provide any analysis or empirical comparison of the computational cost, memory usage, or convergence time between the proposed method and the baselines.

(A2) Thank you for the thoughtful question regarding the computational overhead, memory usage, and convergence time of our method compared to standard retraining (SR).

We first address the memory cost. We discuss the memory overhead of LoRA-ML in Appendix D.3, which arises from the additional adapter parameters. If we assume a model has $m$ layers, each parameterized by a $d \times d$ matrix, and uses rank-$k$ LoRA adapters, then SR uses $md^2$ parameters, while LoRA-ML retraining uses $m(d^2 + 2kdT)$ parameters (where $T$ is the number of tasks). Since $k(T+1) \ll d$ in our setting, the total parameter count of LoRA-ML remains $O(md^2)$ during retraining, so LoRA-ML does not incur any additional asymptotic memory complexity relative to SR. At the fine-tuning stage, both LoRA-ML and SR models require the same number of trainable parameters since standard LoRA is applied.

In terms of the computational cost and nested optimization, we note that while our objective is naturally expressed in a nested form (Eqn. (5)), we emphasize that LoRA-ML differs from methods like MAML in a key way: the inner optimization is over lightweight adapter parameters, not full shared weights. This allows us to implement optimization methods for the LoRA-ML objective either via explicit inner/outer loops or through joint optimization over both the shared and task-specific parameters. In fact, we utilize this joint form in our theoretical setting as we start with the nested formulation in Eqn. (13) and instead analyze the equivalent joint formulation in Eqn. (14).

As you mention, the LoRA-ML objective can be empirically optimized using either an outer/inner loop strategy or simple joint minimization, each incurring different computational cost. In our experiments, we explored both strategies. In the linear experiments, we used a structured inner/outer loop: 10 inner steps on task-specific adapters followed by 1 outer step on shared parameters, for 3000 outer epochs. We compared this to SR trained for 3000 steps, since the SR objective is convex and converges quickly. In the larger vision and language experiments, we adopted a joint optimization strategy for LoRA-ML and used the same number of epochs as SR, without explicit inner steps, achieving stronger performance with no additional training loops. We agree that this is an important design choice to highlight and will update the revised version of the paper to reflect this. Further exploration of the tradeoffs between joint optimization and inner/outer-loop strategies is an interesting direction for future work, particularly for improving the efficiency of LoRA-ML at scale.

In terms of convergence time, while we do not provide a theoretical convergence rate comparison, our empirical results show that LoRA-ML achieves better performance within the same epoch budget as SR in large-scale settings. This suggests the convergence of LoRA-ML is competitive in practice.

We will incorporate the discussion of memory usage, computational cost of different optimization strategies, and convergence time into the revised paper.

(Q3) Some results (for example, Table 1 - left) show that the performance...This suggests that the theoretical benefits may not transfer effectively to nonlinear and classification settings and authors should discuss it.

(A3) Thank you for highlighting this point. As you mention, even though our proposed method demonstrates stronger results than standard retraining (SR) across tasks and experiments as a whole, for some specific tasks, the performance gap can vary, especially when the performance variance is taken into account. This behavior is a function of two key characteristics of each experiment: (i) the complexity of the test task relative to the expressivity of the adaptation mechanism, and (ii) the number of available samples per task.

In Table 1 (left), we test our framework through a vision task on CIFAR-10, where each task involves binary classification between two distinct classes. We employ an MLP-Mixer architecture and adapt to the test task after retraining using LoRA. We observe that the performance gain using LoRA-ML relative to SR is not drastic, and the performance variances of both methods are small. LoRA induces a fairly expressive adaptation mechanism for the relatively easier task of binary classification between CIFAR-10 classes, so it is able to recover from potentially suboptimal models recovered by SR during retraining. Further, since we have access to many samples per task, the performance variance of each method is small.

In contrast, in Table 1 (right), we consider the same vision experiment except we adapt using just the last layer fine-tuning, which is significantly less expressive than LoRA. As such, we see a much wider gap between the performance of LoRA-ML and SR while still achieving small performance variance due to the large number of samples in CIFAR-10.

Lastly, in Table 2, we present the performance of our framework through a language task on ConvAI2, a dataset of conversations between personas. Each task corresponds to predicting the dialogue continuation of a specific persona. Relative to CIFAR-10, this experiment is much more complex and is equipped with many fewer samples per task. This leads to a larger variance in per-task performance. However, when we compare the average performance when adapting to all 10 test tasks, we observe that LoRA-ML significantly outperforms SR, even with variances taken into account.

(Q4) More ablation studies can add great value to the paper...

(A4) Thank you for this suggestion. While we did not have time to incorporate additional PEFT methods or larger datasets during the rebuttal period, we include an ablation for the LoRA-ML method demonstrating how the choice of the adapter rank $k$ during retraining affects performance, as this is the sole additional hyperparameter of LoRA-ML relative to the standard retraining procedure. We test using our language modeling experiment, where we aim to learn the responses of different personas in the ConvAI2 dataset. In Table 2 of the paper, we show results using rank-8 adaptations for LoRA-ML along with other methods. Below, we show the effect of different ranks and include standard retraining followed by rank-$k$ LoRA (SR+LoRA) as a baseline.

Test Task Prediction Acc. (↑)

We report the mean accuracy over the 10 test tasks using 5 random trials. The results demonstrate that LoRA-ML outperforms SR across a variety of ranks, demonstrating the effectiveness of our framework and its insensitivity to the adapter rank hyperparameter. We will include these results in the revised version of the paper.

(Q5) This paper is not easy to read for practical and general ML audiences. A short paragraph bridging the theory and experiments, and an overview figure of the pipeline can improve it.

(A5) Thank you for this helpful suggestion. Readability is an important concern, and we have aimed to keep our exposition and theoretical statements as accessible as possible while maintaining technical precision. While we cannot upload new figures during the rebuttal phase, we appreciate your feedback and will add a pipeline diagram in the revised version of the paper, as well as the following text to better connect the theory and the experiments:

``In our theoretical analysis, we showed that minimizing a meta-learning-based retraining objective infused with LoRA (Eqn. 13) leads to provably stronger test-time adaptation compared to standard retraining for a class of linear models. Building on this, we evaluate our general framework for PEFT-based meta-learning (Eqn. 5) in more practical settings, using deeper and more complex architectures, both regression and classification tasks, and various PEFT methods beyond LoRA. Across these experiments, we find that incorporating the PEFT method directly into the retraining objective via our framework consistently improves downstream adaptation performance."

Concluding Remark:

We hope that our responses have adequately addressed your concerns. If so, we kindly request that you consider revising your score. We are committed to addressing any additional concerns you may have during the discussion phase.

Dear Reviewer eo3s, thank you for highlighting our rigorous guarantees as well as our practical experiments. We address your remaining questions below.

(Q1) The results are not tested on strong datasets (CIFAR is a small dataset, and ConvAI2 does not properly capture task diversity IMO). Additionally, the paper assumes "task diversity" and "linear independence" of task-specific adaptations, which, of course, do not hold in real-world scenarios where tasks can be highly correlated; therefore, I think the method should be tested on more realistic datasets.

Thank you for raising this important point regarding the connection between our experimental setup and the assumptions made in our theoretical analysis. Regarding our theoretical framework, we note that assuming some level of task diversity is standard for meta-learning analyses [1-4], as if the tasks are extremely similar, modeling them individually provides little benefit. In our setting, the linear independence among the column spaces of the adapters forms a natural way to model this notion of task diversity.

We acknowledge that this may not hold in practice for real data, and we agree that evaluating our method on additional large-scale and realistic datasets would further strengthen the empirical evidence for its effectiveness. While our goal is to develop a theoretically principled framework for preparing a model for efficient test-time adaptation using PEFT, we recognize that larger-scale evaluation is a natural next step for future work.

(Q2) Lack of ablation for the impact of different ranks or architectures.

Thank you for this suggestion. While we did not have time to incorporate additional architectures during the rebuttal period, we include an ablation for different ranks during the LoRA-ML retraining phase below for the language modeling task, where we aim to learn the responses of different personas in the ConvAI2 dataset. In Table 2 of the paper, we show results using rank-8 adaptations for LoRA-ML along with various other methods. Below, we show how changing the rank affects performance. As a baseline, we include standard retraining followed by rank-$k$ LoRA (SR+LoRA), where we use the same rank for both methods in each column.

Test Task Prediction Acc. (↑)

We report the mean accuracy over the 10 test tasks using 5 random trials. The results demonstrate that LoRA-ML outperforms SR across a variety of ranks, demonstrating the effectiveness of our framework and its insensitivity to the adapter rank hyperparameter. We will include these results in the revised version of the paper.

(Q3) This can be subjective, while I appreciate the formal notations and explicit definitions, I do believe many parts of the write-up can be simplified (or written in simpler notation).

Thank you for this comment. We note that Reviewer bDJi raised a similar point regarding the readability of the paper, which we agree is an important aspect of our presentation. To address this, we will include a new figure illustrating our proposed meta-learning pipeline in an accessible way, along with additional text bridging the theoretical and empirical sections. Please see our response (A5) to Reviewer bDJi for the exact planned wording.

(Q4) Do you have insights for why Reptile outperforms on some tasks, given the theoretical guarantees?

Thank you for this question. Across all of our experiments, Reptile achieves the highest average performance only on the language modeling experiment for Task 1 using both adaptation mechanisms and for Task 6 using just last-layer adaptations. For our language modeling experiment, the test tasks are equipped with relatively few samples, leading to large performance variations across all methods. This leads to larger variation in the per-task performances, but when averaged over all test tasks, we observe that LoRA-ML comprehensively outperforms both Reptile and standard retraining (SR). Thus, these isolated instances where Reptile achieves better performance do not indicate a deeper trend.

(Q5) How did you conclude that "these spurious minima are almost never found in practice?" I do not think the presented experiments suffice for this claim IMO.

Thank you for bringing this point up. This claim is supported by our extensive synthetic data experiments described in Section 4, which test the LoRA-ML objective for linear models, the same setting as our theoretical analysis. We observe, over all experiments, including the additional ablations in Appendix C.1, that minimizing the LoRA-ML objective with vanilla gradient descent and then fine-tuning the recovered shared parameter estimate $\\hat{A}$ consistently minimizes the test-task loss up to the noise floor. If the LoRA-ML objective converged to a spurious local minimum which did not correspond to the ground truth shared parameter A^\\ast, then the test task performance would significantly suffer.

We appreciate the reviewer's concern about this claim and will add plots that track the convergence of $\\hat{A}$ to A^\\ast within our synthetic data experiments to further justify this claim. While we cannot upload new figures during the discussion period, we will include them in the revised version of the paper.

Concluding Remark: We hope that our responses have adequately addressed your questions and are committed to addressing any additional questions you may have during the discussion phase.

[1] Saunshi, N., et al. ``A sample complexity separation between non-convex and convex meta-learning." ICML 2020.

[2] Collins, L., et al. ``Maml and anil provably learn representations." ICML 2022.

[3] Thekumparampil, K., et al. ``Statistically and computationally efficient linear meta-representation learning." NeurIPS 2021.

[4] Du, S., et al. ``Few-shot learning via learning the representation, provably." ICLR 2021.