Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences

We thank reviewer EQhb for their constructive feedback. We are glad that they find our discussion interesting and the algorithm more memory efficient and faster than baselines. Additional experiments have been added to the appendix and changes in the manuscript are in blue font. We address their points below.

Training Sarcos for longer

Thank you for the suggestion. We have repeated the same experiment for Figure 3(c) but now train for 500 epochs. The results are presented in Figure 14 of the new Appendix F. Again, we see that PaLoRA outperforms PaMaL and converges faster, even for lower number of forward passes $M$.

Ablation on the ranks

We have added an ablation over the different ranks in the appendix, in Table 3.. We have used the Cityscapes benchmark and considered ranks $r\in\{1,2,4,8\}$, our proposed fixed deterministic schedule and $M\in\{3,5\}$. We use 3 seeds per combination. Rank 1 results in an excess of approximately 1.05% of the parameters of the MTL model. The memory requirements for the other cases scale linearly with the rank. We see that the method is robust to the choice of rank.

Providing standard deviations

MultiMNIST-3 is more difficult compared to MultiMNIST, as MultiMNIST is more difficult compared to MNIST. The reason is that in the construction of MultiMNIST, there is an overlap between the digits. This overlap is further increased in the case of MultiMNIST-3, resulting in lower accuracies. We have provided the standard deviations for MultiMNIST in the Appendix. The baseline numbers for Table 1 are drawn directly from [1] and, therefore, we do not have the standard deviations. However, we have added the standard deviation for NYUv2 (Table 2) in the new appendix.

Experiment on UCI-Census

We did not consider UCI-Census since the network used in [1] is a single-layer MLP with 256 neurons. Therefore, parameter-efficient approaches such as PaLoRA have no added value to such small settings. To counterbalance the lack of Census, we have included the rest of the experiments from [1], namely MultiMNIST, MultiMNIST-3, UTKFace and Cityscapes following their implementation. Furthermore, we have expanded beyond the experimental settings of [1] to include the most challenging benchmark in NYUv2 as well as SARCOS which has 7 tasks. In both cases, we show that PaLoRA outperforms the current PFL state-of-the-art PaMaL.

Weakness 5 Thank you for drawing our attention to this. We have updated Figure 3b.

We hope that our response has addressed the reviewer’s comments. We kindly ask them to reconsider their score.

[1] Dimitriadis N, Frossard P, Fleuret F. Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models. ICML 2023

We thank the reviewer for their feedback. Additional experiments have been added to the appendix and changes in the manuscript are in blue font. We reply to the points raised below.

Discussion between PaLoRA and MoE-like methods

We appreciate the reviewer’s suggestion to discuss comparisons with MoE-like methods. First, MoEs are typically employed in very large-scale models (e.g., foundation models) to handle heterogeneous inputs or datasets, often relying on sparse routing mechanisms. In contrast, PaLoRA is designed for end-to-end training with multi-task settings where all tasks originate from the same input but the supervision is different (e.g. in Cityscapes the scene of a road is the input and the tasks are semantic segmentation and depth regression). Therefore, PaLoRA is input agnostic while MoEs are input-specific. Moreover, unlike MoEs, which make discrete routing choices (topk routing), our method explicitly aims for a continuous parameterization of the Pareto Front to support fine-grained control and flexibility across objectives. Finally, by integrating LoRAs into the core model during inference, PaLoRA maintains the same memory and inference costs as the base model, whereas MoEs inherently introduce additional computational overhead and memory due to the routing mechanism. That’s the reason why we do not discuss MoEs in the paper.

Discussion between PaLoRA and adaptive weight learning methods

We have already compared against several baselines that can be considered adaptive weight learning methods. For instance, 9 out of 12 MTL baselines (all excluding LS, RLW and Graddrop) could be considered as adaptive. However, they are different from our approach since they only find one point in the Pareto Front, while we (and Pareto Front Learning methods in general) continuously parameterize the entire Pareto Front in one training run. Therefore, compared to the MTL baselines, PaLoRA has a more general and difficult goal. Nonetheless, our experiments show our method performs at par or better, while remaining efficient. Finally, our Pareto Front Learning baselines can also be considered adaptive weight learning methods and differ in their weight generating mechanism, e.g., [1] uses hypernetworks, [2] uses weight ensembles, while we used task-specific LoRAs. Beyond the much lower memory requirements of our method due to this architectural change, we are also the first to address the sampling mechanism underlying Pareto Front Learning and we show it plays a crucial role, leading to superior results.

Differences with general PFL applications

Can the reviewer please clarify what they mean by general Pareto Front Learning (PFL) applications? Past works on Pareto Front Learning such as [1] and [2] differ in their parameterization of the Pareto Front; [1] uses a hypernetwork while [2] a weight ensemble. Our choice to train LoRAs and one copy of the original model (instead of multiple model copies like [2]) significantly reduces the number of parameters and helps in optimization, e.g., for NYUv2 benchmark in Table 2 our parameter excess if 6.2% and for [2] it is 200% while we achieve better results.

Besides the architectural component, we are the first to focus on the sampling component of Pareto Front Learning (§4.2) where we show that it really improves both convergence and validity of the final Pareto Front (in the entire experimental section and in more detail in §6.1). Overall, these contributions achieve state-of-the-art results, while being more efficient.

GFLOPS/memory comparison

During inference, if a preference ray has been selected by the user, PaLoRA and all other methods we consider have exactly the same GFLOPS/memory requirements. If a preference ray has not been selected, the target model must first be constructed for PFL methods, resulting in additional memory and computation. However, the construction of a model does not need to be performed each time, but only when the user wishes to change the trade-off of the model.

For the comparison, we include an experiment on SegNet with Cityscapes, assuming a batch size of 10. We compare only with PaMaL as a PFL method, since PHN and COSMOS do not scale to this benchmark.

In terms of memory:

• MTL Model size in GPU: 100.13 MB (this is the same for PaLoRA and PaMaL if the preference is set)
• PaLoRA model size in GPU: 104.77 MB (rank=4 as in Table 1)
• PaMaL model size in GPU: 200.26 MB (duplicate memory because we have two tasks)

The forward pass for all MTL models is:

• MTL Forward pass: 15.0716 ms

While the construction times for the PFL methods that scale to this benchmark are:

• PaLoRA model construction: 3.9677 ms
• PaMaL model construction: 1.6577 ms

We thank the reviewer for their feedback. We are glad that they found our paper well written and appreciated our idea. Additional experiments have been added to the appendix and changes in the manuscript are in blue font. We address their comments below.

Related works with LoRA

Thank you for drawing our attention to these works. We have included them in our updated related work section. It is important to note that [1,2,3] are considered contemporaneous works according to ICLR guidelines, since [1] was presented in November (after submission), [2] in August and [3] will be presented in December. Our proposed method is materially different to all these approaches in both settings and objectives. Specifically, we train both LoRAs and the original network from scratch (with the exception of §5.4) while [1,2,3,4] focus on fine-tuning from foundation models.

In terms of objective, we focus on parameterizing the entire Pareto Front with one network, but

• [4] focuses on improving fine-tuning towards a single solution, similar to all our MTL baselines, by learning a routing mechanism for multiple already trained LoRAs,
• [2] on dynamically composing already trained LoRAs and ,
• [1,3] are direct LoRA replacements and not specific to multi-task scenarios.

For these reasons, none of these works are comparable with our setting. We hope that our comment clarifies the vast differences with these approaches.

Clarification on Figure 2

We have updated the caption to make the plot easier to understand. This plot explains how the sampling works between previous works (subfigures a, b) and our proposed deterministic sampling (subfigures c, d). Each dashed line represents the sampling of preference rays for one batch and the arrow on the left shows the direction of time. Following [5], depicted in subfigure b, we sample multiple preference rays $\lambda$ per batch and perform as many forward passes, but each time the weights of the model change according to Equation 2. The plot shows the case for two tasks, where the preference ray is $[\lambda, 1-\lambda], \lambda\in[0,1]$. Therefore, we only depict the weight for the first task in Figure 2. The plot does not present any experimental results, i.e., we do not show the quality of the produced mappings in this plot. The quality of the mappings can be seen in Figures 3ab, 4 and (quantitatively) in Figures 5b, 11, 12, 13 where we show that the proposed deterministic sampling results in more valid Pareto Fronts. We hope that this has clarified the reviewer’s questions.

On the number of compared methods of Figure 3

Each subplot of Figure 3 presents a different perspective on the results. Figure 3b offers a qualitative result showing that PaLoRA can retrieve a wide Pareto Front for 3 tasks. We do the same for another dataset in Figure 7 of the appendix, following PaMaL [5]. We believe that adding more PFL methods would make the plots too dense and hard to parse for the readers. This is why we opted to present results as this. Therefore, we have only presented all the baselines on subfigure a, since the dataset has two tasks.

For Figure 3c, we selected to compare PaLoRA with the state-of-the-art PFL method PaMaL [5] and use a single MTL method as a frame of reference. We selected the simplest one in Linear Scalarization. We did not add more MTL or PFL methods since the goal of this plot is to show how better PaLoRA scales compared to PaMaL for more tasks. We have expanded on the SARCOS experiment in the appendix for the case of longer training (500 instead of 100 epochs), where we observe again that PaLoRA scales better to PaMaL.

Clarification on the purpose of the method

It is important to note that [1,2,3] are considered contemporaneous works according to ICLR guidelines, since [1] was presented in November (after submission), [2] in August and [3] will be presented in December. The main purpose is accelerating convergence and reducing memory requirements only when compared to PFL methods, such as PHN and PaMaL. Therefore, our method is on a different class of methodologies compared to MTL approaches, which find only one point in the Pareto Front. We have added two sentences to highlight this: a sentence in line 53 as well as a sentence in the beginning of Section 4.2. We do include standard multi-task learning methodologies to show that, despite our objective being more general and harder, we still compete or even outperform them. Our experiments and baseline selection follow the sota in PFL [5] but we push the boundary with the even more challenging benchmark NYUv2. Compared to [1-4], both settings (fine-tuning vs training from scratch) and objectives (one point in the Pareto Front vs continuous parameterization) are different, as explained in more detail in Weakness 1.

We thank reviewer 9e2E for their constructive feedback. We are glad that they find our deterministic sampling a notable improvement, our use of low-rank adaptors relevant to real-world applications, and our visualizations rich. Additional experiments have been added to the appendix and changes in the manuscript are in blue font. We address their points below.

On the practical benefits of Pareto Front Learning

Thank you for the suggestion. We have added some application domains in the second paragraph of the introduction to highlight the practical benefits of Pareto Front Learning. Indeed, Multi-task learning seeks to produce a single model. However, constructing the Pareto Front (PF) provides flexibility for scenarios where task priorities shift dynamically, or the user-trade-offs are not known in advance. For example, traffic conditions in autonomous driving may demand trade-offs between segmentation and depth estimation. This flexibility is valuable across diverse applications. The PFL practical benefits can also be traced back to the academic interest in the topic with Multi-objective optimization algorithms such as NSGA-II [6] (with over 50k citations) addressing smaller scale settings using genetic algorithms and Pareto Hypernetworks [3] adapting the construction of a continuous Pareto Front in neural networks. Finally, a recent work [6] (to appear in NeurIPS) focuses on adapting a single LLM to a broad spectrum of human preferences in a Pareto-optimal manner, while [7] focuses on multi-objective generation.

Our experimental setup, therefore, is based on the PFL literature and actually expands on it to show the scalability of our method. For instance, [3] performs experiments up to Cityscapes but with small networks (ENet), [4] scales up to networks like SegNet and we further explore NYUv2, a more difficult benchmark with 3 tasks instead of 2. For these reasons, OOD scenarios are beyond the scope of the paper. We hope that our response and the addition of application domains in the introduction of the manuscript have clarified the benefits of PFL.

Random vs deterministic schedule and Fixed vs Annealed

Modifying the random sampling distribution inevitably introduces hyper-parameters, while a fixed schedule does not. Furthermore, even if a better distribution is selected, the stochasticity can again lead to pernicious updates, discussed in Section 4.2, due to an imbalanced concentration of sampled rays. This is particularly important for lower values of $M$ (the number of forward passes). Our ablation study in the appendix (Figures 11, 12 and 13) show that a fixed deterministic schedule results in a more valid and functionally diverse Pareto Front (y axis for all plots) compared to random sampling. Regarding the benefits of annealed vs fixed deterministic sampling: annealed sampling allows for lower values of M (marker shape in the aforementioned plots) which translates to lower computational requirements. Finally, it is important to note that the major performance boost comes from replacing the random sampling with a deterministic one, while an annealed deterministic schedule compounds on these benefits.

On the possibility of layer-wise scaling

Indeed the behavior is not homogenous across the entire model, as a long line of literature shows that low-level features lie in early layers and task-specific on deeper layers. In our setting, it is imperative that each task is associated with a scalar coefficient $\lambda_t$ because we use the same scalar to weigh the loss for the same task $\mathcal{L}_t$. Doing so steers the model towards the desired Pareto properties and, during inference, users can specify their desired trade-off and get the associated model weights. In contrast, works like [1,2] focus on attaining a single model in a post-training setting.

It would be really interesting to check whether in this after-training setting we can merge two models towards constructing a Pareto Front simply by looking at the coefficients. In that setting, intuitively, an annealed schedule on the layers does make sense.

Regarding layer-wise scaling during training, preference rays are sampled from T-dimensional simplex $\Delta_T=\{{\lambda}\in\mathbb{R}^T_+: \sum_{t=1}^T\lambda_t=k\}$ for $k=1$. Per the reviewer’s suggestion, applying layer-wise scaling during training could be done by adding a schedule on $k$ dependent on the layer. The simplest could be linear, i.e., for layer $\ell$ out of $L$, the “modified” simplex will use $k_\ell=\frac{\ell}{L}$. Effectively, lower layers will use a lower range in the task-specific ray coefficients. However, the network weights may naturally adapt making the layer-wise schedule not needed at all or more complex schedules might be needed for the layer-wise scaling to work. This is an interesting idea, but introduces multiple hyperparameters, and therefore goes beyond the scope of our work.