Continual Optimization with Symmetry Teleportation for Multi-Task Learning

We sincerely thank Reviewer WWrv for the constructive and insightful comments. Below, we provide detailed point-by-point responses to each of your concerns.

• Q1: In eq (4), how do you decide the RHS? How does the model performance and running time vary when you vary the bound on RHS?

A1: In Figure 10 (Appendix), we provide an analysis of the trigger condition on the CelebA dataset. As shown, when the condition is varied from $K/2$, $K/3$, $K/4$, to $K/5$, the corresponding MTL performance scores are [0.93, 0.76, 0.82, 0.65], while the associated training time costs are [40.5, 51.7, 80.5, 112.1] minutes, respectively. Based on this trade-off between performance and computational efficiency, we adopt $K/2$ as the default trigger condition.

• Q2: Eq (7) and and Fig 5 (b) seem inconsistent to me. Since $L_i^\*$ is the loss before teleportation, shouldn't we have $L_i^\*=L_i$ before teleportation? Then shouldn't $L_t$ starts from (0,0)?

A2: We understand the reviewer's concern. Intuitively, if the full dataset or an identical batch order were used to compute both losses, we would expect $L_i^* = L_i$. However, in practice, a randomly shuffled batch-wise training paradigm is adopted, which leads to non-identical losses at the beginning due to inherent sampling variance.

• Q3: Eq (8) and eq (9) seem inconsistent to me. Why does $L(\theta)$ disappear in eq (9)? It seems that you are suggesting ....

A3: We apologize for the confusion — this was a typo introduced for ease of presentation. When applying the weighting factor $R_i$, the term $L(\theta)$ cannot be omitted. However, we would like to clarify that in our actual implementation, we do preserve the subtraction of $L(\theta)$. The relevant code snippet is provided below, and we are committed to releasing our code upon acceptance:

sharp_loss.append(torch.stack([ratio_list[i] * torch.abs(lora_loss[i] - L0[i]) for i in range(n_tasks)]))

Here, ratio_list[i] corresponds to $R_i$, L0[i] represents $L(\theta^\*)$, and lora_loss[i] is $L_i(\theta^\* + \epsilon)$.

• Q4: In line 212, how do you choose $\tilde{n}$? Is there a systematic way to choose $\tilde{n}$? How does the model performance and running time vary when you vary $\tilde{n}$?

A4: For operations akin to zero-order optimization, the gradient approximation is theoretically unbiased in expectation. However, when the number of samples is small (e.g., 1), the variance of the estimate becomes large, resulting in a potentially significant deviation from the true gradient. In our work, we set the number of samples based on common practices in related literature [1]. To further address the reviewer's concern, we conducted additional experiments with 10 and 15 samples. The results are presented below:

As shown, the performance across $n=5$, $n=10$, and $n=15$ remains relatively stable, with the best results observed at $n=15$. However, a larger sample size also incurs greater computational overhead. Hence, we adopt $n=5$ as a trade-off between performance and efficiency.

• Q5: The results for experiments are averaged over 3 seeds, which seems not enough. Improvement in numerical experiments seems marginal when considering the error bar. Averaging over more seeds might reduce the error bar and give a clearer result.

A5: We would like to clarify that nearly all mainstream MTL approaches report results averaged over 3 random seeds (e.g., CAGrad, Nash-MTL, AlignMTL, FAMO, FairGrad, etc.). To further address the reviewer’s concern, we conducted and reported the following additional experiments: (1) As a baseline, we faithfully re-ran FairGrad’s official code over three random seeds (consistent with both their implementation and ours), and reported the performance as FairGrad-R in the table below. (2) As suggested, we further evaluated our method using two additional seeds, and reported the performance as COST (5 seed).

As shown, COST maintains stable and consistent performance when evaluated on more seeds, with only a slight increase in variance. Notably, under the same evaluation protocol, our method demonstrates substantially better stability compared to its baseline, FairGrad-R.

• Q6: minimizing $L_t$ in eq (7) is only an approximation to the loss invariance assumption. What is the stopping criteria? In addition, how to compare the performance under this approximation with the performance under a true loss invariance assumption?

A6: Since training is performed in a batch-wise manner, it is inherently difficult to determine whether the teleported point precisely lies at the symmetric position. Nevertheless, we would like to clarify that even the original symmetric teleportation work [2] does not achieve perfect loss invariance, primarily due to the inexact nature of gradient descent-based optimization.

• Q7: Is COST with HTR more sensitive to parameter tuning vs vanilla COST (lr, batch size, total iteration, etc)? Please clarify.

A7: To address the reviewer’s concern, we conducted the suggested comparison by varying batch sizes in the LoRA optimization and present the results below. As shown, HTR consistently enhances performance across different batch sizes. Furthermore, based on the observed standard deviations, COST with HTR exhibits greater robustness to batch size variation compared to COST without HTR.

• Q8: In Fig 5 (a), I cannot see the K/ 2 Condition bar for NYU v2. Is it 0? If so, please state it clearly.

A8: Since the K/2 condition is only applied to CelebA and not to NYUv2, it is not displayed in the corresponding results. We will revise the text to clarify this distinction.

• Q9: Typos or inappropriate expression.

A9: Thank you for your reminder. We will correct these issues and carefully proofread the paper to ensure clarity and consistency.

If you have any further questions, please don’t hesitate to raise it.

Reference:

[1] ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization. NeurIPS 2018.

[2] Symmetry Teleportation for Accelerated Optimization. NeurIPS 2022.

We sincerely thank Reviewer G25M for the constructive and insightful comments. Below, we provide detailed point-by-point responses to each of your concerns.

• Q1: It seems that the paper does not address the issue of task imbalance in MTL. While the model achieves good performance in most overall metrics, the model's performance varies significantly across different tasks, which raises concerns about the robustness and stability of the proposed approach.

A1: We would like to clarify that the designs in Equation (9) and Equation (10) are specifically tailored to address task imbalance. To facilitate balanced gradient exploration, we first compute the re-weighting factor $R$ based on task gradient norms (Eqn. (10)) prior to teleportation. This factor is then used to re-weight the task sharpness in Eqn. (9), guiding the search toward the most balanced descent direction during teleportation. Additionally, we emphasize that COST consistently achieves state-of-the-art performance across all evaluated datasets, with notable improvements on CityScapes, NYUv2, and CelebA.

• Q2: This method is time-consuming, especially when the trigger condition relaxes. Even with delayed start strategy and frequency control strategy, COST introduces an additional 30% of the training time, making it hard to use in large-scale datasets.

A2: We acknowledge that our method incurs an approximately 30% increase in training time, which we recognize as a limitation and have explicitly discussed in the paper. However, we believe this additional cost is justified by the substantial benefits it offers: (1) Our approach explores a novel perspective in MTL, fundamentally distinct from existing optimization-based MTL methods, whose effectiveness remains a subject of debate [1,2]. (2) By incorporating and adapting symmetric teleportation into deep MTL, our method not only consistently achieves state-of-the-art performance across widely adopted MTL benchmarks, but also remains orthogonal and complementary to existing optimization-based approaches.

• Q3: How can the design of R in Equ.10 mitigate imbalance issues and what’s the performance of this design?

A3: Please refer to A1 for further clarification. As additionally suggested, we conducted an ablation study to assess the impact of the re-weighting factor $R$. The results are presented below:

As shown, removing the re-balancing design leads to a notable performance degradation, indicating the effectiveness of our proposed re-weighting mechanism.

• Q4: Does the delayed start strategy provide any beneficial effects on model effectiveness or convergence, in addition to saving training time?

A4: Currently, the delayed start strategy is adopted purely as a time-saving mechanism.

• Q5: What’s the performance of COST-weak when relaxing from [K/2, K/3, K/4, K/5]？Does COST-dominated offer the best performance in most datasets?

A5：We would like to clarify that we did not evaluate COST-weak on multiple datasets for the following reasons: (1) Defining weak conflicts becomes increasingly ambiguous and challenging in scenarios involving more than two tasks. (2) Even under the dominated conflict condition, CelebA already exhibits frequent teleportation triggering, incurring considerable computational overhead—this would be exacerbated under the weak condition.

Given the limited rebuttal timeframe, it is infeasible for us to conduct such extensive additional experiments. Moreover, as illustrated in Figure 2, weak conflicts may not pose a significant issue when task gradients exhibit approximately equal norms. Finally, Table 7 already presents a comparison between COST-dominated and COST-weak, showing that the former substantially outperforms the latter.

If you have any further questions, please don’t hesitate to raise it.

Reference:

[1] Do Current Multi-Task Optimization Methods in Deep Learning Even Help? NeurIPS 2022.

[2] Can Optimization Trajectories Explain Multi-Task Transfer? TMLR 2024.

We sincerely thank Reviewer sH9s for the constructive and insightful comments. Below, we provide detailed point-by-point responses to each of your concerns.

• Q1: Why do we even need LORA? We can simply optimize the backbone parameters with Eq.11 when conflict occurs.

A1: Yes, it is feasible to directly optimize the backbone for symmetric teleportation. In our work, we employ LoRA primarily for efficiency purposes.

• Q2&Q3: In Section 5.2, I see no support on the statement 'These results might address ... by LoRA?' Could the authors elaborate that? To prove that the improvement is not from LORA, the authors should add the following ablation study: ...

A2&A3: Thank you for your valuable suggestion. We have conducted the comparison experiments accordingly, and the results are presented below:

As shown, applying LoRA alone does not lead to performance gains. One possible explanation is that the paradigm of alternating between pre-training and LoRA-based fine-tuning may not be optimal for this setting.

• Q4: Why do we use L1 norm in eq. 7? What would happen if we use L2 norm?

A4: Thank you for your valuable suggestion. We conducted an additional experiment by replacing the L1 norm in Eq. (7) with the L2 norm, and present the results below. As shown, the L2 norm performs slightly worse than the L1 norm, but still achieves SOTA performance.

• Q5: What would happen if we relax the trigger to 0? By that I mean change the r.h.s. of eq. 4 to 0. Will it significantly improve the results?

A5：As illustrated in Figure 5, setting the right-hand side (RHS) of Eq. (4) to zero results in frequent teleportations, with a conflict ratio of 97% per epoch. Due to the limited rebuttal time, we were unable to conduct further experiments under this configuration. Nevertheless, as shown in Figure 10 (Appendix), reducing the threshold from $K/2$ to $K/3$, $K/4$, and $K/5$ yields results of 0.76, 0.82, and 0.65, respectively. These outcomes are generally consistent with our expectations and further support the rationale behind our design choices.

If you have any further questions, please don’t hesitate to raise it.

We sincerely thank Reviewer Zv8Z for the constructive and insightful comments. Below, we provide detailed point-by-point responses to each of your concerns.

• Q1: Please quantify how often teleportation is triggered in practice and its cumulative time cost. Could more dynamic or adaptive frequency strategies be explored?

A1: As shown in Figure 5, teleportation is triggered infrequently when using the dominated conflict condition on NYUv2 and the $K/2$ condition on CelebA. We further report the per-epoch computational overhead of our method on CelebA in Figure 10 (Appendix), which confirms its efficiency. While our current design uses fixed conflict-triggering criteria, recent studies [1,2] suggest that it may not be necessary to apply MTL at every optimization step. In the same spirit, not every conflicting step may require teleportation. Designing dynamic triggering strategies could therefore be beneficial, though this remains an under-explored area. We thank the reviewer for this insightful suggestion and plan to investigate it further in future work.

• Q2: Can the authors better explain the $L_g$ surrogate used to encourage stronger gradients after teleportation. What metric is being maximized in practice?

A2: In our symmetric teleportation framework, we optimize two objectives: loss invariance $L_t$ and balanced gradient maximization $L_g$. For the design of $L_g$, we first compute a re-weighting factor $R$ based on task gradient norms (Eqn. (10)) prior to teleportation. This factor is then used to re-weight task sharpness in Eqn. (9), thereby guiding the search toward a direction that maximizes balanced gradients during teleportation.

Given that the negative gradient direction represents the steepest local descent, we use sharpness as a surrogate to approximate this descent direction. Specifically, sharpness is estimated by sampling perturbations from the unit sphere multiple times and taking the maximum perturbed loss as the sharpness measure. This approach shares a similar motivation with point-wise gradient estimators commonly used in online optimization literature [3,4].

• Q3: How sensitive is performance to LoRA rank? Why did LoRA outperform LoHa and OFT? Would dynamic rank tuning help?

A3: To address the reviewer’s concern, we conducted a comparison of LoRA ranks and present the results below:

As observed, increasing the rank leads to marginal improvements in MTL performance; however, it also results in increased training cost. Therefore, we adopt the default setting (R=5) to maintain a balance between performance and efficiency.

While a deeper investigation into LoRA is beyond the scope of this work, we note that similar findings have been reported in recent PEFT studies [5,6], where LoRA consistently ranks among the top-performing approaches. Intuitively, dynamically adjusting the rank may be beneficial, as the difficulty of finding a desirable symmetric point during teleportation may vary across steps. We thank the reviewer for this insightful suggestion and will consider exploring this direction in future work.

• Q4: How might COST be adapted to models where LoRA/PEFT is less applicable (e.g., GNNs or Transformers)?

A4: We would like to clarify that LoRA can indeed be applied to Transformer architectures. We understand and appreciate the reviewer’s concern regarding its applicability in graph-based models. Currently, most GNN frameworks offer limited support for LoRA and other PEFT techniques. To accommodate this, in our design, we apply LoRA only to the final linear head of the GNN.

For instance, in one of our evaluated datasets, QM9, which employs a GNN for regression tasks, we restrict teleportation and LoRA adaptation to the last linear layer. Despite this minimal intervention, our approach still achieves notable performance improvements, suggesting the effectiveness of even partial LoRA integration in non-Transformer models.

• Q5: Could imperfect teleportation or over-teleportation lead to optimization instability? Any examples observed?

A5: In our current experimental setting, we did not observe optimization instability. However, to proactively address the reviewer’s concern, we manually constructed an imperfect teleportation scenario. Specifically, we performed early stopping halfway through the LoRA optimization phase, and present the resulting performance comparison below:

As shown, imperfect teleportation—caused by premature termination of LoRA optimization—results in degraded MTL performance. This observation supports the importance of fully optimizing teleportation steps to ensure the effectiveness of our method.

• Q6: Limitation: COST introduces 30% training time overhead and added complexity.

A6: We acknowledge that our method incurs an approximately 30% increase in training time, which we recognize as a limitation and have explicitly discussed in the paper. However, we believe this additional cost is justified by the substantial benefits it offers: (1) Our approach explores a novel perspective in MTL, fundamentally distinct from existing optimization-based MTL methods, whose effectiveness remains a subject of debate [7,8]. (2) By incorporating and adapting symmetric teleportation into deep MTL, our method not only consistently achieves state-of-the-art performance across widely adopted MTL benchmarks, but also remains orthogonal and complementary to existing optimization-based approaches.

If you have any further questions, please don’t hesitate to raise it.

Reference:

[1] Multi-Task Learning as a Bargaining Game. ICML 2022.

[2] Injecting Imbalance Sensitivity for Multi-Task Learning. arXiv:2503.08006, 2025.

[3] Introduction to Online Convex Optimization. Foundations and Trends® in Optimization, 2016.

[4] Online convex optimization in the bandit setting: gradient descent without a gradient. arXiv preprint cs/0408007, 2004.

[5] Empirical analysis of the strengths and weaknesses of peft techniques for llms. arXiv:2304.14999, 2023.

[6] Quantum-PEFT: Ultra parameter-efficient fine-tuning. ICLR 2025.

[7] Do Current Multi-Task Optimization Methods in Deep Learning Even Help? NeurIPS 2022.

[8] Can Optimization Trajectories Explain Multi-Task Transfer? TMLR 2024.

Dear Reviewer Zv8Z,

As the deadline for the author–reviewer discussion phase approaches, we would like to kindly ask whether our responses have addressed your concerns.

Best regards,

The Authors of Paper 9185