Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning

Strengths

• Problem is relevant and well-motivated. The paper tackles a fundamental challenge in fine-tuning across multiple domains. This issue is highly relevant as real-world applications increasingly demand multi-domain adaptability. The authors clearly articulate the shortcomings of existing solutions (e.g., joint fine-tuning, domain adversarial training) and provide strong motivation for their proposed partition-based approach by grounding it in both practical scenarios and theoretical considerations.
• Framework is coherent and mostly clear. The proposed framework is logically structured and well-integrated with the theoretical analysis. The use of a synergy-discrepancy-capacity trade-off objective offers a principled way to partition domains and fine-tune in stages. The theoretical part supports the methodology.
• Extensive experiments. The experimental section is thorough. The authors evaluate across a diverse set of language understanding tasks using multiple backbone models (LLaMA, Falcon). They include comparisons with strong baselines across full fine-tuning, adapter-based methods, domain adaptation techniques, and data selection strategies. Detailed ablation studies and diagnostic analyses further validate the efficacy and robustness of the approach.

Weaknesses

• The theoretical analysis is presented as a rigorous justification for the proposed algorithm, but the connection is tenuous and potentially misleading. The synergy term is introduced heuristically into the objective function G (Equation 12) and unclear how the JS divergence relates to Definition 2. The theory seems to be more as a post-hoc rationalization for a heuristic objective rather than a foundational proof of its optimality. My suggestion is to tone down the claims about the theoretical justification and re-frame the theory as a motivation for their objective.
• Novelty: The core idea of grouping related tasks/domains to improve multi-task learning is not new. This concept has been explored before in the multi-task learning and continual learning literature (e.g., task clustering, routing), e.g., [1,2]. The paper fails to properly position itself within this broader context, overstating its conceptual novelty.

[1] Jacob, L., Vert, J. P., & Bach, F. (2008). Clustered multi-task learning: A convex formulation. Advances in neural information processing systems, 21.

[2] Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, J., & Savarese, S. (2020, November). Which tasks should be learned together in multi-task learning?. In International conference on machine learning (pp. 9120-9132). PMLR.

From perspective of the four dimensions, I can see strengths and weaknesses of this paper as following:

Quality: Strengths: The idea of partitioning training domains to maximize synergy is a fresh perspective in the context of LLM fine-tuning. Prior works in multi-domain or multi-task learning have often used techniques like adversarial training to make a single model invariant to domain differences, or data sampling strategies to balance domains, or at most simple sequential fine-tuning. By contrast, this paper explicitly clusters domains and fine-tunes in stages, which is a novel formulation. The authors combine concepts from different areas in an original way: they use adapter modules for domain-specific capacity (building on parameter-efficient fine-tuning literature) and introduce a clustering algorithm driven by a custom objective (balancing discrepancy and synergy) – this is not a standard approach in prior work. The theoretical derivation of a “synergy offset” term in generalization bounds is, to my knowledge, new. It extends domain adaptation theory to incorporate a notion that certain domains together can effectively reduce error (negative transfer offset) instead of only increasing it. The method also outperforms recent ideas like Tag-LLM (which used task-aware gating) and S2L (which did curriculum ordering), indicating that the authors’ approach isn’t just a minor tweak on those, but a qualitatively different strategy. In summary, the work’s novelty lies in reframing multi-domain fine-tuning as a partitioning problem with a provable optimal grouping, which is a creative and significant departure from straightforward multi-task training or domain alignment approaches.

Weaknesses: The experiments focus on four domains, which is a reasonable starting point, though scaling to more would further strengthen the claims. The setting also blends multi-domain and multi-task learning, and the paper could clarify how differences in task formats are handled. Additionally, the synergy metric is a heuristic based on lexical and embedding similarity; while intuitive, alternative definitions are not explored. Theoretical guarantees rely on standard assumptions (e.g., Lipschitz continuity, bounded norms), which are enforced in practice but may constrain flexibility. Overall, these are minor concerns in an otherwise strong and well-executed paper.

Clarity:

Strengths: The paper is well-written and structured in a logical manner. The introduction clearly motivates the problem of negative transfer when fine-tuning on multiple domains and introduces the key idea of partitioning domains to exploit “synergy over discrepancy.” The method is described with both conceptual reasoning and formal algorithmic steps (Algorithm 1 provides a clear step-by-step overview). The theoretical section is dense but does a good job laying out definitions and assumptions before presenting the theorems, which helps the reader follow the derivations. Important details such as the objective function for partitioning (including the role of lambda balancing synergy vs. discrepancy) are given, and the authors include an illustrative Corollary 3.1 to intuitively explain that high-synergy domains will optimally group together. Figures and tables are effective: Table 1 neatly summarizes the performance across models and tasks, and the improvements are easy to see (with best results bolded). The ablation tables and the memory usage table further clarify the trade-offs. In general, the writing is professional and clear, making the complex approach reasonably understandable.

Weaknesses: While the paper is generally clear, a few points could be more precise. It’s unclear how domain-specific adapters are selected—whether via explicit domain IDs or automatic routing—during training and inference. The handling of different task formats (e.g., summarization vs. classification) is also not fully explained, which affects reproducibility. Additionally, the synergy metric is only briefly described, and a clearer exposition or example would aid understanding. Clarifying that the “domains” correspond to distinct tasks would also help set reader expectations. These are minor issues in an otherwise well-written paper.

Significance:

Strengths: This work tackles an important practical challenge: efficiently fine-tuning one large model on multiple data sources without interference. As large models are increasingly adapted to specialized domains, the ability to handle multiple domains/tasks in one model (instead of training separate models or naively pooling data) is valuable – it can save deployment costs and potentially lead to models that generalize more broadly. The paper’s key insight is to formalize and exploit the idea that some domains can actually benefit each other (synergy) if trained together, while very disparate domains should be isolated to avoid damage. This is a nuanced departure from prior multi-domain adaptation work that mostly focused on mitigating negatives (interference) rather than actively leveraging positives (complementarity). By providing a principled algorithm and theoretical guarantees for this idea, the paper could inspire further research into adaptive multi-domain training strategies. In terms of empirical significance, the method delivers state-of-the-art results on the benchmarks considered. The gains over strong baselines like Tag-LLM (task-aware prompt tuning) and S2L (a curriculum clustering approach) – though modest in absolute numbers – demonstrate that partitioning strategy yields consistent improvements across varied tasks and model scales. Even a 1 point increase in ROUGE or accuracy can be meaningful in competitive settings, especially since it’s achieved alongside reduced memory usage. Moreover, the approach is model-agnostic and could be applied to different LLM architectures, which broadens its potential impact. Overall, the paper’s contributions feel significant for the subfield of multi-domain learning with LLMs, providing both theoretical insight and a practical technique that outperforms existing methods.

Weaknesses: The improvements, while consistent, are modest—typically around +0.5 to +1.0 in accuracy—which may feel incremental given the added complexity. The method focuses on diverse tasks rather than classic multi-domain scenarios (e.g., same task across domains), which slightly limits its demonstrated breadth. Also, it assumes known domain labels and pairwise metrics, which may not hold in more dynamic real-world settings. Still, within the scope of fine-tuning on a fixed set of domains, the contribution is meaningful and practical.

Originality:

Strengths: The idea of partitioning training domains to maximize synergy is a fresh perspective in the context of LLM fine-tuning. Prior works in multi-domain or multi-task learning have often used techniques like adversarial training to make a single model invariant to domain differences, or data sampling strategies to balance domains, or at most simple sequential fine-tuning. By contrast, this paper explicitly clusters domains and fine-tunes in stages, which is a novel formulation. The authors combine concepts from different areas in an original way: they use adapter modules for domain-specific capacity (building on parameter-efficient fine-tuning literature) and introduce a clustering algorithm driven by a custom objective (balancing discrepancy and synergy) – this is not a standard approach in prior work. The theoretical derivation of a “synergy offset” term in generalization bounds is, to my knowledge, new. It extends domain adaptation theory to incorporate a notion that certain domains together can effectively reduce error (negative transfer offset) instead of only increasing it. The method also outperforms recent ideas like Tag-LLM (which used task-aware gating) and S2L (which did curriculum ordering), indicating that the authors’ approach isn’t just a minor tweak on those, but a qualitatively different strategy. In summary, the work’s novelty lies in reframing multi-domain fine-tuning as a partitioning problem with a provable optimal grouping, which is a creative and significant departure from straightforward multi-task training or domain alignment approaches.

Weaknesses: The approach builds on established ideas—adapters, domain discrepancy, and clustering—assembled in a thoughtful and original way. While each component individually is not novel, their integration for multi-domain LLM fine-tuning is new and well-motivated. The synergy metric is heuristic and could benefit from comparison to alternatives, but it’s a reasonable starting point. Overall, the contribution feels fresh despite leveraging known tools.

Strengths:

1. The paper is well-motivated, tackling the significant challenge of domain conflict and negative transfer in multi-domain LLM fine-tuning, a prevalent issue in real-world applications.
2. A complete generalization bound analysis is derived for the multi-domain fine-tuning problem. This theoretical insight directly motivates and justifies the proposed partition mechanism, providing a robust scientific basis for the approach based on domain discrepancy and synergy.
3. the empirical results on 4 various real-world finetuning tasks demonstrate the effectiveness of the proposed method, which consistently outperforms baseline approaches across all the tasks.
4. The comprehensive ablation studies demonstrate the robustness of the proposed method on hyperparameter choices.

Weaknesses:

1. The algorithm, implementation and complexity of the partition mechanism require more explanation. According to Equ. 12, it is not clear how to estimate $\Delta \theta^t$ and $\phi_j^t$ without retraining the model? or are they directly be replaced by the parameter shift bounds $\rho_{\theta}, \rho_{\phi}$?
2. The empirical evidence on the current 4 tasks are limited to relatively easy tasks. Could you give the results on the reasoning tasks, e.g. Hellaswag, MMLU, ARC, SciQ, GSM8K?
3. Lack of comparison to some conventional, reweighting-based multi-task learning approaches, e.g. iMTL [1], FAMO [2], ExcessMTL [3].

[1] Towards Impartial Multi-task Learning. Liu, et. al, 2021. [2] FAMO: Fast Adaptive Multitask Optimization. Liu, et. al, 2023. [3] Robust Multi-Task Learning with Excess Risks. He, et. al, 2024.

Strengths:

(1) Conceptual Novelty & Theory. The paper frames multi-domain fine-tuning as minimising a partition objective $G$ that balances synergy, discrepancy, and adapter cost, then proves new generalisation bounds showing this choice tightens worst-domain risk. Coupling these guarantees with an $O(k^{2}\log k)$ agglomerative clustering yields a rare, theory-backed PEFT recipe.

(2) Consistent Empirical Gains. Across four tasks (news‐sum, sentiment, QA, topic) and three backbones (LLaMA-2 7B/13B, Falcon-40B), PMS-FTP beats eleven strong baselines—including full FT and the latest PEFT methods—by +0.8–2.0 points on standard metrics, demonstrating robustness to both task diversity and model scale.

(3) Resource Efficiency. Stage-wise norm bounds cut GPU memory by $\sim$32 % versus full fine-tuning (e.g., 18.4 GB vs.\ 27.2 GB on an A100) while still improving accuracy; the only extra cost—computing synergy/discrepancy matrices—remains trivial for $k\!\le\!10$ domains.

Weaknesses:

(1) Heuristic similarity metrics. The partition relies on fixed vocab-overlap, sentence-embedding cosine, and JS divergence, which often mis-estimate real transferability; domains can look similar yet teach different skills (or vice-versa), sending data into sub-optimal clusters. A task-aware or learned affinity signal—e.g., gradient similarity—would make the partition less brittle.

(2) Scalability limits. Computing the full $k\times k$ affinity matrix and running agglomerative clustering costs $O(k^{2}\log k)$ time and $O(k^{2})$ memory, while adapter banks grow linearly with $k$. This becomes infeasible when the number of domains reaches dozens; approximate kernels or router-style sharing are needed.

(3) Modest memory edge. While cheaper than full fine-tuning, two adapter banks still use more GPU memory than leaner single-adapter PEFT variants, which matters on edge devices; pruning, quantization, or cross-stage sharing could tighten this gap.

(4) Theory untested empirically. The paper derives a neat worst-stage risk bound, but never shows how the objective $G$ (or its components) correlates with held-out error. Without a plot or numbers, readers cannot tell whether the bound is predictive or merely a sanity check. Reporting the correlation between $G$ and downstream accuracy across random partitions would clarify its practical value.