TS-MOF: Two-Stage Multi-Objective Fine-tuning for Long-Tailed Recognition

Response to Reviewer Fe6u

Thank you very much for your careful review and insightful comments, which are truly beneficial for improving the rigor of our paper.

W1. On the Novelty of the Two-Stage Framework:

Thank you for your valuable suggestions regarding the writing logic. This has led us to deeply reconsider how to best present our core contributions.

We will clarify that the core advantage of our work lies in the synergy between our proposed new components and the two-stage framework:

• R-PLA and RD-PCGrad, as the technical foundation of our method, can stably resolve gradient conflicts among multiple strategies, ensuring the optimization process rapidly moves towards a Pareto-improving direction. This allows the insights from different long-tail strategies to be efficiently integrated.
• The two-stage decoupled framework, in turn, drastically reduces the cost of this strategy integration. Compared to other multi-expert or multi-objective methods that require end-to-end training, our method only needs to fine-tune the lightweight classifier heads, significantly reducing computational overhead and training time.

In short, the new components ensure the "effectiveness of fusion," while the two-stage framework ensures the "efficiency of fusion." We will follow this logic to clarify our innovations and motivation in the revised manuscript. Thank you again for your profound insights.

W2. Adding a Framework Schematic:

Apologies, but due to rebuttal guidelines, we're unable to directly include images.​ We will add a full-process schematic diagram in the manuscript, including Stage 1 pre-training and Stage 2 multi-expert fine-tuning, as well as detailed illustrations of R-PLA weight calculation and RD-PCGrad gradient coordination. This will significantly improve the method's comprehensibility and clearly highlight our contributions.

W3. Regarding the SHIKE Performance and Comparison to Multi-Expert Models:

We apologize for the misunderstanding and would like to provide a clarification. Our paper uses a unified framework to ensure compatibility, allowing different expert strategies to be incorporated into the fine-tuning process. In our experiments, we used the backbone from LOS and retained the original hyperparameters for other strategies. The specific hyperparameters for the backbone and our MOF components were kept consistent across different combinations to diligently validate the method's generality and its performance improvement over baselines. Under this unified framework, the baseline performance of different methods may vary. We will emphasize this point in the subsequent revision. Additionally, we have done our best to supplement experiments to clearly demonstrate the performance gains over the baseline.

Comparison to Multi-Expert Models: Our motivation differs from that of multi-expert methods. Multi-expert models (such as SHIKE and SADE) can work well independently and are often meticulously designed by researchers. In contrast, our work is a study on a gradient-based multi-objective optimization method that is strongly relevant to long-tail learning. Its goal is to accelerate the utilization of these expert strategies and speed up the fine-tuning process. Therefore, it can potentially absorb benefits from different representation learning methods and expert strategies that favor balance. When combined with other strategies, it can be viewed as a new type of long-tailed multi-expert method. We will add comparisons and discussions with classic multi-expert models (such as SHIKE and SADE) in the main text.

W4 & W5. Supplementing Implementation Details and Ablation Studies:

Thank you for your suggestions. We will clarify these points in the main text to enhance comprehensiveness. The strategies fused by MOF in our main experiments are KPS, BCL, and LOS (as hinted in Figure 2), and we will explicitly state this in the revised text. Furthermore, we have diligently supplemented necessary experiments:

Table: Ablation experiments of TS-MOF three modules with IR=10/50/100 on CIFAR100-LT Benchmarks.

Component Variant Analysis: We have conducted experiments comparing our components with alternative methods to demonstrate their superiority.

Table: Comparison of computational cost and model performance between TS-MOF and conventional multi-task optimization in the second stage.

Strategy Combination Benchmark: To showcase the general applicability of our framework, we have benchmarked the performance of various strategy combinations.

Table: Benchmark performance across different strategy combinations with IR=10/50/100 on CIFAR100-LT Benchmarks.The subscript is the improvement of the relative linear weighted combination.

Of course. Here is a response to Reviewer uDJM, addressing questions W1 and Q1 using the provided information.

---

Response to Reviewer uDJM

Thank you for your positive feedback and for highlighting the core strengths of our work. We are grateful for your insightful questions and provide our responses below.

W1: On Hyperparameter Sensitivity Analysis

Thank you for this valuable suggestion. We agree that analyzing the sensitivity of the hyperparameters in R-PLA and RD-PCGrad is important for understanding their robustness. We have conducted a sensitivity analysis for these key parameters and present the results below.

The data indicates that while the modules' performance shows some dependence on hyperparameters, our method remains robust across a reasonable range of values, consistently outperforming baselines. This suggests that meticulous tuning is not required to achieve strong results, although optimal performance can be found with experimental combinations.

Table: Hyperparameter analysis for R-PLA and RD-PCGrad modules on the CIFAR100-LT benchmark (IR=100).

Q1: On the Role of $\epsilon_{\text{sim}}$

Your intuition is correct; $\epsilon_{\text{sim}}$ has a subtle secondary role beyond ensuring numerical stability. While its primary function is to prevent division by zero when the norms of the performance vectors are near zero, it also acts as a stabilizer in cases of extremely low similarity.

When the performance vectors are nearly orthogonal, their dot product is close to zero. Without $\epsilon_{\text{sim}}$, the denominator would also be close to zero, making the similarity value highly sensitive to small numerical fluctuations. By ensuring the denominator has a small positive lower bound, $\epsilon_{\text{sim}}$ dampens this sensitivity and prevents drastic swings in the similarity value (and thus the weight allocation) due to minor noise. In essence, it helps to regularize the similarity computation, ensuring that the weight allocation remains stable and meaningful even when strategies are performing very differently.

Response to Reviewer BSGz

Thank you for your detailed and critical feedback. We have done our best to supplement new experiments to substantially improve the paper.

W1 & Q2. On the Dependency on a Balanced Validation Set:

Thank you for your recognition of our experimental section. We would like to clarify that our method does not strictly depend on a balanced validation set. For the extreme case you are concerned about, such as when a validation set is unavailable, we can also sample from the training set. To explicitly verify this, we have added a new experiment on CIFAR100-LT where we construct an imbalanced validation set by sampling from the training set. The results (shown below) indicate that although there is a slight drop in performance compared to using the original (more balanced) validation set, TS-MOF still significantly outperforms the SOTA baseline (LOS), demonstrating its applicability and robustness.

Table 1: Comparison of model performance on CIFAR100-LT benchmarks using an imbalanced validation set (from Train Set) and the original validation set.

W2. On the Supplementation of Ablation Studies:

Thank you for your suggestion to supplement the experimental section. We have conducted separate ablation experiments for R-PLA, RD-PCGrad, and the Expert Aggregation Module (EOSS). The following are the results.

Table 2: Ablation experiments of TS-MOF's three modules with IR=10/50/100 on CIFAR100-LT benchmarks.

W3. Comparison with Multi-Expert SOTAs:

Thank you for your valuable suggestion. We have done our best to supplement several multi-expert baseline methods within the limited time, such as BalPoE, SADE, and MDCS. All comparisons will be cited and added to the main text. As seen from the table below, our TS-MOF method still demonstrates high performance under the same settings.

Table 3: Comparison for CIFAR100-LT Benchmarks, showcasing the model performance of three multi-expert methods: BalPoE, SADE, and MDCS.

W4. On the Task Selection Mechanism:

Thank you for your question. The appendix aims to demonstrate that our framework can achieve performance improvements through rapid multi-objective fine-tuning across different strategy combinations. Our goal is not to perform a search for a single SOTA performance on the test set, but rather to showcase the general advantage of performance enhancement brought by the two-stage learning + multi-objective fine-tuning based on R-PLA and RD-PCGrad.

To demonstrate this general advantage, we avoided hyperparameter tuning as much as possible. All hyperparameters for the individual strategies themselves were kept identical to their settings in the original papers and public code. The hyperparameters specific to our method (such as those for the Stage 1 representation learning, R-PLA, and RD-PCGrad) were kept consistent across different strategy groups. We have also supplemented additional experiments on these hyperparameters, which you can refer to in Table 5.

W5 & Q1. Deterministic Task Ordering in RD-PCGrad:

In our implementation, the task order is determined by the sequence in which the strategies are passed. This deterministic approach ensures better reproducibility of our experimental results. We have also supplemented a performance comparison between RD-PCGrad and other potential multi-objective methods to illustrate its advantages; please see W6 for details due to space constraints.

W6. On Computational Overhead:

Compared to end-to-end multi-expert learning or multi-objective optimization methods, we only fine-tune for the last 20 epochs and only update the classifier head parameters. This reduces computational costs in real-world applications. Additionally, as per your suggestion, we have compared our method with other potential multi-objective approaches, with the results shown below. Macs represents the computational cost (in millions of operations), Time is the training time (in minutes), $\gamma_1$ is the ratio of total trained parameters, and $\gamma_2$ is the ratio of training time. The results indicate that the computational complexity and time of TS-MOF are roughly equivalent to ordinary methods, while delivering more efficient and accurate performance.

Table 4: Comparison of computational cost and model performance between TS-MOF and conventional multi-task optimization in the second stage.

Q3. Explanation of Eq. 4:

Thank you for this sharp question. We clarify the R-PLA computation process here: for the same training batch, we first compute the performance vectors $\{\mathbf{a}_1, ..., \mathbf{a}_N\}$ for all N tasks through a single forward pass. After all vectors are computed, we then designate the last vector $\mathbf{a}_N$ as the reference and compute the cosine similarity of the other vectors $\mathbf{a}_j$ with it to determine the weights. Therefore, all performance information is available when calculating the weights, and there is no temporal conflict.

Q4. On Classifier Head Architecture:

In the main text, our model and classifier head architectures are consistent with prior works (e.g., LOS). For broader scenarios, such as when users wish to extend to more strategy combinations for rapid fine-tuning, the choice of model architecture and hyperparameters can be guided by our response in W4. We will add more detail on this in the revised manuscript.

Q5. On Introduced Hyperparameters:

The main discussion on hyperparameters is referenced in W4. Additionally, we have done our best to supplement discussions and experiments on other hyperparameters. We hope this answers your questions.

Table 5: Hyperparameter analysis for the R-PLA and RD-PCGrad modules in the TS-MOF framework with IR=100 on CIFAR100-LT benchmarks.

Response to Reviewer LH8y

Thank you, Reviewer LH8y, for your recognition of our work and for your constructive feedback. We address your questions point by point below.

1. On the Deviation of RD-PCGrad from the True Pareto Front:

Thank you for this insightful question. Our method ensures a Pareto-improving direction for stability, which involves a deliberate trade-off. We formally analyze this below.

Formal Analysis of the Trade-off

Given two conflicting gradients $g_A$ and $g_B$ (i.e., $g_A^T g_B < 0$), RD-PCGrad modifies $g_A$ by removing its conflicting component $p_{A \to B} = \text{proj}_{g_B}(g_A)$.

a. Suppressed Benefit & Cost: The suppressed component $p_{A \to B}$ is indeed beneficial for minimizing its own loss $L_A$. The magnitude of this benefit is proportional to: $(g_A^T g_B)^2 / \|g_B\|^2 \ge 0$ This is the cost of our projection. However, this component is equally harmful to task $L_B$. By removing it, we prioritize the "Do No Harm" principle, which is essential for preventing the "seesaw effect" in LTR and ensuring stable, balanced progress.

b. Deviation Analysis: The deviation from task A's optimal direction is proportional to the relative magnitude of the suppressed component, which can be expressed as: $\rho = \|p_{A \to B}\| / \|g_A\| = |\cos(\theta_{AB})|$ This shows the deviation is directly tied to the conflict angle $\theta_{AB}$ between the gradients. In LTR, where conflicts between head- and tail-focused strategies are inherent, this deviation is a necessary compromise. Our goal is a balanced solution, not the extreme optimization of any single strategy. Thus, this shift towards balance is a feature, not a bug, for solving the long-tail problem.

2. On the Rationale for Using a Cube Root in Equation (5):

Thank you for this insightful question. We acknowledge that, in theory, any monotonically increasing concave function can ensure robustness. We chose the cube root based on the specific advantages it offers in terms of weight dynamics and gradient smoothness, which collectively enhance the model's empirical stability.

First, compared to other concave functions like the square root, the cube root provides a better trade-off in sensitivity. It is more sensitive in the low-similarity region (when strategy performance differs greatly), providing a stronger "incentive" signal for underperforming strategies to catch up. Concurrently, it is less sensitive in the high-similarity region (when strategies perform consistently), providing greater stability and preventing unnecessary adjustments to already well-performing strategies.

Second, the cube root leads to smoother gradient flow. Since similarity is a function of model parameters, the weighting function introduces an additional meta-optimization gradient during backpropagation. The derivative properties of the cube root make this gradient flow smoother than that of other concave functions (like the square root), especially in the extreme case where similarity approaches zero. This effectively reduces the risk of exploding gradients.

We will also add an ablation study in the appendix comparing different weighting functions to empirically support our choice.

---

3. On the Motivation for the Two-Stage Framework:

Thank you for this insightful question. We chose the two-stage framework deliberately because it offers significant advantages for our MOO components when tackling the long-tail problem:

1. Ultimate Efficiency: Compared to end-to-end training, our method only applies multi-objective optimization to the lightweight classifier heads during a short final fine-tuning stage, drastically reducing computational cost and time.
2. Feature Protection: By freezing the backbone, we protect the valuable general-purpose features learned in the first stage from being corrupted by strong re-balancing signals, thus preserving performance on head and medium classes.
3. Training Stability: Confining the complex gradient conflict resolution to the low-parameter classifier heads greatly reduces the risk of training instability, making the optimization process more robust and controllable.
4. Plug-and-Play Modularity: Our framework allows researchers to easily "plug and play" any existing long-tail strategy without complex end-to-end adaptation, significantly improving the method's ease of use and extensibility.

Therefore, the combination of the two-stage framework and our proposed strategies enables us to achieve an efficient, stable, and low-cost fusion of multiple long-tail strategies.