Long-tailed Recognition with Model Rebalancing

The baselines are relatively weak. The paper lacks comprehensive comparisons against recent state-of-the-art methods in long-tailed single-label classification. Including stronger baselines would help solidify the claimed improvements.

Thank you for this valuable suggestion. Regarding mixture-of-experts (MoE) methods [1,2], we note that our approach shares conceptual similarities in addressing tail-class underfitting through capacity adjustment. To address your concern, we have included additional experiments comparing MORE against RIDE [1] (using 4 experts) and BalPoE [2] on CIFAR-100-LT, with accuracy (%) results shown below. The results demonstrate that integrating MORE with already strong baseline like ProCo yields consistent improvements. In addition to accuracy, we also report the number of inference-time parameters (third column) to highlight that, unlike MoE-based baselines which expand model capacity via expert branches, MORE requires no additional inference parameters, achieving improvements without increasing model size.

[1] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts

[2] Balanced Product of Calibrated Experts for Long-Tailed Recognition

The connection between the theoretical analysis in Section 3.3 and the methodology in Section 3.4 is unclear. This relationship should be clarified in the rebuttal. It would also be helpful if Section 3.4 included a brief remark explicitly referencing how the theory informs or motivates the method.

Thanks for highlighting this point. To address your concern, we elaborate as follows:

• Theoretical motivation: the theoretical foundation in Section 3.3 motivates and justifies the core components of our MORE method. In particular, Theorem 1 demonstrates that by decomposing the model parameters based on class roles, we partition the hypothesis space. This decomposition reduces the overall Rademacher complexity of the model, which in turn tightens the generalization bound on the balanced risk, particularly benefiting tail classes in long-tailed distributions.

• Methodological realization: Section 3.4 operationalizes the above theoretical insights into a concrete implementation. Specifically, (1) The low-rank parameter decomposition reflects the model-space partition discussed in Section 3.3. (2) The discrepancy-based loss guides the optimization such that the low-rank components learn tail-specific representations, while the shared components capture general patterns. (3) The sinusoidal reweighting schedule further balances the trade-off between generalization and specialization, aligning with the theoretical goal of capacity reallocation over training time.

To make this connection clearer, we will add a brief remark at the beginning of Section 3.4 explicitly referencing Section 3.3 as followes: Motivated by the theoretical analysis in Section 3.3 on decomposing model parameters, we now detail the practical instantiation of parameter rebalancing through low-rank adaptation and a tailored discrepancy-based loss, which adaptively emphasizes tail-specific learning to enhance performance in long-tailed distributions.

For the multi-class image recognition experiments, the authors primarily evaluate on CIFAR100-LT and Places-LT. However, ImageNet-LT and iNaturalist are widely used large-scale long-tailed benchmarks. Including results on these datasets would better validate the generalizability and effectiveness of the proposed method.

Thanks for your thoughtful recommendation. To address this, We conducted supplementary experiments on ImageNet-LT, using a ResNet-50 backbone. The accuracy (%) results are as follows:

These results show that MORE provides consistent and stable improvements across all partitions, with particularly notable gains on Tail classes while maintaining performance on Many and Medium classes. This aligns with our findings on CIFAR-100-LT, Places-LT and other large-scale datasets in multi-label settings, confirming the method's effectiveness in scaling to larger, more diverse datasets. Due to computational and time limits, we are currently conducting experiments on the iNaturalist dataset and will present the results as soon as they are completed.

The paper should be better to clarify which modules of ResNet and CLIP are decomposed using LoRA, respectively. Additionally, it would be helpful to specify the chosen rank $r$ for LoRA and compare the number of learnable parameters with and without LoRA during training.

Thanks for this valuable suggestion. To clarify the decomposition: in our implementation, we apply low-rank decomposition to all convolutional layers of the ResNet backbones, with other layers remaining unchanged. For CLIP-based models, we freeze the text encoder and fine-tune only the vision encoder and decompose all convolutional layers within it.

Regarding the rank $r$, we commonly use $r = 0.1$. For parameter comparisons during training: on a ResNet-34 backbone, the learnable parameters are approximately 25.4M with MORE and 21.3M without. Similarly, for ResNet-50, the counts are about 29.4M with MORE and 25.6M without. This modest training overhead stems from the low-rank parameters and is relatively contained. We will incorporate these details into the revised paper in Section 4.1 or appendix.

It would be valuable to report the values of $M(x;\theta)$ across samples from different classes in the final trained model. Furthermore, presenting the difference $f_y(x;\theta^g \oplus \theta^t) - f_y(x;\theta^g)$ across class samples would help support the effectiveness of the proposed loss L_{\mathrm{MORE}, as increasing the ground-truth logit is key to improving model performance.

Thanks for this insightful suggestion. We conducted an additional analysis of $|f_y(x;\theta^g \oplus \theta^t) - f_y(x;\theta^g)|$ across samples from head, mid, and tail classes in the final trained models on various datasets. The averaged values are as follows:

These results suggest that $|f_y(x;\theta^g \oplus \theta^t) - f_y(x;\theta^g)|$ is smaller for head classes and larger for tail classes across all settings. This indicates that the tail-specific parameters exhibit a stronger response to tail-class samples and effectively amplify their representations. Note that the order across groups is more important than the absolute values, as it reflects their differential effect on the final logits. This pattern supports the effectiveness of our method, as it aligns with the goal of increasing ground-truth logits for underrepresented classes. We will include these analyses and visualizations in the revised paper to further substantiate the method's mechanisms.

There are also some typos, such as in line 40, where "Rachemacher complexity" should be "Rademacher complexity".

Thanks for pointing this out. We have carefully proofread the entire manuscript and corrected the typos. These changes will be included in the revised version.

While the paper correctly claims no increase in inference cost, the impact on training cost is not addressed. The formulation, particularly as implied by Equation (2), suggests that the training process requires two forward passes per iteration. The lack of empirical data (GPU hours) on this training overhead is a significant omission. This information is crucial for assessing the practical feasibility of the method, especially for training on large-scale datasets.

Thanks for your comment. To address your concern, we conducted additional experiments on the CIFAR-100-LT with the imbalance ratio of 10 on a single NVIDIA 3090 GPU over 200 epochs. The results are shown in the table below, which indicates that the overhead is relatively tolerable. We will include these details in the final paper for enhanced transparency.

I have some reservations about the sensitivity of the method to its key hyperparameters (the rank r and the balancing coefficient α). A critical trade-off exists for the rank r: if it is too small, the representational capacity of the tail-specific parameters $\theta^t$ may be insufficient; conversely, if it is too large, the benefits of the low-rank assumption are diminished. It is unclear how these hyperparameters should be set for new datasets or different backbone architectures without extensive tuning. Furthermore, the assumption that a single shared low-rank space $\theta^t$ can effectively serve all tail classes might be a limitation. If certain tail classes are semantically distinct and require features that are largely "orthogonal" to those of other tail classes, this shared space may fail to adequately capture their unique characteristics.

Thanks for this insightful feedback. Regarding hyperparameter sensitivity, our empirical evaluations show that our method is relatively robust, with similar settings generalizing well across diverse datasets and backbones, without the need for extensive per-task tuning. Furthermore, leveraging prior characteristics of the dataset-such as class imbalance ratios or feature distributions-to guide the selection of hyperparameters like the rank $r$ remains a valuable direction for future exploration.

Your point about the potential limitation merits deeper investigation. Indeed, without our parameter decomposition, semantically distinct features in tail classes would compete for representation space with other head and tail classes. By introducing the decomposition parameters, our method already mitigates competition from head classes, allowing $\theta^t$ to focus more effectively on tail characteristics. To futher address this concern, one approach is to moderately increase the rank of tail-specific parameters for greater capacity; alternatively, we can partition tail classes into multiple groups based on semantic similarity (e.g., via clustering on pre-trained embeddings) and assign group-specific low-rank parameters, thereby alleviating potential orthogonality issues among tail features.

We appreciate these suggestions and will include further discussion in the revised paper to highlight potential avenues for enhancement.

The paper could further explore the relationship between the general ($\theta^g$) and tail-specific ($\theta^t$) parameters. While the loss function applies them to different objectives, they are still optimized jointly. Have the authors considered enforcing a stricter decoupling, for instance, through an orthogonality constraint between ($\theta^g$) and ($\theta^t$)? This could prevent representational overlap and lead to a more modular model.

Thank you for this constructive suggestion. While certain orthogonality constraints can pose optimization difficulties—such as the ones discussed in [1], where some forms of orthogonal regularization negatively impact the convergence rate—we still believe that appropriate orthogonalization strategies could further minimize representational overlap and potentially improve performance on tail classes. This remains a valuable direction and is worth further exploration. We will include a discussion in the conclusion section of the revised paper to reflect the merit of this suggestion.

[1] On Orthogonality and Learning Recurrent Networks with Long Term Dependencies

Lack of conceptual differentiation from LoRA. Although the authors claim their low-rank decomposition is “fundamentally different” from LoRA, the distinction is not clearly articulated in terms of optimization behavior or empirical ablation. Figure 3 alone is insufficient to convincingly demonstrate that LoRA is ineffective in the long-tail setting. Without a sharper contrast, the contribution risks being viewed as an application of existing adaptation techniques to long-tailed learning. Can the authors clarify how their method is conceptually and empirically different from existing low-rank adaptation techniques? What evidence shows that this difference is critical in the long-tailed setting?

Thank you for your comment. To address your concerns, we elaborate on the key differences below:

• Purpose and Design Motivation: LoRA targets efficient fine-tuning by adding low-rank updates to frozen weights, aiming to reduce computational cost. Our method, in contrast, is designed specifically to address class imbalance by explicitly decomposing parameters into general and tail-specific components, without relying on a frozen backbone.

• Optimization Behavior: LoRA applies uniform optimization across all classes using standard losses (e.g., cross-entropy), which may favor head classes. In contrast, our method incorporates a discrepancy-based loss that prioritizes tail classes, guiding tail-specific parameters to learn tail-class representations and enabling imbalance-aware learning. This positions our contribution as a novel extension of low-rank ideas to imbalance mitigation, rather than a mere application of existing techniques.

To further support these conceptual differences, we conducted additional ablations on the VOC dataset with BCE, comparing variants grounded in low-rank structures: (1) BCE ($\theta = \theta^g + \theta^t$), which adopts our decomposition strategy but retains the standard BCE loss; and (2) BCE (LoRA), which applies LoRA for incremental fine-tuning of the pre-trained model with BCE. Results show MORE method surpasses these baselines with superior performance. This is because low-rank parameters like LoRA merely change update dynamics within a fixed-size network, without increase model capacity. This indicates that the performance gains of MORE stem not from the form of low-rank decomposition, but rather from its overall design, which enables the rebalancing of model capacity to effectively counter class imbalances.

Evaluation breadth. Although the experiments cover a diverse range of tasks, they are limited to relatively small- or medium-scale datasets. Common benchmarks in long-tail learning—such as ImageNet-LT and iNaturalist—are notably absent. This limits the ability to draw conclusions about the method’s scalability and robustness in more realistic or challenging scenarios. Can the authors evaluate the method on larger and more standard long-tailed benchmarks to better demonstrate its scalability and generalization?

Thanks for your thoughtful recommendation. To address this, We conducted supplementary experiments on ImageNet-LT, using a ResNet-50 backbone. The accuracy (%) results are as follows:

These results show that MORE provides consistent and stable improvements across all partitions, with particularly notable gains on Tail classes while maintaining performance on Many and Medium classes. This aligns with our findings on CIFAR-100-LT, Places-LT and other large-scale datasets in multi-label settings, confirming the method's effectiveness in scaling to larger, more diverse datasets. Due to computational resource and time limits, we are currently conducting experiments on the iNaturalist dataset and will present the results as soon as they are completed.

The paper does not explore strong baselines that manipulate model capacity, such as mixture-of-experts methods [1, 2] or self-supervised learning approaches [3], which offer alternative solutions to tail-class underfitting. Moreover, using extra parameters is a common and effective strategy—parameter-efficient finetuning approaches like [4] have also shown strong performance in such scenarios. Including comparisons or ablations against these baselines would help clarify whether the performance gains come from the specific design of MORE or from the general benefit of added parameter capacity. Can the authors include comparisons or ablations against alternative capacity-based baselines to isolate the source of performance improvements?

Thank you for this valuable suggestion. Regarding mixture-of-experts (MoE) methods [1,2], we note that our approach shares conceptual similarities in addressing tail-class underfitting through capacity adjustment. To address your concern, we have included additional experiments comparing MORE against RIDE [1] (using 4 experts) and BalPoE [2] on CIFAR-100-LT, with accuracy (%) results shown below. The results demonstrate that integrating MORE with a strong baseline like ProCo yields consistent improvements. In addition to accuracy, we also report the number of inference-time parameters (third column) to highlight that, unlike MoE-based baselines which expand model capacity via expert branches, MORE requires no additional inference parameters, achieving improvements without increasing model size.

Our method does not inherently involve self-supervised learning. Similarly, parameter-efficient finetuning like [4] primarily deals with freezing pretrained parameters for adaptation in single label settings, which differs from our core contribution. To further enhance comprehensiveness, we will include additional discussions on methods [3] and [4] in the revised version of the paper.

[1] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts

[2] Balanced Product of Calibrated Experts for Long-Tailed Recognition

[3] Generalized Parametric Contrastive Learning

[4] Parameter-Efficient Long-Tailed Recognition

Modest overall gains in some settings. On certain benchmarks (e.g., NUS-WIDE, CIFAR100-LT and Places-LT), the overall gains are relatively small, raising questions about trade-offs and consistency across datasets. How consistent are the improvements across different datasets, and can the authors explain the relatively modest gains observed in some settings?

Thanks for your feedback. To address your concerns, we provide the following elaboration:

• Consistent Improvements Across Settings: We would like to emphasize that our method actually shows reliable enhancements across diverse datasets and settings, including both single-label and multi-label long-tailed scenarios. Previous methods almost cannot achieve such generalizability and consistency.

• Focus on Underrepresented-Class Performance: Our gains are particularly evident in tail classes, where our mechanism helps effectively mitigate underfitting due to data imbalance. For instance, on CIFAR-100-LT, MORE improves tail-class accuracy by 1.5% (IF=10) and 1.6% (IF=100). Similarly, it boosts tail-class performance by 2.5% on Places-LT and 1.9% on NUS-WIDE-SCENE, underscoring its targeted benefits for underrepresented classes.

• Performance Saturation in Certain Settings: Regarding the relatively modest overall gains in some settings, we are inclined to attribute to the fact that, with constrained data availability in tail classes and specific model architectures, the trained models often approach a natural performance upper bound. As this ceiling is neared, the learned features become increasingly saturated, making incremental improvements more challenging to achieve. This phenomenon is common in the settings where head-class dominance already pushes baseline accuracies relatively high, leaving limited room for global gains despite substantial tail improvements. We will expand on these insights in the discussion section of the revised paper to provide further clarity.

Dear Reviewer SoG6,

We sincerely thank you for your thoughtful and constructive feedback, which has significantly helped us improve our manuscript. Below we summarize our key responses:

• Differentiation from LoRA: We distinguished our method from LoRA by clarifying their fundamental differences: our approach tackles class imbalance via parameter decomposition and discrepancy-based loss, contrasting with LoRA's uniform fine-tuning for efficiency. Additional empirical ablations further demonstrated these distinctions and our method's effectiveness in long-tailed scenarios.

• Evaluation Breadth: We conducted additional experiments on ImageNet-LT to demonstrate our method's scalability and generalization on large-scale long-tail benchmarks. Our method consistently improved performance across all class partitions when integrated with strong baselines like LA and ProCo, confirming its broad applicability.

• Comparisons Against Capacity-Based Baselines: We compared MORE with Mixture-of-Experts (MoE) methods like RIDE and BalPoE. Our method delivered consistent improvements without increasing inference-time parameters. This demonstrates that MORE's performance improvements stem from its specific design, rather than expanding model capacity as MoE approaches do.

• On Modest Overall Gains: We clarified that while tail-class improvements are substantial, overall gains may appear modest because, with constrained data and specific architectures, models approach a natural performance upper bound. This limits substantial global improvements especially when head-class performance is already high.

We hope that our detailed responses and the additional experiments address your concerns. Please let us know if anything is unclear. We truly value your time and would be most grateful for any feedback you could give to us.

Best Regards,

The authors of Submission 10376

Dear Reviewer SoG6,

Thank you for taking the time to review our responses and provide your valuable feedback. We are glad that our clarifications have addressed your concerns. Your thoughtful comments have been essential in improving the clarity and rigor of our work. We will ensure that these improvements are incorporated into the revision. We sincerely appreciate your support and the constructive suggestions you have provided.

Best regards,

The authors of Submission 10376