Breaking Long-Tailed Learning Bottlenecks: A Controllable Paradigm with Hypernetwork-Generated Diverse Experts

W1: Thank you for raising the question. Let me try to further clarify some key concepts based on the content of the paper:

• Concept of environment: In Section 3, "environment" refers to a dataset with different class prior probability distributions. Each environment Em has its own class prior probability vector πm. Traditional empirical risk minimization (ERM) methods only train on a single training distribution, making it difficult to handle distribution differences between environments. Therefore, we propose to minimize the empirical risk in multiple training environments to learn a diverse set of experts that capture the distributional characteristics of different environments.
• Motivation for minimizing empirical risk in multiple training environments: Although the training set is usually fixed, the authors believe that by constructing multiple training environments with different class distributions, the distribution shifts during testing can be better addressed. By learning a set of experts on these environments, the model can capture the characteristics of data under different long-tailed distributions, thereby gaining stronger generalization ability and distribution adaptability. This approach can reduce the distribution difference between the training environments and the testing environment.
• How to integrate experts during testing: During testing, we introduce a preference vector α to control the trade-off between head and tail classes. Given the well-trained preference vector r, the authors calculate the preference vector r for testing and input it into the hypernetwork to generate the classifier head parameters for each expert. By adjusting α, the model's attention to head or tail classes can be controlled, achieving flexible trade-offs. You can refer to the answer to W4 for reviewer VYLm.

W2: Thank you very much for the reviewer's question. Our expression in the paper may not be clear enough, leading to some misunderstandings. Please allow me to provide a detailed explanation here.

In our method, we made a special setting for the hyperparameters α of the Dirichlet distribution. Specifically, we set α to a vector of length 3, where each component is equal to 1.2, i.e., α=[1.2, 1.2, 1.2]. This means that we are actually sampling independently from three Dirichlet distributions with the same parameters, i.e., [Dirichlet(α), Dirichlet(α), Dirichlet(α)].

This setting is quite common in the literature related to hypernetworks and multi-objective optimization. For example, in the paper [reference number] that we refer to, the authors also adopt a similar approach, setting the hyperparameters of the Dirichlet distribution to a vector with the same elements. Additionally, we explored using different hyperparameter settings and reported them in the main text. To avoid similar confusion, we will provide a clearer explanation of this point in the revised version.

Thank you again for the reviewer's careful review and valuable comments, which help us further improve the quality and readability of the paper.

W3: We have supplemented the necessary baseline comparisons in the appendix and will include them in the official version. Thank you for your suggestion.

W4: Thank you for your comment. We will correct these errors in the official version.

W5: Thank you for your attention to detail! We will add these references in the official version.

Thank you for your efforts on the rebuttal! The authors have clarified some important details about my concern. Hence, I decide to raise my rating to 7.

W1: Thank you for your correction. $R = (\theta, \phi)$ represents our preference vector, where $\phi$ is the radian representation mentioned in Equation (14). Figures 3 and 4 depict the control of different preference vectors on model performance.

W2: $R = (\theta, \phi)$ represents our preference vector, where $\phi$ is the radian representation mentioned in Equation (14). Thank you for your correction. We will add revisions in subsequent versions to improve readability.

W3: Thank you for your question. During the training process, the preference vector is obtained through training as a learnable parameter. During the testing process, the preference vector is jointly determined by the learned preference vector from the training stage and the input preference vector according to Equation (14). For the specific process, please refer to the answer to your W4.

W4: In the training stage, we optimized the joint loss function of all experts using the Stochastic Convex Ensemble (SCE) strategy, obtaining a set of well-trained expert models and a hypernetwork. After training is completed, we can dynamically integrate these expert models based on the user-specified test preference vector to adapt to different testing scenario requirements. The specific steps are as follows:

1. Calculate the preference vector for testing: $\hat{\mathbf{r}} = (\mathbf{r} \odot \mathbf{\alpha^*}) \oslash (\mathbf{r} \cdot \mathbf{\alpha^*})$ where r is the training preference vector, ⊙ represents the Hadamard product, ⊘ represents element-wise division, and · represents the dot product.
2. Input $\hat{\mathbf{r}}$ into the hypernetwork to generate the test-time classifier head parameters for each expert: $\hat{\mathbf{W}}_ i = h_\psi(\hat{\mathbf{r}}), \forall i \in \{1, \dots, T\}$
3. For a test sample x, the output of the $i_{th}$ expert can be calculated using the following formula: $\hat{\mathbf{y}}_i = \hat{\mathbf{W}}_i^\top \mathbf{x} + \hat{\mathbf{b}}_i \quad \text{or} \quad \hat{\mathbf{y}}_i = (\hat{\mathbf{W}}_i/\|\hat{\mathbf{W}}_i\|_F )^\top (\mathbf{x}/\|\mathbf{x}\|_2)$
4. The final ensemble output can be obtained by a weighted combination of all expert outputs. The weights can be uniform or set according to the performance of experts on the validation set. By adjusting the test preference vector, we can dynamically change the ensemble weights of the expert models, achieving flexible trade-offs between head and tail classes to meet different application requirements.

We appreciate the reviewer for pointing out this shortcoming. If there are any further questions or suggestions, we welcome discussion at any time. Thank you again for the valuable comments from the reviewer.

W1: You raised a very insightful question. The probability density function of the Dirichlet distribution is:

$f(x_1, \ldots, x_K; \alpha_1, \ldots, \alpha_K) = \frac{1}{B(\alpha)} \prod_{i=1}^K x_i^{\alpha_i - 1}$

where $\alpha=(\alpha_1,\ldots,\alpha_K)$ are the hyperparameters of the distribution, and $B(\alpha)$ is the normalization constant.

Intuitively, the hyperparameters $\alpha$ of the Dirichlet distribution control the characteristics of the generated weight vector $x=(x_1,\ldots,x_K)$:

When $\alpha_i>1$, the generated weight vector tends to take larger values in the $i$-th component; when $\alpha_i<1$, the generated weight vector tends to take smaller values in the $i$-th component; when $\alpha_i=1$, the $i$-th component of the generated weight vector follows a uniform distribution.

Therefore, the value of the hyperparameter $\alpha$ affects the diversity of the combination weight vectors generated by the controller network: when the values of all $\alpha_i$ are large, the generated weight vectors tend to concentrate in the central region, reducing the diversity of expert combinations; when the values of all $\alpha_i$ are small, the generated weight vectors tend to be dispersed in each corner, increasing the diversity of expert combinations; when the values of different $\alpha_i$ differ greatly, the generated weight vectors will have obvious preferences in certain components, leading to an imbalance in expert combinations.

In the long-tailed context, this diversity and imbalance will further affect the model's performance:

• Moderate diversity helps the model adapt to different test environments, but excessive diversity may lead to some extreme weight combinations, affecting the model's stability.
• Reasonable imbalance helps the model focus on certain key experts, but excessive imbalance may cause some experts to be ignored, affecting the model's generalization ability.

Therefore, choosing appropriate Dirichlet distribution hyperparameters is crucial for balancing the diversity and stability of expert combinations. We also report experimental results of other weight combinations in the paper. Thank you again for your question.

W2: We have supplemented additional experiments in the Appendix. Please refer to the Appendix.

W3: Figure 2 intuitively demonstrates the superiority of the method proposed in this paper in dealing with distribution shifts and flexibly controlling preferences. Figure 2 aims to illustrate two advantages of the proposed method: the ability to overcome distribution shifts and the flexibility of preference control.

In Figure 2, we use two three-dimensional coordinate systems. The first coordinate system represents the value space of the preference vector, with each dimension corresponding to a preference (such as preference for head classes, tail classes, or balance). The second coordinate system represents the model's performance on three distributions (forward50, uniform, and backward50) of the CIFAR100-LT dataset.

The dark plane in the figure represents the value plane corresponding to different preference vectors, while the outer surface represents the corresponding performance on the three distributions. The yellow points are the results of the SADE method. Since its preference is uncontrollable, the results of each run are random points, and they are all located below the purple plane of the method proposed in this paper, indicating that its performance is inferior to the method proposed in this paper (i.e., it is dominated in the Pareto optimal set).

This figure shows that the method proposed in this paper can cover unknown distributions without requiring additional training, and unlike previous methods, it can balance performance on different distributions by adjusting the preference vector. These two advantages will be further analyzed in the experimental section.

W4: We will revise the errors mentioned by the reviewer in the formal version. Thank you for your reminder.

W1 and Q1: Thank you for your question. The four perspectives in the public response section are intended to answer this question. Please refer to the four perspectives in the public response due to space limitations.

W2 and Q2: Thank you for your question. The explanation of this part in the original text is indeed brief. It is related to the loss functions that guide different experts. The specific answers have been mentioned in perspectives 2 and 3 of the public response. Here we summarize and will promptly add them to our paper:

• This paper selects three experts with different loss functions: forward expert, uniform expert, and reverse expert. Their different objectives enable different experts to adapt to different class distributions. During training, by sampling weight vectors from the Dirichlet distribution and combining the outputs of the three experts with weighted averaging, the Dirichlet distribution parameters are adjusted to simulate different class distributions. By optimizing the hypernetwork to minimize the expected loss, a set of expert parameters that perform well under different weights is obtained, which is equivalent to minimizing the expected loss of the ensemble model on the test distribution. This adaptive ability enhances the robustness of the model, enabling it to cope with distribution shifts in real-world scenarios.

W3: Thank you for your suggestion! In this paper, the Chebyshev polynomial we refer to for implementation uses the log-sum-exp function to replace the max function, which plays a smoothing role (i.e., Section 4.3 of the method). The idea mainly originates from STCH. In the context of long-tailed learning in this paper, its role is to dynamically adjust the importance of experts, thereby better handling the long-tailed distribution problem. We will add necessary explanations and citations to our paper.

W4: Thank you for your comment. We will make proper citations in subsequent versions to facilitate a better understanding of this paper.

W5: Your question is excellent. Although the focus of this paper is on solving more problems of real-world distribution shifts based on long-tailed methods, some proofs based on long-tailed settings are indeed still necessary. In addition to the response to W1 and W2, we will supplement the necessary explanations in our paper.

W6: To facilitate subsequent work in this direction, we will briefly introduce this concept and include it in subsequent versions. Additionally, we have supplemented some content regarding hypervolume in our answer to your Q6. Please refer to it.

W7: We will correct the errors in subsequent versions.

Q3: To evaluate the robustness of the algorithm under unknown test class distributions, we refer to LADE and SADE and construct three types of test sets: uniform distribution, positive long-tailed distribution, and negative long-tailed distribution. The positive and negative long-tailed distributions contain multiple different imbalance ratios ρ. For ImageNet-LT, CIFAR100-LT, and Places-LT, the imbalance ratio is set to ρ ∈ {2, 5, 10, 25, 50}. For iNaturalist 2018, since each class has only 3 test samples, the imbalance ratio is adjusted to ρ ∈ {2, 3}. We will supplement the explanation in the revised version.

Q4: The decision process for the preference vector during testing is as follows:

• By adjusting the user-specified preference vector, the attention of the model between head and tail classes can be controlled, achieving flexible trade-offs to adapt to different application requirements. The preference vector during testing is calculated based on the pre-trained preference vector and the user-input preference vector. (Due to space limitations, please refer to the answer to W4 for reviewer VYLm.)

Q5: In this method, the computation of hypervolume is mainly reflected in the following two aspects:

The method guides the computation and utilization of hypervolume by introducing reference vector information in both the training and testing stages.

• During training, by sampling preference vectors from the Dirichlet distribution and using a hypernetwork to generate expert model parameters, sampling and modeling of the Pareto front are achieved.
• During testing, by combining the pre-trained reference preference vector and the user-specified preference vector, the preference vector for testing is obtained, reflecting the localization and utilization of the hypervolume.

This setting of reference vectors provides flexibility, enabling the model to adapt to different task requirements.

Q6: Regarding your question, I guess the competing methods may refer to some long-tailed learning baselines. Although some existing competing methods such as LADE and SADE also adopt the idea of multi-expert models, they have considered the trade-offs in long-tailed distributions and used multi-expert structures to some extent, allowing them to cover different regions of the Pareto front to a certain degree. However, the completeness and continuity of the coverage are difficult to guarantee, and controllability cannot be achieved. In contrast, this paper generates a continuous expert space through a hypernetwork, which can more comprehensively approximate and cover the Pareto front.

Q7: To evaluate the robustness of the algorithm under unknown test class distributions, we refer to LADE and SADE and construct three types of test sets: uniform distribution, positive long-tailed distribution, and negative long-tailed distribution. The positive and negative long-tailed distributions contain multiple different imbalance ratios ρ. For ImageNet-LT, CIFAR100-LT, and Places-LT, the imbalance ratio is set to ρ ∈ {2, 5, 10, 25, 50}. For iNaturalist 2018, since each class has only 3 test samples, the imbalance ratio is adjusted to ρ ∈ {2, 3}. We will supplement the explanation in the revised version to improve the completeness and readability of the paper.

The $\mathbf{\alpha}\in\mathbb{R}_{+}^{k}$ in Eq 9 of the paper is inconsistent with the $\mathbf{\alpha}=(\alpha_1,\alpha_2,\alpha_3)$ here.