Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification

Dear Area Chairs,

We believe that this review was intended for a different paper because the review contains discussion of "Informative Data Selection (IDS)" and "Generative Data Augmentation (GDA)", which are not related to our work.

Could you please re-evaluate the review and re-link it to the correct paper? Additionally, could you please check if there is another review intended for our paper?

Thank you.

Thank you for your thoughtful review of work. We are glad that you found our work to have strong potential. Additionally, we appreciate your comments and questions as it gives us the opportunity to improve the paper. We responded to some of your concerns in the general response and others inline below. We hope that these answers help clarify our work.

---

The paper uses Focal Loss and VS Loss as examples of successful applications of the LCT framework. Why were only these two loss functions chosen? Are there more generalizable insights on which types of losses are or are not applicable?

We chose to use VS loss because it is a generalization of many methods proposed for imbalanced classification. Additionally, Focal loss is an alternative loss that is effective for addressing imbalance and a popular choice in practice. While simple, these two loss functions encompass many loss functions used in the literature and provide a strong foundation for demonstrating the potential of training over a distribution of loss-function-hyperparameters, instead of a single value. We expect that our method would also improve other loss functions that have hyperparameters which control the tradeoff between accuracy on head and tail classes.

---

Have the authors considered alternative methods for conditioning the model on lambda (lines 248-253)?

We have not considered alternative methods for conditioning the model on lambda, but agree that this is an interesting suggestion. We used FiLM layers because these are used in the original LCT formulation.

I thank the authors for their effort preparing the rebuttal. I have read the authors' responses as well as comments from other reviewers. The authors' response partially address my concerns (i.e., Discussion on choosing Lambda). But the discussion on more general insight about the method is still unsatisfactory (when the proposed method will and will not work).

Overall i will keep my original rating.

Thank you for your thoughtful review of work. We are glad that you found our work to be novel. Additionally, we appreciate your questions as it gives us the opportunity to clarify our work and improve the paper. We responded to some of your concerns in the general response and others inline below. We hope that these clarifications address your concerns.

---

How is SAM trained with LCT? Since $\lambda$ is associated with each data sample, but SAM is an optimizer?

We can train a model with both LCT and SAM because they affect different parts of the training procedure. As you correctly stated, when training with LCT, a $\lambda$ value is associated with each data sample. This value affects (a) the model’s inputs and (b) the calculation of the batch’s loss. After calculating the batch’s loss in the forward pass, we use an optimization method, like Stochastic Gradient Descent (SGD), to determine how to update the model’s weights to minimize that loss. Sharpness Aware Minimization (SAM) is simply an alternative to SGD and can be easily combined with LCT. Thank you for bringing this up, we have clarified this in Section 3.4 of the revised paper as well. How is $P_\Lambda$ calculated? Is it a small network or it is just a pre-defined distribution? It is a pre-defined distribution. We choose a distribution that includes reasonable values for each of the hyperparameters. We discuss the distributions that we use for $P_\Lambda$ in Section 4.3 and Table 2.

---

Given there is an underlying distribution over these hyper-parameters, could a VAE approach work as well?

We agree that there is potential for methods besides LCT to be used to train one model over a distribution of hyperparameters. If you could elaborate on this suggestion, we’d be interested in discussing more!

Thank you for your thoughtful review of work. We are glad that you agree this is an important area of work and think that our method is well-motivated. We responded to some of your questions and concerns in the general response and others inline below. We hope that after reading these responses you will have a better understanding of our work.

---

Only binary classification is studied in the paper. It is not clear if the proposed method will work on multi-class classification or unsupervised learning.

We chose to focus on binary problems for two reasons. First, despite the fact that many motivational examples in class imbalance are on binary problems, much of the state-of-the-art work in class imbalance has focused on longtailed multi-class problems. Thus, we wanted to study how these SOTA methods do or do not translate to binary problems. Second, studying binary classification problems allows us to use more interpretable evaluation metrics, like Precision-Recall curves, instead of aggregated metrics, like balanced accuracy. This helps us understand the mechanisms and limitations behind current work and describe the advantages of our proposed method. We fully expect that this method will work on multi-class classification and it should be easy to extend our method to this case. As for unsupervised learning, LCT has already been used in unsupervised problems, like Variational Autoencoders, and has shown that it can approximate several fixed-rate models for these applications, but at the cost of some performance degradation. Our contribution is showing that training a single model over a distribution of hyperparameters can improve the performance of models trained on datasets with class-imbalance, which inherently have labels.

---

The datasets used in the experiments are very small. Experiments on more large-scale datasets are required (e.g. ImageNet).

Respectfully, we disagree with this statement. One of the datasets we use (SIIM-ISIC Melanoma) has more than 45,000 training samples (please see table 12 in our submission for the number of samples each of our datasets has). Additionally, we described our rationale for focusing on binary datasets, instead of multi-class datasets like ImageNet, in the previous answer.

---

Hyper-parameter search space for focal loss and VS loss is very small, making the comparison somewhat unfair.

Again we respectfully disagree with this statement. We chose wide hyperparameter ranges that encompass the best-performing models in terms of AUC. Using a wider range of hyperparameters would only lead to training more models with poor performance.

---

It is not clear how pdf for sampling $\lambda$ is defined in section 4.3

We use a linear probability distribution function (pdf), which is a general form of probability distribution functions like uniform and triangular distributions. Our submission included a detailed explanation of this in Appendix D. Please let us know if you have more specific questions about this distribution or how to sample from it.

---

Standard deviation is not mentioned in the result table. How many different seeds were used for each section?

As stated in line 350 of our submission, we use 3 random seeds for each set of hyperparameters. We left standard deviation off of the table in the main body for clarity, but added those results to Appendix I.

---

Why was the model trained for 500 epochs on cifar (section 5.1)? That's excessive for CIFAR. Why do you need gradient clipping? Is the loss unstable?

We train for 500 epochs because LCT models take longer to converge since they are training over a distribution of hyperparameters, instead of a single value. We found that training the baseline models longer does not hinder their performance and do this for consistency. Additionally, we use gradient clipping because the randomness added by LCT can cause spikes in the loss, especially early in training.

Thank you for your thoughtful and detailed review of work. We are glad that you found our work to be relevant and significant. Additionally, we appreciate your comments and questions as it gives us the opportunity to improve the paper. We responded to most of your concerns in the general response and the others inline below. We hope that these clarifications address your concerns.

The results in Table 4 indicate that without FiLM, LCT performs worse than the baseline. Could the authors provide an analysis of why FiLM is critical and why its absence degrades performance? Similarly, why does the method not consistently outperform VS-SAM?

Yes. The FiLM layers are critical because without them the models do not receive $\lambda$ as an additional input. Specifically, when we train with LCT without FiLM, we use $\lambda$ to control the loss function of each batch, but never feed this information to the model. Thus, from the model’s perspective, the loss value is partially random. This has the effect of hindering training since the model is learning random noise. The interaction between LCT and SAM is more complicated. While both LCT and SAM can improve the performance of baseline models, we observe that using them together produces inconsistent results. We find that when combining SAM with LCT, many models fail to converge and, for some, training completely fails due to exploding gradients. This suggests that there may be an underlying conflict between LCT and SAM. SAM introduces variability in the training process because it uses the gradient at an adversarial point, instead of the actual point. LCT introduces additional variability by drawing different hyperparameters with each batch and conditioning the model on this. This dual source of variability might result in the model moving in nonsensical directions on the loss landscape and reduce the model’s ability to converge to effective solutions. Studying the interaction between these two methods is an important area of future work.

Requests for more explanation about the FiLM module (TQCe, KA6n).

We apologize for the lack of clarity about the Feature-wise Linear Modulation (FiLM) module used in the LCT models. We have added a section to the appendix (Section H) that explains these and includes a diagram to help the reader visualize this. Additionally, we provide implementation details in Section 5.1 under “model architecture.”

Choice of baselines (TQCe, KA6n, DgPZ).

We appreciate the reviewers’ suggestions for additional baseline methods. We would like to clarify that many of the State-of-the-Art methods are primarily designed for multi-class long-tailed recognition problems and many of the ideas proposed within them are not directly applicable to the binary classification setting considered in our work. For example, SuperDisco (CVPR 2023) introduces super-class discovery to learn the hierarchical relationship between classes in longtailed multi-class classification settings. Binary classification, by definition, does not involve such inter-class relationships, making this approach unsuitable for our problem. Similarly, Constructing balance from imbalance for long-tailed image recognition (ECCV 2022), uses Dynamic Label Space Adjustments (DLSA) to first separate head and tail classes to create a series of more balanced classification tasks for one longtailed multi-class classification task. Balanced product of calibrated experts for long-tailed recognition (BalPoE) is more transferable since it uses an ensemble of models, each trained with different logit adjustments. Each of these models is a special case of VS loss. We are actively working on comparing our models with BalPoE.

VS loss is a generalization of many methods proposed for imbalanced classification, including the logit adjustments proposed by BalPoE. Focal loss is another commonly-used and effective method for addressing imbalance. While simple, these two loss functions encompass many loss functions used in the literature and provide a strong foundation for demonstrating the potential of training over a distribution of loss-function-hyperparameters, instead of a single value.

We hope this clarifies our choice of baseline methods. We have added the suggested references to our related works section and agree that comparing these methods to ours in the multi-class case is important future work.