Regularizing Energy among Training Samples for Out-of-Distribution Generalization

Dear reviewer aGfa,

Thank you for your valuable feedback. We address each of your concerns and questions in the following.

The connection between Sec.3 and Sec.4 appears somewhat disjointed. The motivation for the proposed method in Sec.4 seems detached from Sec.3, lacking continuity and a strong foundation.

In Sec.3, we theoretically show that energy reguarization can be decomposed into reweighting and margin control, two main stream of methods in long-tail recognition. We further empirically show that long-tail recognition methods apply implicit energy regularization. Based on the analysis in sec.3, in Sec.4, we propose a method to explicitly regularize the energy and extend the scenario from long-tail recognition to subpopulation shift and domain generalization.

The explanation of the re-weighting and margin control methods as implicitly affecting training energy feels somewhat forced. For instance, the paper primarily uses the first term in Eq. (6) to justify the impact of $\hat{\beta}_x$ . However, the second term in Eq. (6), which also contains $\hat{\beta}_x$, influences the final loss. Thus, the effectiveness of energy regularization remains somewhat ambiguous.

For Eq.6, the two terms both have coefficient $1-\hat{\beta}\_x$, which corresponds to data reweighting. The first term further have $\bar{p}(y|\mathbf{x}) -1$ changed to $\bar{p}(y|\mathbf{x}) -\frac{1}{1-\hat{\beta}\_{\mathbf{x}}}$ which corresponds to margin control, changing the margin defined as $f_θ (x)[y] − max_{j\neq y} f_θ (x)[j]$.

To make Eq.6 more clear, we derive it as $\frac{\partial \mathcal{L}(\mathbf{x}, y, \theta)}{\partial \theta} = (1-\hat{\beta}\_{\mathbf{x}})\cdot \left( \left[\bar{p}(y|\mathbf{x}) -\frac{1}{1-\hat{\beta}\_{\mathbf{x}}} \right] \frac{\partial f\_{\theta}(\mathbf{x})[y]}{\partial \theta} + \sum\_{y'\neq y} \bar{p}(y'|\mathbf{x})\cdot \frac{\partial f\_{\theta}(\mathbf{x})[y']}{\partial \theta} \right).$

where $\mathcal{L}(\mathbf{x}, y, \theta)=\mathcal{L}\_{ce}(\mathbf{x}, y, \theta) + \hat{\beta}\_{\mathbf{x}}\cdot E\_{\theta}(\mathbf{x})$

The method's overall effectiveness is limited. For example, in Tab.3, when IAER is combined with cRT, the performance across all classes declines, and with LWS, the improvement brought by IAER across all classes is only about 0.2%

For iNaturalist and ImageNet-LT, the few-shot classes only have less than 20 images (the least class in iNaturallist only have 2 images), which makes it infeasible to sample a balanced validation set. Therefore, we approximate influence function on training data of Many-shot, Medium-shot and Few-shot classes emphasizing on part of the classes. In Tab. 3, we show that the energy regularization do affect the generalization ability.

There are some typos, such as in Tab.3, where "iNatrualist" should be "iNaturalist".

Thank you for the feedback, we have modified it in our paper.

We hope the response have addressed your concerns. We are looking forward to your reply.

Dear Reviewer 16Hz, thank you for your time and effort in providing us the valubale feedback. In the following We address each of your concerns.

The main concern lies with the experimental settings. Although the proposed IAER method shows some improvement over the standard baseline (ERM), it does not achieve competitive results compared to state-of-the-art methods. Specifically, Table 2 only includes comparisons with two works from 2019. Why are more recent works not included for comparison? This same question applies to Tables 3, 4, and 5. I would except more recent work to be included and disucssed. Just name a few: 1) Gradient Reweighting: Towards Imbalanced Class-Incremental Learning 2) Dynamic residual classifier for class incremental learning, 3) Towards principled disentanglement for domain generalization. 4) Flatness-aware minimization for domain generalization

Thanks for the advise, we will include more recent works in our paper. For subpopulation shift, we use subpopbench (2023) to conduct experiments. For domain generalization, we follow the widely used domainbed to conduct our experiments, which is also compatible with many recent works such as the mentioned DDG method.

As we are still preparing the additional experiments, we will update experimental results promptly.

Lines 270–282 lack clarity. For instance, the statement “The least we can do is to prevent the model from being over-confident… note that the energy does not correspond to the prediction of the classifier (Remark 4.1), which is the reason most previous works overlook the energy disparity” requires further clarification: 1) yes, we usually do not have knowledge of test distribution. But how to prevent it from being over-confident? 2) I am not clear why Remark 4.1 explains why most previous works overlook the energy disparity between training data samples. 3) Additionally, the main point of Remark 4.1 is unclear. Could you clarify its purpose and significance?

Thanks for the good question, the following is our answer to the questions:

The main point of Remark 4.1 is that models with identical classification performance (e.g. measured by cross entropy loss) could have different energy value on training data samples. It may also affect the generalization ability as shown in this paper.

While we have witnessed many previous works focusing on regularizing the risk (classification performance) across different domains, the energy on training data samples has been overlooked.

In preventing the model from being over-confident, we introduce energy regularization during training, which controls the energy (probability) assigned to different data samples.

Once again, thank you for your valuable comments and support. We are more than happy to respond to any further questions.

Thank you for your response. I have carefully reviewed the rebuttal as well as the comments from other reviewers. After consideration, I maintain my original score. As mentioned in my previous comments, this work is an early exploration of leveraging the energy of in-distribution data as a regularization for training models. I continue to encourage the authors to incorporate more recent works to further demonstrate the effectiveness of this regularization approach.

Dear reviewer f4ew,

thank you for your time devoted to the reviewing process and providing us the valuable and detailed feedback. In the following, we address each of your comments.

Inconsistent symbol usage and the definition of symbols should be clear and unambiguous.

We have modified our paper. The modified part is in red. To better clarify the symbols, we use $y$ for the ground truth label and $y'$ for all the training classes. In line 150, we use $y'$ to represent all the labels. In line 210 we use $y$ to represent ground truth. Thank you for the feedback, we will keep polishing our paper.

The baselines used in the study appear to be outdated. For instance, the baseline in Table 2 dates back to 2019, and the baseline in Table 3 is from 2020. Given the rapid advancements in the field, using more recent benchmarks would provide a more accurate and relevant comparison, thereby enhancing the validity and reliability of the results.

Generally, the energy regularization is orthogonal to most previous methods (because models of the same risk/performance can have arbitrary energy value on each data sample as indicated in Remark 4.1). For subpopulation shift, we use subpopbench (2023) to conduct experiments. For domain generalization, we follow the widely used domainbed to conduct our experiments.

We will include comparison with more recent work in our paper. In the following, we provide some additional experimental results on PACS combining our energy regularization with more recent methods. We are still preparing the additional experiments and will update experimental results promptly.

Inconsistency from Eq.5 to Eq.6.

Because Eq.6 is too long for one line, we use $\mathcal{L}(\mathbf{x}, y, \theta)$ for the $\mathcal{L}_{ce}(\mathbf{x}, y, \theta) + \hat{\beta}_{\mathbf{x}}\cdot E_{\theta}(\mathbf{x})$. We have further clarified this in our paper. We sincerely apprreciate your attentive feedback.

Is the influence function calculated every epoch?

No, the influence function is calculated once for one training. Therefore, with limited the number of approximation iteration, the computation overhead is acceptable.

In Tab. 3, the selection of the validation set is crucial for model performance, especially in long-tailed scenarios where the test distribution can significantly differ from the training distribution. Could the authors provide a more detailed explanation of their validation set selection strategy, including the rationale behind their choices for different datasets?

Yes, the selection of the validation set is crucial. For long-tail recognition like CIFAR datasets, we sample a balanced validation set from the training set. It is under the motivation to make the model more balanced over classes, same as some previous works reweighting data points. However, for extremely long-tailed dataset like iNaturalist, the few-shot classes have less than 20 images, which makes sampling a balanced validation set infeasible. Therefore, we take the training data of each Many-shot, Medium-shot and Few-shot classes to calculate influence and as shown in Tab.3, the choice of validation set do affect model performance.

For domain generalization, because we have no access to the test domain data, we sample from the training domains to construct validation set.

In line 317, the terms "positive influence" and "negative influence" are used. Could the authors provide clear definitions of these terms in the context of their method? Additionally, could they explain how these influences relate to the energy values of data points, particularly for low-energy points?

In our method, we use influence (approximated influence function) to determine the coefficient for energy regularization. Positive influence generally indicates the energy regularization would encourage a higher energy (lower probability) while negative influence indicates the energy regularization would ecourage a lower energy (higher probability).

We hope the response have addressed your concerns. We are looking forward to your reply. Thank you for your valuable feedback.

Thanks for the response. My concerns have been addressed. I decide to raise my score to 6.

Dear Reviewer GFVu,

we scincerely appreciate the time and effort you devote to the reviewing process. We address each point of your concerns in the comments below.

It seems that the proposed method does not perform consistently better than baseline methods and relatively unstable.

In this paper, we use influence function to empirically bridge subpopulation shift methods with implicit energy regularization. The main limitation of the proposed method also comes from the use of influence function, where we use traninig set as a proxy validation set for fair comparison and we limit the time consumption for influence function approximation.

Generally, for subpopulation shift tasks (long-tail recognition as a special case), our method is significantly better than baseline. For domain generalization, the proposed method also shows improvement. It is limited by the limitation of the influence function. However, since our method is orthogonal to existing algorithms (focusing on training energy regularization), we leave it for future works to further overcome the limitation of the influence function.

We are preparing additional experiments and will update the experimental results promptly.

The designed influence function requires a large amount of time and computational recouces for approximating $\beta$

Yes, approximating influence function is computational intensive. We provide a detailed time complexity analysis in Appendix B.3. Since the influence function only need to be approximated once, the time consumption is acceptable as we limit the approximation iteration.

In Fig 3, it can be observed that class 7 is a outlier. What cause this class significantly deviate the trend?

The following is our conjecture. The class 7 in Fig. 3 corresponds to horses in CIFAR10. There are many other classes with similar background and similar object shape such as class 3 (cat), class 4 (deer), and class 5 (dog). With a very low influence function value, it indicates the model need to pay more attention to this class and increase the implicit probability of horses shown in the figure, which may explain why the class 7 is an outlier.

Some tables use testing error, while some use accuracy. It would be more readable to make them consistent.

Sorry for the inconvenience. We have modified our paper and use testing accuracy in every table. The disparity is because in different scenarios we follow different papers to conduct experiments such as long-tail classification, covariate shift and domain generalization.

We hope the rebuttal can address your concerns and we are more than happy to answer further questions. We are looking forward to your reply.

Dear Reviewers,

We have conducted an additional experiment combining energy regularization with one of the recent domain generalization methods Risk Distribution Matching (RDM) [1]. We report the results on PACS with model selection using the training-domain validation set.

While the time is limited, we are conducting more experiments and looking forward to your reply.

Best regards,

Authors

[1] Nguyen, Toan, et al. "Domain Generalisation via Risk Distribution Matching." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.