On Feature Diversity in Energy-based Models

Q: $\beta$ is exceptionally minute …. Is it feasible to employ Monte Carlo estimation to approximate the second term, thereby preventing it from becoming excessively large and dominating the original loss?

Note that the proposed regularizer contains two sums: the first sum is over the whole batch and the second sum is over all pairs of units within the layers. This yields a total of $ND^2$ terms, where N is the batch size and D is the number of units within the layer. This results in empirically large values of the second term ($~10^9$ ). Thus, $\beta$ needs to be small so that the loss is not dominated by the second term. Empirically, we found that $[10^{-13},..,10^{-11}]$ corresponds to a stable range. Note that the proposed regularizer is merely a proof-of-concept to show that encouraging feature diversity can improve the performance of EBMs. If we normalized with $ND^2$ , the range of values of $\beta$ would be higher without any change in the results. However, to stay coherent with our theoretical diversity, we did not include any normalization terms.

The Reviewer makes an interesting suggestion of using a Monte Carlo estimation to approximate the second term. This can be a promising future research direction. In fact, as shown in our work, a simple diversity strategy can improve results. This is why we believe feature diversity-based research can lead to developing more advanced diversity-inducing strategies that can boost the performance of EBMs even further.

Q: Could you apply the augmentation loss to generative modelling with high-dimensional datasets, such as CIFAR-10? Achieving success with this straightforward regularization approach in high-dimensional datasets would be both theoretically and practically promising.

Generative tasks with large datasets with EBMs are computationally expensive and require access to high computational power. This is why for the generative task, we limited the results for MNIST. Note that for the continual learning task, we use a large dataset CIFAR10 and, as shown, the performance gain is consistent. Moreover, now we include results for large-scale image regression in Appendix C.1.

We hope that our response has addressed all your questions. We also hope that in light of the revision, the Reviewer increases his score.

Q: My observation is that the theoretical upper bound is heavily contingent on the specific task. It appears to be a refined calculation of previous work in Lemma 1

It is true that Lemma 1 presents the starting point of this work. However, that is not a limitation of the work. In fact, Lemma 1 presents the classic upper-bound of generalization using Rademacher complexity. As our proof technique here depends on the Rademacher complexity, it is convenient to use Lemma 1 as a basis for the developed theory. Note that this is typical for theoretical works based on Rademacher complexity (For example [1-3])

[1] Yin, D., Kannan, R. and Bartlett, P., 2019, May. Rademacher complexity for adversarially robust generalization. In International conference on machine learning (pp. 7085-7094). PMLR.
[2] Mohri, M. and Rostamizadeh, A., 2008. Rademacher complexity bounds for non-iid processes. Advances in Neural Information Processing Systems, 21.
[3] Cortes, C., Kloft, M. and Mohri, M., 2013. Learning kernels using local rademacher complexity. Advances in neural information processing systems, 26.

Q: The theoretical contribution is based on the upper bound presented in Lemma 1. However, this result seems to originate from an unpeer-reviewed work.

We respectfully disagree with the reviewer on this point. While Lemma 1 originates in a non-peer-reviewed work, note that the result presented there, i.e., upperbouding the generalization with Rademacher complexity, is a classic result in learning theory. See Theorem 3.3 or 3.5 or 11.3 in Mohri el al. (2018).

Q: The experiment performance is not strong. The tasks are limited to 1-d regression and continual learning, which is insufficient. The improvement exists but is relatively marginal.

Note that in addition to the continual learning and the 1-D example, in Appendix B, we provide extra results for image generation. We also now add results for a large-scale regression task in Appendix C.1.

Q: minor typos

Thanks for pointing this out. We fixed the minor typos in the revised manuscript.

Q: The diversity concept used in this paper is not usual. Does it have something to do with the homonymous concept in datasets/search [1]?

In [1], they consider ensemble diversity and how to diversify search results from sets. In this work, the scope of diversity is different. In particular, we are interested in feature diversity. The problem formulation is as follows: ‘Given an EBM model which relies on a set of features to make its prediction, how does the diversity of these features affect the performance?’. In our analysis, we find that learning diverse features indeed helps and can improve generalization. To avoid confusion, we rephrased the problem definition in the introduction.

Q: Besides examples already provided, are there some other problems can be encoded as EBM?

Energy-based models have received a lot of attention lately and have been used to tackle a wide range of tasks, e.g., density estimation (Michael et al. 2021) or anomaly detection (Zhai et al. 2016). We now include more related work in Appendix A.

Q: Is there any generic generalization upper bound for the EBM according to -diversity, without exploring the specific form of the task?

Unfortunately, theoretical analysis typically depends on the problem characteristics, e.g., loss used, data distribution, etc. Note that in our work, we show consistently that diversity helps and it is interesting to compare the effect of diversity in different contexts. For example, in Theorem 4, we see that in implicit regression, the effect of diversity is quadratic while in standard classification, it is linear.

Q: What is the correlation between diversity regularizers and other popular regularizers? Can they exist together, or do their functions overlap?

The main scope of this work is feature diversity. We want to theoretically study how diversity affects generalization. As shown in our results, diversity is a good property for EBMs to generalize better. While there is no apparent direct interplay between diversity and classic regularizers, the Reviewer makes an interesting point. Indeed, it is interesting to study the interplay between diversity and other regularizers. Note that in our experiments, we did not cancel out the other regularizers. In each experiment, we followed the standard protocol used in the original paper and we added our diversity on top of it. So the standard ‘EBM’ in Tables 1 and 2 already includes other regularizers and, as shown, adding diversity (ours) improves the results. This shows that diversity plays a different role and can complement classic regularization techniques.


We hope that our response has addressed all your questions and concerns. We kindly ask you to let us know if you have any remaining concerns, and - if we have answered your questions - to reevaluate your score.

No rebuttal has been posted by the authors yet, but I have read all other reviews.

The other reviewers also give the paper quite borderline scores, I don't think they give me any particular reason to increase my score.

First, we apologize for the late reply. We were running additional experiments, as requested by several reviewers.

Q: Could you add at least one more experiment? Since you use the 1D regression datasets from (Gustafsson et al., 2022), could you not apply the proposed method to one of their four image-based regression datasets as well?

We now include results with an additional dataset from Gustafsson et al., (2022), namely the UTKFace Age Estimation dataset. We follow the same protocol as in Gustafsson et al., (2022). As shown in the results in Appendix C.1, using a diversity regularization helps even in the context of large-scale data for the regression task and it yields lower error rates.

This further confirms our early results that leveraging feature diversity is indeed a promising research direction and can improve EBM models even for large datasets.

Q: Could you add a separate Related work section?

Due to page limits, we opted not to have separate related work. Now, in the revised manuscript, we include a Related work section in the appendix.

Q: "The two valid energy functions which can be used for regression are and " However, this is not what is used in (Gustafsson et al, 2022; 2020b;a)

Note that there is a difference between the loss function used to train the EBM and the energy function. While (Gustafsson et al, 2022; 2020b) use different losses, our results are still valid for their approach, as our analysis is agnostic to the loss function or the optimization strategy used.

Q: Minor things

Thanks for pointing this out. We have now fixed the mentioned points in the revised manuscript.


We hope that our response has addressed all your questions. We also hope that in light of the revision and the additional experiments added, the Reviewer reevaluates his score.

Q: The effect of $\beta$ on the performance improvement is not well characterized. Only 3 values are ever considered.

Here, the goal of the empirical section is to show that diversity-inducing regularization is a promising research direction and that the proposed regularizer works in different applications. Note that for all three values of the hyperparameters, the model achieves superior performance.

Q: The performance improvement is admittedly quite small.

The experimental results and the regularizer are simply a proof-of-concept to show how our theory can be used in practice. We hope it leads to future research in this area in order to develop more advanced feature diversity-inducing regularizers. As shown in experimental results, even our simple regularizer boosts the performance of EBM across multiple tasks. Regarding the statistical significance of the results in Table 1, note that for these two particular small datasets, the losses are small in general which yields in tight gap. (Same is observed in Gustafsson et al., (2022)). However, note that the experiments are repeated over 20 random seeds, and the average result is reported, which makes them statistical-significant.

Q: It should be possible to visualize the learned features... Why is this not done?

The Reviewer here refers to using t-sne or similar approaches to visualize the features. First, note that the main scope and aim of this work is the theoretical analysis of feature diversity and that is why more attention and in-depth analysis was given to the theoretical Section 2. As the experiments here are a proof-of-concept, we wanted to show that indeed diversity works in different contexts and thus we opted to include results from different tasks and scenarios, i.e., regression, continual learning, and image generation rather than an in-depth analysis of one task. Moreover, in general, interpreting visualizations of features in high dimensional space can be both impractical and unreliable, as simply changing the initialization can dramatically change the final feature embedding. From this perspective, we think visualizing the features would be hard to interpret with respect to diversity and would not provide any value. As for visual results, note that in Appendix C.1, we show the effect of diversity on the speed of convergence in image generation tasks.

Q: In the CIFAR-10/100 experiments, did the authors consider applying the regularization term to representations other than that outputted by the last of the intermediate layers?

When we defined the regularizer for the empirical experiments, we tried to stay as close as possible to Definition 1 and the derived theorems, where the feature set is considered to be the last intermediate layer (after it, there is only one linear transformation w). Although It is beyond the scope of this paper, the Reviewer suggests a good future work idea and it is our belief that applying such regularization strategies on other layers might be beneficial to the EBMs.

Q: The regularization term in Eq (15) has a computational complexity that will grow quadratically with the number of features...

That is a valid point. The regularization term grows quadratically with the number of features. However, note that in most state-of-the-art models, the last hidden layer typically has a limited size of ~1024. Note that the term is simply the sum of pairwise distances which can be computed efficiently with most deep learning libraries. We found that, for example, for the CIFAR10 experiment our regularizer has a small additional time cost in training time (less than $1\%$). We now include these details in the revised manuscript.

It should be noted here that the proposed regularizer is simply a proof-of-concept to show how our theory can be used in practice. We are not claiming that our regularizer is the best way to enforce diversity. However, as shown in the empirical results, even a simple feature diversity strategy can improve the performance of EBMs. This is why we hope that our findings inspire more future research to inspire more advanced feature diversity-inducing strategies.


We hope that our response has addressed all your questions and concerns. We kindly ask you to let us know if you have any remaining concerns, and - if we have answered your questions - to reevaluate your score.

Q: The paper may lack a detailed explanation of the specific regularization techniques developed to address the limitations of empirical energy minimization, which could limit the reproducibility of the results.

In this paper, we show theoretically that reducing the redundancy of the feature set can improve the generalization error of energy-based approaches. Afterwards, inspired by our theoretical findings, we develop a proof-of-concept regularizer described in Section 3, which leverages this idea and shows that indeed such approaches can improve results. To improve reproducibility, we now describe in more detail the proposed approach in the revised manuscript. Furthermore, we submit the codes for reproducing our results in the supplementary material.

Q: The paper may not provide a clear discussion of the limitations or potential challenges associated with the proposed approach

In the revised manuscript, we now include a discussion of the limitations of our approach at the end of the conclusion.


We thank the reviewer for his feedback and we hope he raises his score. Note that we added now an additional experiment with a large regression dataset