WATT FOR WHAT: RETHINKING DEEP LEARNING’S ENERGY-PERFORMANCE RELATIONSHIP

The proposed metric has several flaws:

• The hyperparameter ß seems useless, as its sole purpose appears to be scaling to obtain “nice” values (the authors claim the purpose of “better understanding of the values”, but given that the metric has no physical meaning and hyperparameters are chose quite arbitrary, there is no point to that, and absolute values of the metric have no meaning whatsoever. Energy scale is comparable if it is fixed for all investigated cases)
• Using the log of the energy consumption and the power of the accuracy has exactly opposite effects on the resulting metric as intended. The authors chose accuracy^a to reward high accuracies. But their own experiments demonstrate that achieving the last few percent in accuracy gain is the most costly in terms of electricity consumption. And on the other hand, scaling the electricity consumption with the log penalizes smaller, more energy efficient models much harder, than those with excessive energy consumption. For example, the GQI of ViT-H-JFT and ViT-B-scratch is roughly the same, but the energy consumption of the Former is two orders of magnitude larger, while the accuracy difference is only ~10%.
• The hyperparameters an and B are tuned solely to promote certain models (EfficientNet), because for other values, MoblieNet always achieved the highest GQI. This is highly non-scientific.
• The properties of the metric: in Sec 3.1 in the first bullet point, the authors claim the nominator to be in the range [0,inf], and directly following up in the next bullet point state that it is the power function of a number smaller then 1, directly contradicting the first statement. Moreover, the denominator cannot be 0, thereby rendering the statement on its range false as well. Furthermore the claim on the importance of the metric being non-negativeis nonsense. A metric can also be e.g. between -1 and 1 and would make sense. Since the metric has no physical meaning or units, the scale can be completely arbitrary, just like MSE for example.

The experimental evaluation:

• Only computer vision tasks based on images are investigated. In the introduction, the authors clearly discuss that current LLMs are the prime energy consumers. Hence, it would be good to include NLP tasks in the evaluation, especially with respect to the corresponding metrics, such as perplexity, BLUE, or ROGUE.Also, models whose predictive performance is not evaluated on an accuracy between 0 and 1, e.g. regression tasks with MSE/MAE
• There is no proper description of the experiments (see questions)
• While presenting the results, no attempt is made to gain insight into why specific architectures are more efficient than others.
• All experiments disregard data-parallel training (which cannot be assessed using CodeCarbon), but this is an essential part of DL training these days.

Certain assumptions (in the appendix):

• Interpolation: This is dangerous, since the number of epochs for each model can vary gravely. Even if you assume the number of epochs stated in the original papers (which are not always clearly given), this does not factor in things like early stopping. Moreover, the considered number of samples, batch size and other hyperparameters are strongly linked and can gravely affect model convergence.
• The section on hardware effects is pointless. The authors state that the substantial variance in GPU memory are mitigated by using the log of the electricity consumption. But it is well known that GPUs of newer generations (the ones with bigger memory) are also designed under energy efficiency aspects, which is likely to play a much more important role.

Presentation: Though nicely written, the graphical presentation of the paper is really poor:

• The Figures are terrible to read in print-out
• The table references and captions do not match, e.g. Table 1 is referred to as Table 5.2, Table 2 is referred to as Table 5.3, etc.
• Table layouts lack uniformity, even when their content is similar.
• In the description of Fig.2 x- and y-axis are swapped.
• The separation in Training and Test in Tables 1 and 3 is unclear
• All tables and plots match units for electricity
• Figure 1 is nonsensical, it seems like a rather half-baked comparison

• It's unclear what the paper is actually contributing. Their GQI metric seems arbitrary and lacks a clear use case, their experimental methodology is inconsistent and narrow, and their conclusions seem to be extremely broad questions rather than quantifiable statements. E.g.: "is large-scale pre-training truly justifiable?" Yes, obviously, to those footing the bill. Are the externalities in carbon footprint damaging to others? Also yes, and that raises some tough questions. This issue is not new, but the paper doesn't seem to add anything novel to the discussion.
• The experimental methodology is weak. The authors' use of downsampling and epoch limiting is not fundamentally problematic, but they ignore a lot of nuance in scale. For instance, distributed training comes with costs, data centers have different power distribution and cooling efficiencies, etc.
• The authors ignore non-renewable electricity costs (capex). This has been shown to be non-negligible and, in some cases, dominates operational costs. (E.g.: Gupta, et al. "Chasing Carbon: The Elusive Environmental Footprint of Computing", 2021)
• The authors vastly oversimplify the fundamental tradeoffs in computational cost and accuracy. They completely ignore the economics of these decisions (if someone is willing to pay for dozens of MWh to train a model, they're not doing it blindly---energy is not cheap).
• The conclusions in Appendix 2 are starkly incorrect. There is a huge literature on energy-efficient accelerators for neural nets which refutes this directly, not to mention the entire commercial sector built on top of the idea that specialized hardware does substantially affect energy consumption.

• There isn't anything fundamental about the expression the authors have proposed. It has some nice properties, but so do other members of a family of functions. It isn't even clear that a scalar is needed for comparisons. Multiobjective comparisons are possible, too, and the appropriate trade-offs depend on the application. The authors have essentially taken on the problem of considering the generally accepted value of accuracy and energy on the same scale, which is an important problem, but it is not clear that their expression solves this problem.

• I think there might be some dropped units in the GQI expression. If GQI is to be unitless, Beta needs to compensate for log10(kW-h).

• The text in Figure 2 is so small that it is impractical to read. It looks like a 3pt or 4pt font.

• Not sure where to put this. It's not really a weakness, more a comment intended to indicate what would make this work great. If the authors could find a broadly accepted relationship between the value of energy and the value of accuracy that holds for many applications, that would be important, high-impact work.

The contribution is limited.

1. The GQI only considers electricity. As the paper cited, the life cycle assessment (LCA) is a comprehensive evaluation rather than electricity to measure the environmental impact, since LCA includes hardware manufacturing, transportation, and recycling. In addition, a recent paper [1] shows that the manufacturing carbon may be even higher than electricity carbon.
2. GQI's electricity estimation is from the existing tool CodeCarbon, which also limited this paper's contribution. In addition, there existing metrics that consider the more comprehensive electricity consumption. For example, PUE [2] is a ratio that measures how much energy is directly used for computing rather than for cooling and other support systems in a data center.

[1] Gupta, Udit, et al. "ACT: Designing sustainable computer systems with an architectural carbon modeling tool." Proceedings of the 49th Annual International Symposium on Computer Architecture. 2022. [2] Avelar, Victor, et al. "PUE: a comprehensive examination of the metric." White paper 49 (2012).