LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences

1. The paper proposes a GNN-based model to predict the carbon footprint of LLM inferences. The integration of graph embedding, data sampling, and the Roofline model does not introduce fundamentally new concepts but rather repurposes established techniques in a novel application context. This approach lacks substantial innovation. The primary contribution is an improvement in prediction accuracy over previous models. However, these improvements are incremental, and the research lacks a clear breakthrough in model efficiency or in introducing a new predictive paradigm.

2. While the model aims to predict the carbon footprint, it does not adequately address the broader trade-offs in energy efficiency versus accuracy. The model's usage of multiple GPUs and its impact on real-world energy savings are not thoroughly discussed.

W1: I seriously doubt the credibility of the papers cited in this line of research, like Faiz et al., which is an important building block in the argumentation of this paper (see first and second paragraph). They cite Strubell et al. 2019, which was famously (or at least I thought) debunked by Patterson et al. (https://arxiv.org/pdf/2104.10350) as providing an 88x higher estimate of the energy consumption compared to what was really used, highlighting how hard it is to perform this research in practice. Patterson et al. was also not cited, which, from my understanding, is one of the few papers of people with actual end-to-end access to all production metrics, making this more trustworthy than most other papers that estimate costs. I take specific issue with general statements "(...) with a single epoch requiring three times the FLOPs of an inference", "(...) the period required for inference emissions to match training emissions is rapidly decreasing." and "For instance, training the Google T5 LLM generates 40% more carbon emissions than a round-trip flight between San Francisco and New York (Faiz et al., 2024)." While the first two statements would have to have a large caveat of the specific use-case attached to it and potentially not being representative at the current time, the last statement is, to my understanding, entirely wrong. Patterson et al. states that T5 training took 46.7 tCO2e (Table 4), which is 26% of a roundtrip between NYC and SF (180 tCO2e). The authors argue that this took 140%, an error by a factor of 140/26 ~= 5x.

W2: I question the addition of empirical data from older hardware towards the training dataset of LLMCO2. Inference for frontier models like OpenAI's (which was used in the motivation) likely does not happen on older GPU architectures due to their missing support of sparsity and reduced FLOP and memory bandwidth per $ performance, making that analysis void. When arguing that inference is costly at scale, one needs to analyze the actual costs of said scale and not use older hardware for that comparison.

W3: The authors cite the Azure LLM trace as a basis for their decision to use a <2 batch size and argue that this is representative. The same paper (Patel et al. '24) also says this only counts for the prefill (Sec 3.D) but not for decoding, which scales for that particular model-hardware setup until a batch size of 64 (only limited by memory in that specific instance, making the results even more questionable as this is dependent on the context lengths of the requests due to them affecting the KV-cache). This is not addressed by LLMCO2, making its prediction very likely not applicable to other providers and use cases. Additionally, Patel et al. argue that the largest latency (hence, time and energy consumption at ~50% energy consumption) comes from the decoding phase, so they split these two phases to run on different hardware setups. However, the authors do not consider this use case (at least to my understanding) when generating the dataset to train LLMCO2, discounting their own motivation of "Disregard for prevalent configurations", while Patel et al. was used to argue this exact motivation.

W4: The idle time of the nodes was not considered, which is also likely to be a large chunk of energy spent if services are not utilized to capacity. Arguably, this can be omitted by assuming that on-demand up-/downscaling of hardware works efficiently. However, this was never stated. Generally, the assumptions of when LLMCO2 will likely provide accurate results and when it does not are not stated concretely enough.

W5: While I appreciate the empirical-based approach of this paper, it is lacking on multiple fronts. No SOTA deployment methods are discussed (while citing Patel et al., which proposes one instance when splitting prefill and decoding phase on different hardware generations), and continuous batching was barely addressed in how it affects energy consumption. No sparsity, no speculative decoding or streaming, no TensorRT, no NVIDIA Triton, no vLLM, no torch.compile, and a very limited definition of workloads. Just as an example, if speculative decoding improves the decoding phase by 2x, its carbon emissions will be approximately halved (due to the smaller model being multiple magnitudes faster and cheaper to run, making it a negligible cost). This is a very common technique and is likely to be used by most inference providers, which is not addressed at all.

W6: Peak FLOP is wrong for the H100 in Table 2 (see the H100 datasheet https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet). FP32 is stated as 989 TFLOP, while it is 67 TFLOP (presumably, the authors mistyped TF32, which would also be wrong as this is the performance with sparsity). FP16 performance is stated as 1979 TFLOP, but this is 989 TFLOP, as this is the performance with sparsity enabled. Same thing for INT8. Same issue for all precisions for the A100. This makes me question the other results from this paper if such an important consideration was overlooked.

W7: The second paragraph states, "(...) the period required for inference emissions to match training emissions is rapidly decreasing." While I agree that this can be the case, the way this is argued is not conclusive. We see a trend that compute utilization is doubling every 6 months (https://epochai.org/trends#compute). To argue the point of inference costs becoming closer to the training costs, the same analysis would need to happen for inference (presumably with AI usage numbers, which are likely to be kept private at large). However, due to the issues from W5, the trace used in this paper would not be representative of the real world, making it probably impossible to argue outside of specific deployments.

W8: Eq. 1, "energy_per_operation x PUE x carb_intensity." I doubt that this is the correct way to estimate energy consumption. This excludes the energy for the nodes itself, including networking, PC internals, and cooling. A DGX H100 node uses roughly 10.2kW, while 8xH100 use 8x700W=5.6kW, making your resulting estimate off by a factor of 2x (https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html)

W9: What were the exact dataset splits? The dataset and evaluation sections seem to suggest that all evaluated models were part of the training dataset, making me afraid of an overfit happening.

W10 (Summary and final notes): While the basic premise of the paper is interesting to use a computational-graph analysis and using a GNN to process it, the application of this model would lead to the following problems:

• A misguided understanding of how energy costs come about from LLM inference due to very limited real-world application scenarios
• Potentially misleading results about being energy-efficient in theory or inference being much more grave w.r.t. energy-consumption than training. It is important to note that I am not averse to thinking this might be the case, but how it is stated here is definitely wrong. I fear this work being misunderstood and misused, similar to Strubell et al. (and how prior cited work like Faiz et al. was used similarly by the authors).

Minor issues:

• Figures 9, 10, and 11 are squished and are hardly readable.
• Figure 10 was referenced when the authors wanted to reference Figure 11 in " Figure 10 shows the operational carbon footprint of Bloom-7b1 (...)".
• Figure 11: It is hard to understand what training is and what is inference due to the legend's labels.

• This is good work. However, this paper is more suitable for an HPC-related conference; I didn't see much relation to the submitted track of `alignment, fairness, safety, privacy, and societal considerations'.
• While the paper presents several innovations, it looks like combining existing methods such as GNNs, the Roofline model, and active learning. It may lack of novelty.