CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization

We thank the reviewer for the detailed review of our work and thoughtful suggestions. Below, we answer the questions and comments in order.

Question 1, Weakness 1: Tuning lifetime and regional parameters

We fully agree that factors such as grid carbon intensity, device lifetime, and inference frequency significantly impact the total carbon footprint. To support diverse deployment scenarios, CATransformer exposes all of these parameters as configurable inputs. In addition, hardware design parameters, such as process technology, are also fully configurable.

These factors primarily influence the balance between operational and embodied carbon. For example, high inference frequency, long device lifetime, or deployment in regions with high grid carbon intensity increases operational energy consumption, making operational carbon the dominant contributor. Conversely, in scenarios with low inference frequency, short device lifetime, or low grid carbon intensity, embodied carbon plays a more significant role.

We include a sensitivity analysis on inference frequency. The table below reports the percentage of operational carbon as a fraction of total carbon footprint across CarbonCLIP model variants. Increasing the inference rate from 1 to 10 inferences per second leads to an average of 50% increase in the operational carbon share.

Appendix K complements this analysis by demonstrating how external factors like grid carbon intensity influence whether CATransformer prioritizes reducing operational energy or minimizing hardware footprint. Importantly, the trends observed with varying grid intensity generalize to changes in inference frequency and device lifetime, as all induce similar shifts in the operational versus embodied carbon tradeoff. This confirms that CATransformer adapts its optimization strategy consistently across deployment scenarios—whether in high-throughput mobile environments or in systems with shorter lifespans.

We will clarify these insights in the final version of the paper to better highlight CATransformer’s flexibility and its nuanced handling of carbon-aware optimization.

Question 2: Carbon zero regions and varying grid intensities

We thank the reviewer for important questions regarding carbon assumptions. On a zero-carbon grid, operational carbon becomes negligible, shifting optimization entirely to minimizing embodied carbon, potentially favoring smaller hardware. In such cases, the pruned submodel size and architecture are likely determined solely by latency constraints, since operational energy (i.e., the energy required to run the model) is no longer a factor. The goal then becomes to run the largest possible model (to maximize accuracy) on the smallest hardware that still meets the latency requirement. While CATransformer currently uses a single grid intensity, it can be readily extended to ingest a distribution of intensities, enabling robust designs for varying regional or fluctuating grid conditions. These clarifications will be added to sections 3.3 and 4.5.

Question 3, Weakness 2: Details of estimators and open source tools used

We agree with the reviewer that providing exact parameters and configurations for each toolchain is essential for reproducibility. We have submitted the full codebase as supplementary material, including all configuration files, tool versions, and experimental parameters. These details will also be made publicly available upon open-sourcing the code. Below is a summary of the key tools and configurations used:

• CACTI: Version 7, using 22nm process technology node configurations from commit 1ffd8d
• Accelergy: Version 0.4
• ACT: Used an IC yield factor of 0.875 and accounts for the carbon footprint of manufacturing gases per unit area, assuming 95% abatement.
• Carbon Intensity: Based on 2024 data from Electricity Maps. Exact values are specified in the configuration files:
  • California: 224 CO2eq/kWh
  • Taiwan: 524 CO2eq/kWh We will ensure that all configuration files and tool references are clearly documented and accessible in the final open-source release to support artifact evaluation and reproducibility.

Question 4: Cost of optimization process

Runtime per candidate varies with model size and available compute. In our experiments, optimization was conducted on a single node with 8 V100-SXM2-16GB GPUs. Under this setup, the full optimization takes ~5 hours for BERTbase and up to 20 hours for CLIP, each with 100 trials. This translates to 3–12 minutes per trial, including ~2 minutes for hardware estimations. Fine-tuning adds another 1–10 minutes per candidate, depending on model size. With newer or more powerful hardware, per-trial costs would decrease proportionally. To reduce overhead, we implement result caching, avoiding redundant retraining of previously evaluated candidates across hardware settings and amortizing cost over search iterations.

Question 5: Extension to other sustainability axes

While CATransformer currently focuses on carbon footprint, we agree that sustainability extends to factors like electronic waste, water usage, and rare mineral consumption. Our optimizations, which often reduce chip area, can implicitly benefit these metrics. In principle, CATransformer could support multi-objective optimization over broader environmental impacts. The main challenge lies in the current lack of standardized, fine-grained data for these additional sustainability metrics. We will highlight these considerations in the conclusion to underscore the broader societal relevance and need for holistic, sustainable hardware design.

Weakness 3: Extensibility to training and datacenter accelerators

This work focuses on making model inference carbon-efficient through hardware-model co-design. Carbon footprint of inference in aggregate is significant due to much wider usages [1]. We also agree with the reviewer that large-scale pretraining of foundation models can result in substantial carbon emissions. As our GPU-based results from Appendices L and M show, our proposed design methodology can scale to datacenter-level accelerators. CATransformer inherently supports training by modeling forward and backward pass latency and energy at the operator level. It can be extended to incorporate parallelization strategies and training system design, enabling optimization of carbon efficiency in distributed training.

[1] Luccioni, S., Jernite, Y. and Strubell, E., 2024, June. Power hungry processing: Watts driving the cost of ai deployment?. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency (pp. 85-99).

Weakness 4: Improvements on formatting for Tables and Abbreviations.

In the final version of the paper, we will enhance the presentation of tables and ensure that all abbreviations and equations are clearly defined.

Weakness 5: Related works in Sustainability in AI

We thank the reviewer for highlighting these relevant works on AI sustainability. Broadly, they address the topic through three main approaches: (1) Quantifying the carbon footprint of training: [1] and [5] estimate the emissions and energy use of training deep learning models or evaluate tools for doing so; (2) Analyzing Deployment Energy Use and emissions: [2] and [3] focus on energy consumption during inference and advocate for standardized environmental assessment frameworks; and (3) Exploring Energy-Performance Trade-offs: [4] explores the balance between model accuracy and electricity use, proposing metrics to encourage more energy-efficient architectures. We will incorporate a discussion of these works in Section 2 of the final version.

Limitations

We agree that broader societal impacts, such as privacy concerns in edge deployment and potential resource inequities from hardware specialization, are important considerations. We will include a discussion of these challenges, alongside the limitations of fixed carbon-intensity and lifetime assumptions, to provide a more balanced perspective on the societal implications of our work.

We thank the reviewer for their thoughtful feedback and suggestions. Below, we answer the questions in order:

Weakness 1: Extensibility to Training and datacenter accelerators

This work focuses on making model inference carbon-efficient through hardware-model co-design. Carbon footprint of inference in aggregate is significant due to much wider usages [1]. We also agree with the reviewer that large-scale pretraining of foundation models can result in substantial carbon emissions. As our GPU-based results from Appendices L and M show, our proposed design methodology can scale to datacenter-level accelerators. CATransformer inherently supports training by modeling forward and backward pass latency and energy at the operator level. It can be extended to incorporate parallelization strategies and training system design, enabling optimization of carbon efficiency in distributed training.

[1] Luccioni, S., Jernite, Y. and Strubell, E., 2024, June. Power hungry processing: Watts driving the cost of ai deployment?. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency (pp. 85-99).

Weakness 2, Question1: Generalizability to Non-Transformer architectures

CATransformer uses Torch.fx to extract operator graphs and estimate latency and energy, making it extensible to any model representable by Torch.fx, including CNNs like ResNet. However, Torch.fx cannot trace dynamic control flow, such as if statements or data-dependent loops, limiting support for models like state-space architectures (e.g., Mamba) that involve input-dependent state transitions. In such cases, the model must be modified or constrained to make the control flow more static for Torch.fx to function properly.

Extending to other model types also requires latency and energy estimations for their unique operators, such as custom kernels or non-standard layers not found in Transformers. These must first be profiled or estimated for runtime and energy consumption on the target hardware to enable optimization within our framework. If such toolchains become available, CATransformer can integrate them with minimal changes. Additionally, different models may require customized pruning methods. Attention head pruning, for instance, does not apply to CNNs and Mamba-style architectures.

We will clarify these limitations and generalization challenges in the paper.

Weakness 3, Question 2: Choosing the optimal configuration

We agree that clearer guidance on interpreting the Pareto frontiers would enhance the practical relevance of our work. Choosing an optimal configuration requires balancing carbon emissions and latency based on deployment constraints. For instance, latency-critical applications (e.g., medical imaging, autonomous vehicles) may prioritize low-latency points, while non-critical tasks (e.g., captioning, search) may favor carbon savings. Interactive applications like augmented reality may benefit from a middle ground. Additionally, the trade-off between operational and embodied carbon depends on context. For example, regions with high grid intensity may prefer configurations that reduce operational energy. We will clarify these considerations in the final version.

Question 3: Challenges in plugging-in CATransformer into PyTorch

While components of CATransformer, such as the optimized Transformer models, can be used within PyTorch due to its open-source foundation and use of HuggingFace models, fully integrating the entire framework into PyTorch is challenging. CATransformer relies on external tools such as CACTI, Accelergy, Phaze, and ACT for detailed hardware and carbon analysis. These capabilities are not supported natively by PyTorch.

We thank the reviewer for the follow-up question. Currently, CATransformer uses two user-configurable parameters to guide the optimization process. The first is a latency constraint, which is especially important for latency-sensitive tasks. This ensures that the optimization only outputs configurations that satisfy the specified latency target, while minimizing carbon emissions within that constraint. The second set of parameters includes accuracy and carbon thresholds. These allow users to specify target thresholds, and the optimization then searches for configurations that meet or come close to those targets, minimizing carbon while maximizing accuracy.

These are the only knobs the user needs to adjust, making it easy to adapt the system to different workloads or use cases. This approach balances user control and abstraction, allowing users to express their preferences in terms of latency, carbon, and accuracy without needing to manage low-level detail

We thank the reviewer for their thoughtful feedback and suggestions. Below, we answer the questions in order:

Weakness 1: Details of the methods and key components

We will open-source the codebase upon acceptance and include detailed tutorials and examples to support reproducibility. In the final version of the paper, we will also expand the method section to provide additional implementation details.

Weakness 2, Question 1: Extensibility to training and datacenter accelerators

This work focuses on making model inference carbon-efficient through hardware-model co-design. Carbon footprint of inference in aggregate is significant due to much wider usages [1]. We also agree with the reviewer that large-scale pretraining of foundation models can result in substantial carbon emissions. As our GPU-based results from Appendices L and M show, our proposed design methodology can scale to datacenter-level accelerators. CATransformer inherently supports training by modeling forward and backward pass latency and energy at the operator level. It can be extended to incorporate parallelization strategies and training system design, enabling optimization of carbon efficiency in distributed training.

[1] Luccioni, S., Jernite, Y. and Strubell, E., 2024, June. Power hungry processing: Watts driving the cost of ai deployment?. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency (pp. 85-99).

Weakness 3, Question 2: Carbon Footprint of the optimization process

We quantify CATransformer’s carbon footprint using CodeCarbon [1] . On average, the optimization process takes 5 hours for BERTbase and up to 20 hours for CLIP. In the most resource-intensive case (CLIP), 100 optimization trials emit approximately 57 kgCO2e, while final model training emits 454 kgCO2e per model. This means the optimization process costs roughly 1/13th the carbon budget of training the final model. The CarbonCLIP models are trained for 2 epochs on the MetaCLIP dataset—roughly 40% of the training steps used in prior work (MetaCLIP, TinyCLIP, CLIP). Despite the one-time cost of optimization, CATransformer achieves overall efficiency gains through reduced training steps post-pruning, along with inference gains that scale with the number of devices.

[1] Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, MarionCoutarel, Boris Feld, Jérémy Lecourt, LiamConnell, Amine Saboni, Inimaz, supatomic, Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Hugues de Lavoreille, Niko Laskaris, Edoardo Abati, Douglas Blank, Ziyao Wang, Armin Catovic, Marc Alencon, Michał St˛echły, Christian Bauer, Lucas Otávio N.de Araújo, JPW, and MinervaBooks. mlco2/codecarbon: v2.4.1, May 2024.

Extended Limitation Discussion

We agree that sustainability trade-offs from custom accelerators, such as increased electronic waste and maintenance complexity due to hardware heterogeneity, are important concerns. In the final version, we will discuss potential mitigations, including reusing existing hardware and co-optimizing models with commercially available accelerators.

We also acknowledge the limitations of proxy-based accuracy estimation. While such proxies enable scalable design space exploration, they can deviate from real-world outcomes due to overfitting to short fine-tuning runs or underestimating pruning’s impact on downstream performance. These approximations may miss interactions between model structure, data, and training dynamics. In future work, we aim to develop more reliable alternatives that improve accuracy estimation without sacrificing scalability.

We thank the reviewer for their thoughtful feedback and for recognizing the significance and quality of our work, noting that the paper “provides a basis for important decision-making…for sustainability-focused end-users” and “highlights the complex trade-offs that were not previously considered in earlier works.”

Question 1: Optimizing pretrained models

CATransformer takes a pretrained model as input and identifies pruned submodels and hardware architecture configurations that together minimize the total carbon footprint. We show that post-pruning training on the selected submodel can achieve accuracy comparable to the original model while significantly reducing carbon emissions (section 4.5 Table 3). Alternatively, given a fixed pretrained model, CATransformer can perform a hardware architecture search to identify the configuration that minimizes carbon footprint under specified latency constraints (Section 4.3 Table 2).