CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework

We greatly appreciate the positive comments and insightful suggestions from the reviewer. Below are our answers to the reviewer's questions.

Experiments are California only

We will demonstrate more results for different locations in our future work.

Compared to the work proposed by Zhang et al [1]

Thanks for providing this useful related work. We will add this baseline in the final version of our paper.

The overhead of switching between the models has not been considered considering carbon intensity variations.

Compared to the entire NAS overhead, the carbon/energy cost of the remaining components, such as the carbon intensity predictor and reinforcement learner, is negligible.

How would the CE-NAS perform if we have a deadline for used GPU hours?

We provided an ablation study based on the time constraint in the appendix (Section F2 and Figure 9). We will move this result to the main paper if we have more space in the final version.

How robust is CE-NAS in regions that we have high carbon intensity prediction errors?

If the carbon intensity prediction is incorrect, we might allocate GPU resources inappropriately among vanilla and one-shot NAS methods. This could impact the accuracy of the searched architectures and the carbon emissions. However, the total carbon emissions should still be less than directly using vanilla NAS, and the searched architectures should still be better than directly using one-shot NAS. This is partly because of our novel use of LAMOO to refine the search space.

Furthermore, we anticipate that our RL-based GPU allocation strategy will exhibit tolerance towards carbon intensity prediction errors. The adverse impact of these prediction errors will be perceived by the RL agent through its continuously updated input states, enabling it to adjust its decision-making accordingly. For instance, if the predictor underestimates carbon emission intensity, the agent may initially adopt aggressive decisions to schedule carbon-intensive tasks. This results in a rapid decrease in the remaining carbon budget, which is reflected in the input states. The RL agent perceives this change and switches to more conservative decisions in the subsequent process.

Reference

[1] Zhang, Xiaoyang, and Dan Wang. "A GNN-based Day Ahead Carbon Intensity Forecasting Model for Cross-Border Power Grids." Proceedings of the 14th ACM International Conference on Future Energy Systems. 2023.

We sincerely appreciate the reviewer for a thorough review and valuable comments. Below are our responses to the reviewer's concerns.

Spatial carbon intensity variations

Thank you for the suggestion. Considering the spatial variation of carbon intensity is an intriguing idea, and it is indeed feasible to further extend our scheduler to include such information in its strategy. For example, inspired by the EuroSys paper[1], the most straightforward approach is for the scheduler to first pick the data center with the lowest carbon intensity and subsequently leverage temporal variation to optimize GPU allocation.

However, to further enhance scheduler performance, such as enabling dynamic switching between different data centers during the training process, we anticipate a key challenge will be managing the carbon costs associated with spatial shifts. Transferring large volumes of data over the Internet incurs significant carbon costs, and estimating these costs can be complex. The data can traverse a random number of intermediate routers across a large geographic region, making it difficult to ensure that the benefits of performing a spatial shift outweigh its overhead. We will add this discussion to the final paper and cite related work in carbon-aware scheduling.

NasBench101

We apologize for any confusion caused. What we meant is that NasBench101 does not provide accuracy for all architectures in the defined search space. While NasBench101 does provide accuracy for 423k architectures, the actual search space is much larger. This means that if we use the search space defined in NasBench101, we may encounter architectures that are not included in the 423k provided by NasBench101 and therefore lack accuracy information.

Beyond CNN

We agree that exploring transformer-based architecture search space is a very important and timely research topic. As mentioned in our paper, our transformer-based carbon intensity predictor is designed by the LaMOO NAS algorithm, running inside our CE-NAS framework (Sec. 3.5.2 and Sec. D). The NAS-designed transformer-based predictor outperformed all existing state-of-the-art baselines, as shown in Table 1 of the paper.

We greatly appreciate the reviewer pointing out the good hybrid search space of CNN and transformer[32]. However, the search space described in this paper is not yet open-sourced, which makes it hard for us to investigate the hybrid search space of CNN and transformer. Despite it being a promising future direction. Moreover, for this work, due to our limited computing resources, we are unable to conduct large open-domain tasks for searching transformer-based architectures. The existing NAS benchmarks[33] are all based on CNN search spaces, so we currently lack transformer-based benchmarks to validate our CE-NAS. In future work, it will be promising to implement CE-NAS into transformer-based search spaces, such as for large language model fine-tuning, adapter (LoRA) design, and more.

Dominance number o($\alpha$)

Sorry for causing the misunderstanding. $\alpha$ refers to a unique architecture. In this paper/context, a sample means a neural architecture. To clarify, I would like to provide a toy example: we aim to search for good architectures with higher accuracy and lower inference latency. Consider we have sampled 5 architectures in the search space; one of them, referred to as $\alpha$, has a dominance number O($\alpha$). The O($\alpha$) can be calculated using the remaining four sampled architectures. Specifically, if an architecture has a higher accuracy than $\alpha$ and also a lower inference latency than $\alpha$, that means this architecture dominates $\alpha$. The number of architectures among these 5 that dominate $\alpha$ is the dominance number of $\alpha$ (O($\alpha$)). As such, an architecture with a smaller dominance number is more promising than an architecture with a higher dominance number. Now, back to CE-NAS, we use the dominance number to determine the order for sampling and evaluation. Specifically, during the sampling phase, we will sample many architectures and sort them based on their dominance number. When CE-NAS executes the evaluation component, the evaluation order of architectures is based on the dominance number; i.e., the architecture with the lowest dominance number has the highest priority to be evaluated.

L47

Our intention was to convey that we are not only focusing on performance metrics like accuracy. We can consider more metrics such as inference latency, #PARAMs, #FLOPs, etc. We will rewrite this part to clarify our meaning and remove any misleading information.

More related works

Thanks for providing useful resources. We carefully read the suggested papers and further explore starting from these sources. We have organized the papers into three categories as detailed below:

• Category one[2,3,4,5,6]: This category focuses on measuring carbon emissions and the impact of carbon intensity on AI training or NAS tasks but does not include any designs to reduce carbon emissions.

• Category two[7,8,9]: This category proposes energy-aware scheduling for NAS tasks but omits the effect of carbon intensity variations on carbon costs.

• Category three[10,11,12,13,14]: This category considers the effect of either temporal or spatial carbon intensity variations on the carbon emissions of data center tasks, relying on inter-job execution order scheduling and location selection to reduce total emissions. However, the scheduling techniques proposed by these works to exploit temporal carbon intensity variation do not apply when optimizing for a single NAS task.

We will include them in the final paper.

Limitations

Thank you for your suggestion. We will add the discussion of the carbon intensity predictor in the final paper.

Reference

Due to the characters limit, the references can be found in Common Rebuttal for All Reviewers.

We sincerely thank the reviewer for your insightful comments, please check our answers below.

How easy is it in practice to access live information about carbon efficiency?

There are third-party providers, such as ElectricityMap and WattTime, that allow users to query carbon intensity through APIs [APIs1] [APIs2]. As sustainability becomes an increasingly important issue in computing, we anticipate that more resources for obtaining carbon information will become available. We will include additional details about the accessibility of carbon information in the final paper.

[APIs1] https://static.electricitymaps.com/api/docs/index.html

[APIs2] https://docs.watttime.org/#tag/GET-Regions-and-Maps/operation/get_reg_loc_v3_region_from_loc_get

Discussion on the availability of carbon emissions data across different countries/regions

You are correct that carbon data signals are only available in certain parts of the world. For example, there is a lack of such information for Africa and large parts of Asia [CarbonAvailability]. However, we believe there will be a growing range of grid regions as commercial services flourish.

Currently, in our work, CE-NAS leverages the temporal variation of carbon signals to strategically allocate GPU resources for NAS methods. This means that CE-NAS can be used on GPU clusters that operate in regions with available carbon emission data. Users from regions that currently do not have these carbon signals can still contribute to sustainability goals by launching CE-NAS on remote GPU clusters.

[CarbonAvailability] https://watttime.org/docs-dev/coverage-map/

Make Figure 1 better

We will simplify this figure and write a descriptive caption based on your suggestion.

Inconsistency in the abstract and the introduction

In abstract, we want to highlight we searched small model still has high accuracy. In introduction, we want to emphasis that we can find state-of-the-art models in terms of accuracy. Sorry for making this confusing inconsistency in the different context of the paper. We will make them consistent in our paper.

Citation for the claim on lines 97-99

It shares the same citation ([60] in the paper) with the last sentence. We will add this citation to the paper in this sentence.

Definition of carbon emission and hypervolume metrics

Thanks for catching this. We will move he definition of relative carbon emission and hypervolume metrics to a more appropriate place in the paper.

Inaccurate caption statement in Figure 2

Thanks for catching this. We will fix it in our final version.

Appropriate term for NasBench

We will rephrase the dataset to benchmarks in our final version.

MnasNet should be bolded in Table 3

We will bold it as the lowest TensorRT Latency in the table.

Caption of Figure 9

Figure 9 describes the search efficiency among different NAS methods under time constraints. The total carbon cost of each method is shown above each method. We will also add more details in the figure caption.

We sincerely thank the reviewer for your insightful comments and suggestions, please check our answers below.

Comparison between CE-NAS and zero-shot NAS

We first thank the reviewer for pointing out the missing related work on zero-shot NAS in our paper. We will add the following discussion, comparison, and potential integration with zero-shot NAS in our final version.

First, Zero-shot NAS leverages training-free score functions/proxies to efficiently evaluate the performance of neural architectures [15][16]. Compared to vanilla NAS and one/few-shot NAS, zero-shot NAS is cost-effective due to its training-free nature at the search stage. However, the main disadvantage of zero-shot NAS is its evaluation performance, which can be very inaccurate compared to one/few-shot NAS and vanilla NAS. According to [15], mainstream zero-shot NAS methods, such as gradient norm [17], SNIP [18], Synflow [19], GraSP [20], GradSign [21], Fisher [22], Jacobian_cov [23][24], Zen_score [25][26], and NTK [27][28] did not achieve high rank correlations between architectures’ predicted and true accuracy. For example, in the NasBench201 search space for CIFAR10, the highest Kendall’s tau value (a ranking correlation metric ranging from -1 to 1, where 1 indicates identical ranking and -1 indicates completely reversed ranking) from these zero-shot methods is below 0.6 [21], with most values ranging from 0 to 0.5. In contrast, one/few-shot NAS [29][30] can achieve Kendall’s tau value of 0.75. Importantly, for the top 5% of architectures ordered by true accuracy, most zero-shot methods [17,18,19,20,22,23,24,27,28] have Kendall’s tau value around or below 0, equivalent to random search. Zen-score [25, 26] and GradSign [21] perform slightly better than others but still below 0.3. Based on our experiments, few-shot NAS [29] achieves Kendall’s tau value of 0.56 for the top 5% accurate architectures, significantly higher than zero-shot NAS.

Second, Our CE-NAS integrates few-shot NAS and vanilla NAS, leveraging vanilla NAS in a low-carbon intensity period to provide accurate performance feedback, effectively guiding the search algorithms and identifying promising regions in the search space. Our CE-NAS effectively guides the exploration of optimal neural architectures with minimal carbon emissions. Zero-shot NAS, due to its high randomness and inaccurate performance proxies, struggles to find satisfactory neural architectures. Additional evidence from [15] shows that the top-1 accuracy of the best zero-shot NAS method [27, 28] on the ProxylessNAS search space is 73.63%, only slightly better than random search sorted by the highest FLOPs (73.08%). In comparison, one-shot-based ProxylessNAS[31] achieves 75.1% top-1 accuracy, and few-shot ProxylessNAS further improves this to 75.91%[29]. Our CE-NAS achieves a top-1 accuracy higher than 80% on ImageNet.

In summary, we acknowledge that zero-shot NAS can be very cost-effective and energy/carbon-efficient. However, relying solely on training-free proxies to evaluate neural architectures can be extremely inaccurate, compared to one/few-shot NAS. Nevertheless, we note that CE-NAS is complementary to zero-shot NAS. Specifically, CE-NAS can integrate zero-shot NAS methods for quick evaluations, replacing one/few-shot NAS. The vanilla NAS component is a valuable complement that helps zero-shot NAS search in promising regions, thereby improving the search effectiveness of zero-shot NAS.

The integration of multiple components introduces complexity in implementation.

We appreciate reviewer g77N’s attention to this matter and would like to highlight that we have conducted many ablation studies of CE-NAS components in the appendix (Sec. F). These ablation studies provide strong evidence that these components work well together and are indispensable to the framework. Additionally, compared to the entire NAS overhead, the carbon/energy cost of the remaining components, such as the carbon intensity predictor and reinforcement learner, is negligible. Lastly, we have included the source code as supplementary material, and we invite the reviewer to inspect our research prototype, which we will open-source upon acceptance.

Reference

Due to a limit of 6000 characters, the reference can be found in Common Rebuttal for All Reviewers.

Thank you for your kind feedback. We apologize for the code being inaccessible earlier. We have now made it public, and we will organize and release the code on GitHub once the paper is accepted.