Dear Reviewer zBpp

We deeply appreciate your invaluable feedback on the further investigations of computation speed and backbone neural architecture encoders.

Below, we provide detailed responses to each of your comments.

Best regards,

The Authors.

---

Weakness 1.1

While a ResGatedGCN with FGP outperforms Flowerformer in Fig 8a, training with FGP increases the encoding time of the ResGatedGCN significantly (from to) presumably due to the flow based objective, which suggests that the Flow based objective still introduces significant pretraining overhead compared to other non flow based methods.

However in Fig 8b, FGP is faster than other pretraining methods. It is unclear which pretraining method has been compared for the ResGatedGCN in Fig 8a and how that is different from the baselines in Fig 8b.

Response 1.1

We apologize for the confusion. Our response is three-fold, and we will revise the manuscript by adding these details.

• [Clarification on Figure 8a] The X-axis values in Figure 8a represent the total time required for both architecture processing (encoding) and the pre-training phase using our method (FGP). Therefore, (1) the additional time observed should not be interpreted solely as increased architecture processing time, and (2) other pre-training methods are not used in Figure 8a.
  • [Detail 1] For ‘ResGatedGCN (non-flow-based) w/o FGP’ and ‘FlowerFormer (flow-based) w/o FGP’, the X-axis values reflect only the architecture processing time for ResGatedGCN [1] and FlowerFormer [2], respectively.
  • [Detail 2] In contrast, for ‘ResGatedGCN (non-flow-based) w/ FGP’ and ‘FlowerFormer (flow-based) w/ FGP’, the X-axis values include both the pre-training time and the architecture processing time. Thus, the time gap between the w/ FGP and w/o FGP settings reflects the cost of pre-training, not additional architecture processing time.
• [Clarification on Figure 8b] Figure 8b presents the pre-training time required for the ResGatedGCN architecture under each pre-training method. It shows that FGP achieves a pre-training time comparable to that of the fastest baseline method.
  • [Detail] Compared to the fastest pre-training method (ZC-Proxy [3]), our method (FGP) requires only 10 additional seconds to train 14K architectures over 200 epochs—a relatively minor overhead in practice. Despite this marginal overhead, FGP outperforms ZC-Proxy in 26 out of 27 settings.
  • [Note] The sum of the pre-training time for FGP in Figure 8b and the architecture processing time for ResGatedGCN w/o FGP in Figure 8a is equivalent to the time for ResGatedGCN w/ FGP in Figure 8a.
• [Speed clarification] Pre-training with FGP does not introduce additional overhead in architecture processing. Moreover, while it requires some extra pre-training time, this cost amounts to only a few minutes and results in substantial performance gains, as detailed in our response to Weakness 1.2.

---

Weakness 1.2

Overall, the time taken for pretraining with FGP seems to significantly increase over other pretraining methods. It would be nice to have a Pareto curve for performance vs time taken for each pretraining method to highlight the efficiency of FGP.

The empirical evaluations of the proposed method shows that the flow based objective can improve encoding performance, but it seems to come at additional cost. The exact tradeoffs between performance and efficiency, specifically due to the flow based objective can be made clearer by the authors.

Response 1.2

Our response is three-fold.

• [Comparison with baselines] Our method (FGP) achieves a pre-training time comparable to that of the fastest baseline method.
  • [Detail] Compared to the fastest pre-training method (ZC-Proxy [3]), our method (FGP) requires only 10 additional seconds to train 14K architectures over 200 epochs—a relatively minor overhead in practice. Despite this marginal overhead, FGP outperforms ZC-Proxy in 26 out of 27 settings.
• [Experiments] As detailed in Appendix B.5 of our manuscript, our proposed method (FGP) outperforms the strongest baseline methods (ZC-Proxy and GMAE [4]) under a fixed pre-training time constraint.
  • [Setting] We fix the pre-training time for each method and use ResGatedGCN [3] as the backbone neural architecture. All other experimental settings follow those described in Section 4.2.
  • [Results] As shown in Tables 1 and 2, FGP consistently outperforms all baseline methods across all the settings.
• [Implication] These results indicate that substantial performance gains can be achieved with just a few minutes of pre-training.

Table 1. Performance prediction results in the NAS-Bench-101 dataset under diverse fixed pre-training time.

Table 2. Performance prediction results in the NAS-Bench-201 dataset under diverse fixed pre-training time.

---

Weakness 2

While FGP is claimed to capture the flow based information implicitly, it is unclear how combining FGP with FlowerFormer is able to increase performance. It would suggest that the FGP objective adds information other than the flow to training, can the authors ablate this effect?

Response 2

Our response is two-fold.

• [Hypothesis] We hypothesize that FGP enables FlowerFormer to learn a broader range of information flow beyond what is present in labeled architectures by leveraging a large number of unlabeled architectures, thereby enhancing its ability to capture more diverse flow patterns.
  • [Detail] When trained solely on labeled samples, FlowerFormer tends to learn information flow patterns limited to that specific set. In contrast, FGP leverages a much larger pool of neural architectures whose ground-truth performance is unknown, exposing FlowerFormer to a wider range of information flow patterns. This exposure enables FlowerFormer to learn more diverse and generalizable information flow representations.
• [New experiments: Empirical evidence] Our hypothesis is empirically supported by our new experiments, which reveal that the performance of FlowerFormer tends to be positively correlated with the size of the pre-training dataset.
  • [Intuition] By varying the number of neural architectures used during FGP pre-training, we aim—though not precisely—to control the diversity of information flow patterns to which FlowerFormer is exposed. A positive correlation between dataset size and FlowerFormer performance suggests that FGP helps FlowerFormer encounter a broader range of information flow patterns, leading to improved accuracy in performance prediction.
  • [Setting] We vary the proportion of the pre-training dataset used for FGP training—0% (no pre-training), 20%, 40%, 60%, 80%, and 100%—and evaluate FlowerFormer’s performance on the performance prediction task. Kendall’s Tau is used as the evaluation metric, and all other settings follow those described in Section 4.1 of our manuscript.
  • [Results] As shown in Table 3, FlowerFormer’s performance tends to improve with larger pre-training dataset sizes, providing empirical support for our hypothesis.

Table 3. Performance prediction results of FlowerFormer under varying sizes of pre-training dataset

---

References

• [1] Residual Gated Graph ConvNets
• [2] FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer
• [3] Dynamic Ensemble of Low-Fidelity Experts: Mitigating NAS “Cold-Start”
• [4] Graph Masked Autoencoder Enhanced Predictor for Neural Architecture Search

Dear Reviewer VhSE,

We sincerely appreciate your valuable feedback on the flow surrogate analysis and presentation.

Below, we provide detailed responses to each of your comments.

Best regards,

The Authors

---

Weakness 1

There are some typos in this paper. Line 23: 'extensive' computational resources

Response 1

Thank you for pointing out the typos. We will revise the manuscript to address all identified typos, including the case you mentioned (replacing ‘extensive’ with ‘expensive’).

---

Weakness 2.1

The insight of this paper is not solid enough. The proof that the 'flow surrogate' can effectively represents the information flow is lacked.

Response 2.1

• [New preliminary analysis] While rigorous proof is challenging, we present a new analysis showing that our flow surrogate can distinguish neural architectures with distinct operations, resulting in different patterns of information flow.
  • [Setup] We conduct three binary classification tasks to determine whether a given architecture includes: (1) a 1×1 conv. operation, (2) a 3×3 conv. operation, and (3) a pooling operation. We use flow surrogate representations as input features and train an MLP as the classifier, performing 10 trials. The architectures are split into 80%/20% for training/testing.
  • [Result] As shown in Table 1, our flow surrogate achieves over 92% accuracy across all cases, indicating its ability to distinguish architectures with different operations that give distinct patterns of information flow.

Table 1. Operation classification results.

---

Weakness 2.2

Moreover, after reviewed the details of the generation of 'flow surrogate', there is no constraint that ensures different information surrogates for two architectures with largely different performance, especially when the inputs are random initialized.

Response 2.2

We acknowledge the reviewer’s concern that the flow surrogate may lack a strong constraint to effectively distinguish architectures with distinct performance. However, we offer evidence suggesting that our flow surrogate can make such distinctions. Our response is two-fold.

• [Effectiveness of flow] We claim that the ability to distinguish architectures based on their information flow allows us to differentiate between high- and low-performing architectures.
  • [Evidence 1] As shown in Fig. 6 of our paper, our flow surrogates effectively separate high- and low-performing architectures in the embedding space, even without any knowledge of the architecture performance.
  • [Evidence 2] As detailed in Sec. 1 and 2 of our paper, SOTA architecture encoders designed to capture information flow [1, 2] have shown remarkable success in performance prediction tasks, suggesting that the flow is a key feature for accurately estimating architecture performance.
• [Effectiveness of random vectors] We also claim that random vectors can still effectively capture information flow.
  • [Evidence 1] Prior work [3] shows that random vectors can yield meaningful representations when combined with adequate computational structures (e.g., convolutional layers). Similarly, our propagation mechanism enables random vectors to capture the flow.
  • [Evidence 2] Compared to learning-based alternatives, our flow surrogate outperforms architecture embeddings derived from the supervised-trained flow-aware encoder TA-GATEs [2] (see Table 2). See Appendix B.9.2 for details.
  • [Evidence 3] Note that our method is robust to the choice of random vectors, with different initializations consistently yielding similar performance. See Appendix B.9.1 for details.

Table 2. Comparison with a learning-based alternative.

---

Weakness 3

There is no clear explanation that why the current flow-based methods are computationally heavy. Given the inference time of GNN, it is hard to understand why it takes such a long computation time.

Response 3

• [Clarification] Sorry for the missing detail. The computational burden of flow-based encoders comes from their asynchronous message passing, which is difficult to parallelize. We will elaborate on these details in the revised manuscript.
  • [Mechanism] To capture information flow, they sequentially pass messages according to the topological order of nodes in a directed acyclic graph (DAG), instead of passing messages between all linked node pairs at once.
  • [Detail 1] As detailed in Sec. 3.2.1, each node is assigned a topological order based on its position in the DAG. Nodes with no incoming edges (typically the input node) are assigned order 1. Nodes with incoming edges only from order-1 nodes are assigned order 2, and nodes with incoming edges only from order-1 and -2 nodes are assigned order 3, and so on.
  • [Detail 2] Message passing begins at the order-1 nodes, which send messages to order-2 nodes, where the messages are updated. These updated messages are then moved to order-3 nodes. The process continues through the DAG, passing messages to later-order nodes until reaching the highest-order node (typically the output node). The process is then reversed: messages are propagated backward through the graph until they reach the input node again. This process constitutes a single message-passing layer in the flow-based encoder. Note that this process is hard to parallelize, as each sub-step depends on the output of the preceding one.
  • [Summary] Thus, unlike typical GNNs that perform message passing in a single step per layer, flow-based encoders need multiple steps per layer proportional to twice the length of the longest path from the input node to the output node, which is hard to parallelize.

---

Weakness 4

Figure 3 is confusing. In the middle sub-figure of Process 1, why the 'In' information is directly used for the computation of the 'Out'? Even they are not directly connected. Is this means that all historical information will be used when computing the high-level information? Will this cause a requirement of large memory?

Response 4

Sorry for the confusion. Our response is three-fold, which we will elaborate on in the revised manuscript.

• [Figure clarification] We represent deactivated computations using grey-colored arrows. Thus, when updating the embedding of a given node, only the embeddings of nodes directly connected via incoming edges are used; others are not used.
  • [Detail] The middle sub-figure of Process 1 shows the computation for updating the embeddings of ‘1×1’ and ‘3×3’ operations. Here, the embeddings of the respective operations and the ‘in’ node are used, while the ‘out’ node is not involved.
• [Embedding clarification] We do not store all historical embeddings. In the forward pass, order-k nodes’ embeddings are updated using only order-(k-1) and current order-k nodes’ embeddings; in the backward pass, only order-(k+1) and order-k nodes’ embeddings are used.
  • [Detail] Node ordering is detailed in our Response 3 and Sec. 3.2.1 of our paper.
• [New experiments: Memory & time] Our flow-surrogate computation requires less than 10MB of memory and takes under 1 minute to process 14K architectures.
  • [Setup] We measure the difference in memory usage before and after the computation, along with the execution time.
  • [Result] As shown in Table 3, our method processes 14K architectures (# of architectures in NB-101 and 201) in under 1 minute while using less than 10 MB of memory.

Table 3. Memory and time consumption in obtaining flow surrogates.

---

Weakness 5

The sum pooling for multiple incoming information is not reasonable, since the summation can not distinguish the incoming source which is too rough for simulating the real information flow.

Response 5

Our response is two-fold.

• [Clarification] While sum aggregation (pooling) is permutation-invariant, our flow surrogate computation mechanism allows the distinction between different sources of incoming information.
  • [Goal] We need to distinguish information from different sources when each source involves distinct operations (since sources with identical operations yield indistinguishable information, we do not attempt to differentiate them).
  • [Detail 1] As detailed in Sec. 3.2, messages (i.e., vector representations of incoming information) are transformed based on the operations they pass through, with each operation type having its own projection function. Thus, messages that have undergone different operations are likely to be distinct.
  • [Detail 2] Therefore, since messages are sufficiently distinct when originating from different operations, the aggregation function does not need to explicitly account for the specific operation each message has undergone.
• [New experiments: Alternatives] We show that sum aggregation outperforms other commonly used aggregation functions in GNNs.
  • [Setup] We use 2 variants of flow surrogate where the sum aggregation is replaced with other common aggregations in GNNs: mean and max. Other settings are the same as in Sec. 4.2.
  • [Result] As shown in Table 4, our method that uses sum aggregation outperforms its alternatives.

Table 4. Comparison with other aggregations.

---

References

• [1] FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer
• [2] TA-GATES: An Encoding Scheme for Neural Network Architectures
• [3] Deep Image Prior

Dear Reviewer VhSE,

We are grateful to hear that you are happy with our clarification, and we deeply appreciate that you have increased the score accordingly.

We also thank you for your constructive feedback. We will thoughtfully incorporate your comments, along with the additional analyses, into the revised manuscript.

Warm regards,

The Authors of Submission 11597

Dear Reviewer df83,

We sincerely appreciate your invaluable feedback on our hyperparameter and encoder analyses.

Below, we provide detailed responses to each of your comments.

Best regards,

The Authors

---

Weakness 1

The values of \lambda_1 and \lambda_2 are not mentioned, only their search space. I'm concerned that the performance of the method might be tuned separately for each test setting, or that performance is not robust to the choice of \lambda

Response 1

Our response is two-fold.

• [Clarification] We apologize for the missing detail. All hyperparameters were tuned using the validation set, which is described in Section 4.1 of our manuscript.
  • [Configuration upload] We will report the validation-best hyperparameter configurations to our revised manuscript and upload them to Anonymous GitHub after the rebuttal phase since the rebuttal policy prohibits uploading additional content during this period.
• [New Experiments: Robustness Analysis] Our method is robust to the choice of the $\lambda_{2}$ and $\lambda_{2}$ hyperparameters.
  • [Setting] Recall that our search space for $\lambda_{1}$ and $\lambda_{2}$ includes three configurations: $(\lambda_1$, $\lambda_2) \in \lbrace (\frac{1}{3}, \frac{2}{3}), (\frac{2}{3}, \frac{1}{3}), (\frac{1}{2}, \frac{1}{2}) \rbrace$, as detailed in Appendix A.4 of our manuscript. In this additional experiment, we report the performance prediction task results under each configuration. We use ResGatedGCN [1] as our backbone architecture encoder, and other settings are the same as those in Section 4.2 of our manuscript.
  • [Results] As shown in Table 1, the performance variations across these settings remain marginal and consistently outperform the strongest baselines (GMAE [2] and ZC-Proxy [3]). These results highlight the robustness of our method to hyperparameter choices.

Table 1. Results of performance prediction under each hyperparameter configuration.

---

Weakness 2

It seems to me that a major part of this work is that no "special components" are needed to predict the flow, as that information is now captured in the embedding. That unlocks a large design space of less-constrained encoders. However, I don't see any mention of what other encoder designs / architectures were considered.

Response 2

Our response is two-fold.

• [Clarification] Sorry for the missing detail. Since a neural architecture can be represented as a graph (see Section 2.2 of our manuscript), we adopt graph neural networks (ResGatedGCN and GIN) and a graph transformer (FlowerFormer) as backbone architecture encoders. We will incorporate these details in our revised manuscript.
  • [Detail] This choice aligns with common practice in neural architecture encoding work [2, 3, 4], where graph neural networks and graph transformers are widely used to encode neural architectures.
• [New Experiments: Additional Encoders] The superiority of our method over baseline pre-training methods holds across different types of neural architecture encoders.
  • [Clarification] As the reviewer noted, our method is compatible with a wide range of neural architecture encoders.
  • [Setting] To validate this, we additionally evaluate our method using two alternative encoders: an MLP-based encoder [5] and a transformer-based encoder [6]. All other experimental settings follow those described in Section 4.2 of our manuscript. We compare against the two strongest baselines, GMAE and ZC-Proxy.
  • [Results] As shown in Tables 2 and 3, our method (FGP) outperforms both baselines, demonstrating that its effectiveness holds valid under other choices of backbone neural architecture encoders.

Table 2. Results of performance prediction using the MLP-based architecture encoder. Since this encoder does not produce node embeddings, the node-feature-generative method (GMAE) is not applicable.

Table 3. Results of performance prediction using the Transformer-based architecture encoder.

---

Weakness 3

Other comments: line 111: variation -> variational

Response 3

Thank you for pointing out the typos. We will revise all identified typos, including the one mentioned by the reviewer.

---

References

• [1] Residual Gated Graph ConvNets
• [2] Graph Masked Autoencoder Enhanced Predictor for Neural Architecture Search
• [3] Dynamic Ensemble of Low-Fidelity Experts: Mitigating NAS “Cold-Start”
• [4] Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• [5] A study of neural architecture encoding
• [6] TNASP: A Transformer-based NAS Predictor with a Self-evolution Framework

Dear Reviewer RQZD,

We deeply appreciate your invaluable feedback on our theoretical analysis and the discussion of our flow surrogate.

Below, we provide detailed responses to each of your comments.

Best regards,

The Authors

---

Weakness 1

The paper acknowledges that FGP demonstrates empirical effectiveness across various benchmark, further theoretical investigation into their flow surrogates would make this work very strong.

Response 1

• [Permutation invariance] In response to the reviewer’s suggestion, we provide a preliminary theoretical analysis demonstrating that our flow-surrogate computation is invariant to node permutations (i.e., node indexing), a desirable property when representing the information flow of a neural architecture to ensure effective downstream task performance.
  • [Intuition] Prior works [1, 2] have shown that accurately capturing the information flow within a neural architecture is critical for downstream tasks, such as performance prediction. Therefore, for our flow surrogate to be effective in such tasks, it should accurately represent this information flow.
  • [Crucial aspect of flow] In the NAS-Bench datasets used in our experiments, the information flow within a neural architecture is independent of the indexing of individual operations. Accordingly, it is desirable for the computation of flow surrogates to be permutation-invariant with respect to these operations.
  • [Proof sketch] Thost and Chen [3] theoretically demonstrate that asynchronous message passing on a directed acyclic graph—also used in our flow-surrogate computation—is invariant to node permutations when the permutation-invariant pooling function is employed for neighbor aggregation. Their theoretical result extends to our setting, indicating that our method likewise ensures permutation invariance. We will add this analysis to our revised manuscript.

---

Weakness 2

Even though the authors have touched on the issue that they have limited understanding of the types of information flows that FGP could capture, it would be good to see a thourough discussion around it.

Response 2

Our response is two-fold: (1) empirical evidence supporting the effectiveness of the flow surrogate in capturing information flow, and (2) a discussion of the challenges in providing more rigorous theoretical justification.

• [New experiments: Preliminary analysis] We present a new preliminary analysis suggesting that our flow surrogate can differentiate neural architectures with distinct operations, resulting in different patterns of information flow.
  • [Setting] We conduct three binary classification tasks to determine whether a given neural architecture includes: (1) a 1×1 convolution operation, (2) a 3×3 convolution operation, and (3) a pooling operation. We use flow surrogate representations as input features and train an MLP as the classifier, performing 10 trials. The architectures are split into 80% for training and 20% for testing.
  • [Results] As shown in Table 1, our flow surrogate achieves over 92% accuracy across all cases, indicating its ability to distinguish architectures with different operations that give distinct patterns of information flow.

Table 1. Operation classification results. The mean and standard deviation over 10 trials are reported.

• [Challenges] Unfortunately, a more rigorous theoretical analysis is non-trivial, primarily due to two key challenges:
  • [Definition of flow] It is challenging to formally define information flow within a neural architecture with a single mathematical formulation. This is because the actual information flow is highly dependent on the specific parameter values, which can vary significantly across models.
  • [Non-linearity] Both the actual neural architecture and our flow-surrogate computation involve multiple non-linear functions, which complicates the derivation of rigorous theoretical conclusions.

---

Weakness 3

Without the above two points, the claims are purely emperical and not strong.

Response 3

Given our responses 1 and 2, while our additional results may not fully meet the reviewer’s expectations, we believe they adequately address the two concerns raised.

• [Regarding point 1] In response to the concern about the lack of theoretical analysis, we provide a theoretical justification showing that our flow surrogate computation is permutation invariant with respect to node ordering, which is an essential property for effectively capturing information flow.
• [Regarding point 2] In response to the concern regarding the lack of thorough analysis on the information flow captured by the surrogate, we provide a preliminary analysis indicating that our flow surrogate can distinguish information flows involving distinct operations.

---

References

• [1] FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer
• [2] TA-GATES: An Encoding Scheme for Neural Network Architectures
• [3] Directed Acyclic Graph Neural Networks

Dear authors,

I know it's late but for some reason none of the reviewers have asked that, so here it goes: why are all your Kendall Taus much higher than 1? I'm guessing you multiplied them by 100, but this should be clearly explained and not subject to guessing...

Best AC

Dear Area Chair wYwe,

We apologize for the confusion, and thank you for your detailed question regarding our results. As you noted, all experimental results reported using the Kendall’s Tau metric are multiplied by 100.

We will clarify this in the revised manuscript and will also review the entire manuscript to address any points that may potentially cause confusion for readers.

Once again, thank you for your question, which helps improve readers’ understanding of our experimental results.

Warm regards,

The authors of submission 11597