We thank the reviewer for their feedback. We have consolidated the reviewer concerns into the following topics and will include them in our updated paper.

1. Computational cost compared to fine-tuning

Fine-tuning introduces two additional implicit costs that do not impact our approach:

• Fine-tuning requires a dataset that is reasonably large and diverse. Ensuring quality, quantity, and diversity of the dataset can present a high cost.
• Fine-tuning also requires the model weights to be accessible which is not always the case. This enables a broader applicability of our approach.

2. Tool use

Tool use is synergistic with Flow-of-Options. In our paper, FoO seeks to improve the base reasoning capabilities of the LLMs even in the absence of tools. However, tools can be a force multiplier in our work. We conducted a small experiment to illustrate this further. We incorporate external research paper retrieval as a tool with FoO. The tool retrieves and provides two papers as context into the Option Generator LLM, for the example task of fine-tuning protein language models. We note the two retrieved papers incorporated into the option generator’s context: ProtBERT (Elnaggar et al. 2021) and ESM (Rives et al. 2021). Following is the resultant FoO produced: https://ibb.co/GQL0P06N. We see that the nodes 1 and 2 in the FoO incorporates the information in the papers when proposing options. In this way, tool use is synergistic with our work.

3. Creativity and Discovery

This is indeed an interesting point. We believe Flow-of-Options incorporates “combinational creativity” as described in Boden 1998. For instance, in the case of the Drug-target Interaction problem (DTI task from Table 4), our approach proposes the following ML model architecture option: Linear → Swish → dropout → Linear → GeLU → Linear (for regression). This is not an existing model per se. Although the individual components, such as the linear and activation layers exist, the specific combination of these layers can be considered novel and performs well on the task.

The individual nodes can also be combinations of existing methods. For instance, in the Drug Combination problem (DC task from Table 4), existing human baseline on the leaderboard computes features of the drug molecules using packages such as RDKit, combined with a feed-forward neural network for prediction. In contrast, our approach proposes a novel ML pipeline that combines feature extraction using ChemBERTa (an existing language model for computing the embeddings of drug molecules), with feed-forward neural network for prediction. Although the individual components exist, this combination can be considered novel. In this sense, Flow-of-Options can support combinational creativity and discovery.

4. Synergy with humans

Q1

Figure 14 compares the performance of the ML approaches produced by FoO vs. human designed baselines on the TDC leaderboard. The figure shows that our approach mostly achieves > 80% of an expert human’s performance. In other words, on average, it is comparable to (though not always better than) a human expert. In some cases, it outperforms the human baseline. For instance, in the drug combination (DC) task from Table 4, our approach outperforms the human baseline on the TDC leaderboard. For naive users however, FoO can significantly democratize ML.

Q2

Currently, the LLM proposes the options. However, these options could be informed through human guidance. Similar to our demonstration on tool use incorporating external references, the option generation can be conditioned on user inputs. In this way, the Flow-of-Options data structure can be a synergy of LLM and human knowledge leading to discovery of novel combinations.

5. FoO with learning

Our current implementation of Flow-of-Options seeks to improve the base reasoning capabilities of LLMs. However, it may be complementary to approaches that incorporate closed feedback learning. For instance, it may be possible to learn over walks produced by the Flow-of-Options data structure. Since each walk is associated with a corresponding metric, it can be treated as a supervised learning problem, or incorporated with reinforcement learning. This would make for an interesting future exploration.

6. Domain expertise solutions

We explore two such algorithms (neither use LLM-based agents): AutoGluon (Erickson et al. 2020) for data science tasks in Section 4.1, and DeepMol (Correia et al. 2024) for TDC ADME-Tox tasks in section 4.2. DeepMol is a specialized framework for ADME-Tox domain, and AutoGluon is a specialized framework for typical Data Science domain. Both DeepMol and AutoGluon optimize for ML models without using any LLM agents. AutoGluon has demonstrated improvements over Bayesian Optimization based AutoML such as Auto-Weka and other AutoML optimization frameworks. Hence we chose AutoGluon as the sota model. DeepMol is currently one of the sota AutoML methods for TDC tasks.

We thank the reviewer for their constructive feedback. We will incorporate the suggestions into our paper.

Q1

We measure the reduction in execution time when consistency checker is added to identify invalid paths (as opposed to w/o it), and planner when the adapter is added (as opposed to w/o it). We also measure costs for each component when it is added. We measured these on average for tasks from Table 1. The results are noted here: https://ibb.co/7d4vmw7f. Adding these elements offer performance benefits at minor additional cost. We also performed a cost ablation for all components: https://ibb.co/LXgW5wBs

Q2

Please note our scaling performance response to Reviewer utUi (Q1)

Q3

This is a good question. We believe there are two cases in this scenario:

• Case 1: A single task plan suffices, but has conceptually different option decompositions: For instance, a string denoting a drug molecule can be processed using the specialized RDKit package which computes chemical properties of the molecule such as molecular weight. Alternatively, we can convert the strings to vectors via NLP methods. These are conceptually different. However, the task plan constructed by the planner is sufficiently high-level that conceptually distinct options are supported via the option generator LLM. See the example FoO: https://ibb.co/GXXgj5p. Options 1 and 2 are feature processing, denoting the RDKit method and "NLP style" vector embedding respectively.
• Case 2: A single task plan cannot denote the different decompositions: The experiments in our paper currently do not fall into this category. However, it is possible to envision a nested Flow-of-Options in our future work, with FoO data structure over task plans as well. In this case, the depth will be $n=1$ and width $k$ denotes the different task decompositions. Internally, each task plan node can be captured similar to our work in this paper. Hence, a nested version of FoO can be explored for such problems.

Expanded evaluation on LLM backbones

We provide additional results here: https://ibb.co/NRMNTw4. Please see response to Reviewer 4BTz (W4) for more details. We will expand further details around this in our paper as well.

Comparison to the tree and DAG

The current implementation of Data Interpreter does not appear to save the produced data structure. SELA produces a text formatted tree shown in the excerpt below:

[Node 0-2]
Engineer features if necessary to improve model performance. Additionally, generate a correlation matrix heatmap to identify highly correlated features, which might be candidates for removal or transformation.

We convert this to a visualization of the tree for a data processing task and demonstrate it alongside FoO for the same task. Tree: https://ibb.co/NdzGtFYW. FoO: https://ibb.co/mVqnMpk4. We note the qualitative option diversity and connectivity improvements of FoO here.

WordClouds

We chose WordClouds as a compact representation of the frequency and diversity of the model choices made by our approach. We'd be happy to replace the wordclouds with bar charts (for proportion of options, i.e. frequency) which is more quantitative: https://ibb.co/3yVLzv0d (For Fig 2), https://ibb.co/tTS4Zwmz (Fig 5)

Hyperparams of baselines

We have performed hyperparameter tuning on the key number of iterations parameters for SELA, DS-Agent, and Data Interpreter prior to running them (the other frameworks do not include hyperparameters apart from LLM backbone to use).

• SELA: Increasing the number of iterations would lead to a significant explosion in terms of time (5 → 6 iterations increased time from ~21 mins → to ~57 mins on average), but we did not note a corresponding improvement in accuracy beyond 5 iterations. This could be related to the complexity of the problems in our experiments. More complex problems may require more iterations of SELA (albeit at a significant time cost).
• Data Interpreter: Increasing the number of iterations increases the amount of time (to a much lesser degree than SELA), but like SELA, it did not impact accuracy beyond 5 iterations. It is also worth noting that the execution failures in Data Interpreter did not correlate to the number of iterations (possibly because failures are related to the presence of cycles in the LLM-built DAG, noted as one of the shortcomings of DI).
• DS-Agent: The number of iterations increases the amount of time, and we also noted an improvement in accuracy however, that would stabilize at about iterations 4 to 5. We set this to 5 to be consistent in our cost/time comparisons across all methods.
• Our approach: For our approach, the number of iterations increases the amount of time with improvement in accuracy. For consistency with all the baselines, we fix number of iterations to 5.

We generally chose 5 iterations to maintain consistency across all our comparisons on time and cost assessments. We note relative advantages of the different methods in Appendix F.

We thank the reviewer for their constructive feedback. We will incorporate the suggestions into our updated paper.

Experimental Designs or Analyses

Point 1

The documentation for DS-Agent, AutoGluon, and SELA note that they are indeed suited to tabular tasks similar to the ones explored in our experiments. This is noted in their corresponding papers. Our dataset is not specifically curated by us, but is rather an existing dataset obtained from (Guo et al. 2024). We note the following potential reasons for the Table 1 observations:

• AutoGluon: AutoGluon explores a fixed set of models which is well suited for some, but not all the data science tasks. Indeed some of the problems in the dataset, such as language-based tasks, are currently not supported by AutoGluon (which is a shortcoming of this method). Nevertheless, AutoGluon is a quite popular framework and is also an example of a non-LLM based AutoML framework. Hence we felt it would be useful to incorporate it as one of the baselines in our experiments.
• SELA and DS-Agent: Both these frameworks incorporate self-correction over the LLM proposed methods. The LLM proposes a method and then repeatedly reflects on it to modify the method and improve the result. In “Large Language Models Cannot Self-Correct Reasoning Yet” by Huang et al. ICLR 2024, the authors note LLM performance can degrade after self-correction. Accuracy drops as the number of repeated self-correction calls increase (Table 3 of the cited paper). Both SELA and DS-Agent perform $>1$ self-correction calls. Besides zero-shot, Data Interpreter (DI) does not incorporate this form of self-correction, and tends to perform better among the baselines (although it has other shortcomings as noted in the paper). LLM self-correction is not needed with Flow-of-Options, where the different options are already encapsulated and systematically explored via the FoO data structure. However, this does not mean that SELA and DS-Agent are not suited for the types of tasks in our experiments, but rather that there is room for improvement here, which we believe Flow-of-Options contributes. It is possible that if self-correction were removed from DS-Agent and SELA their performances could improve. However, we do not change the baseline implementations available on Github for the purposes of comparison.

Point 2

This is correct with the exception that DS-Agent (alongside FoO) also incorporates mechanisms to improve upon past iterations. The other approaches do not have this design element. Our intention with Figure 6 is to:

• Add experimental and quantitative evidence to our claims of being able to improve over subsequent iterations.
• Demonstrate that some of the baseline approaches fail in certain iterations (e.g., DeepMol and data interpreter).

We wanted Figure 6 to demonstrate the benefits of our approach at the agentic design level in comparison to the baselines.

W1

We seek to mitigate this with the beam search (Section 2: beam traversal, page 2), which selects the top $b$ options for exploration, thereby limiting the explored paths to the more promising options. This prevents the bad options from being revisited. We note additional computational improvements in Section 3.2, Page 5 (Also note response to Reviewer utUi Q1).

W2

We hope that our response to Point 1 above regarding baselines in Table 1 offers additional context around our compared baselines. Please also note additional experiments reported for W4 to strengthen our evaluation.

W3

Currently, in addition to the classification/regression tasks, the RL and image generation tasks, we explored the following tasks in Appendix B.2 and B.3 to further demonstrate generalizability:

• Clustering
• Machine Translation
• Traveling Salesman task
• Case study on a math problem

We hope that these additional tasks help demonstrate broader generalizability of our framework.

W4

We provide additional results for LLMs (including an open-source LLM) over a subset of the TDC tasks from Table 2: https://ibb.co/NRMNTw4. Arrows indicate whether lower or higher metrics are preferred. Our approach helps consistently improve performance across the newly added LLMs and also outperforms the baselines for the tasks. Note that for some cases "--" indicates that the model failed to produce working code within three attempts. Hence, we see that in addition to improving the overall task performance compared to baselines, FoO can also help mitigate the failure rates in code generation.

Q. SELA cost

A lot of the time consumed in SELA is for the execution of the code from the MCTS rollouts. Each code execution is performed sequentially (it does not support parallelization like our framework -- parallelization is discussed in Section 3.2, Page 5) and therefore takes a significant amount of time. However, the code execution does not involve LLMs per se, therefore it does not reflect in the cost, which is associated with querying LLMs.

We thank the reviewer for their constructive feedback and will incorporate these suggestions into our updated paper.

W1

While our paper is on Application-Driven Machine Learning, FoO does not explicitly specify steps such as model selection or feature engineering, but rather adapts to the task plan that is produced. We show this generalizability (beyond classification/regression of sections 4.1 and 4.2) as follows:

• RL and Image Generation (Section 4.3) - uses a different ML pipeline than tasks in 4.1 and 4.2 (e.g., reward formulation)
• Unsupervised clustering (Appendix B.2.1) – is a different model than the data science tasks
• Machine Translation (Appendix B.2.2) – does not involve a specific feature engineering like the data science tasks (only tokenization)
• Traveling Salesman Problem (Appendix B.2.3) – involves neither ML model selection nor feature engineering
• Case study on a math problem for solving the complex solutions to $x^2 + 2x = i$ using FoO (Appendix B.3) - does not involve an ML model selection nor any feature engineering.

In particular, the TSP and math problems are larger deviations from the typical data science pipeline.

W2

FoO enforces diversity in LLM solutions through compressed, interpretable representations that support memory of past explorations when combined with case-based reasoning. Specifically, FoO offers the following advantages over exhaustive combinatorial search:

• FoO acts as “memory” of key info on previously explored task solutions - can be saved, reused and adapted from one task to another via deployment (Section 3.2). Reusing FoO from task $T_1$ to $T_2$ is fast and achieves good results (Tables 1, 2 and 3). Hence, prior knowledge encapsulated in the FoO can be effectively reused for $T_2$, whereas an exhaustive search would have to be repeated for each task.
• Exhaustive combinatorial search w/o FoO assumes expert knowledge to set up the combinations themselves. An expert ML scientist can enumerate $k$ options for each step in the task, but a naive user may lack this knowledge. FoO encapsulates the knowledge of LLMs enabling even naive users to specify their problem in natural language, without demanding expertise. Even for experts, FoO can serve as a "force multiplier”, potentially enumerating options that the expert may not have thought of.
• The formulation of FoO also supports integration with tool use (Please see Reviewer cGuq Point 2).

Q1

Average number of options $k = 4$ and average number of steps $n = 3$. These are specified as hyperparameters into the framework. The computational complexity of FoO is indeed dependent on $n$ and $k$ and we have currently implemented the following solutions to mitigate the computational complexity (From Section 3.2, Page 5):

• Parallelization: We parallelize the executions of walks through the FoO so that even if there are a large number of walks, the computational time taken is reduced.
• Pruning: We prune some of the low-scoring edges of the FoO (so that they are not explored) which also helps reduce the computational complexity.

Lastly, the consistency checker (Section 2, page 3) identifies invalid paths in the FoO. Although the total number of paths are $k^n$, not all paths are valid (as $n$ increases, the proportion of invalid paths is also higher). Example of invalid paths are shown in Fig 4 (Page 3). Consistency checker empirically results in $\approx 22.8$% reduction in the number of paths to explore (Empirical results noted in response to Reviewer zYfD Q1). Hence, in practice, computational complexity scales quite differently. Please see measured time performance with $n$ and $k$ (averaged across three runs on the CW TDC task of Table 2): https://ibb.co/0VvqpBR5. From $n=1, k=3$ to $n=3, k=3$, the number of paths is $9\times$ more. However, corresponding time only scales by $\approx 7\times$ (this includes parallelization and consistency checking, but excludes pruning which can further improve the efficiency).

Q2

The development phase took about 13.29 mins on average in our work for the data science tasks. The development time is dependent on the complexity of the task based on $n$ and $k$ as noted in Q1.

Q3

In beam search, our goal is to narrow the search to the most effective set of options, by selecting the top $b$ options at each level, and exploring paths between them to discover potentially improved combinations of the top options. This can be visualized as exploration over a reduced FoO where the nodes are just the top performing options found thus far.

In our experiments, we start with a full beam width of 100% (exploring all the options), and reduce it to the top 50% of the options in the last two iterations of development. In Appendix C (Figure 13), we demonstrate that beam search can discover new combinations of top performing options, resulting in improvements in the final performance of the methods.