We thank the reviewer for the detailed and constructive review and appreciate your recognition of our paper’s strengths. Below, we address your specific comments.

Relationships to Magentic-One [7] and Agent K [2]

R1 Thank you for pointing out these valuable concurrent papers that we had previously missed. Although our Agent Manager functions similarly to [7], it performs constraint-aware plan selection and multi-stage verification, making it more task-specific and grounded.

For Agent K [2], the current version is not open-sourced, which limits direct comparison. Nevertheless, we provide a qualitative analysis. Although Agent K also leverages LLMs to orchestrate ML pipelines, it is tailored for Kaggle competition settings and relies on a training-based approach with high search overhead. In contrast, AutoML-Agent introduces a platform-agnostic, RAP and multi-stage verification framework designed for broader AutoML applications beyond Kaggle with a training-free search method.

We have clarified these connections and distinctions in Related Work, including those in [1–8].

Sensitivity to prompts and rationale for system prompts

R2 While agent-specific prompt design is not the primary focus of this paper, for user prompts, our results suggest that AutoML-Agent is minimally sensitive to prompt specificity. In both constraint-free (somewhat vague) and constraint-aware (precise) settings, the system achieves high success rates with comparable average NPS (0.804 vs. 0.810), indicating robustness to prompt variation. Clearer prompts generally lead to better outcomes, but the system is designed to handle vague instructions gracefully: the combination of the verification mechanism and RAP allows the system to tolerate a range of prompt qualities. Even when the Prompt Agent interprets vague inputs broadly, execution verification ensures that only valid and high-performing solutions are accepted.

For system prompts, please refer to Reviewer mnKA-R4.

Restrictive skeleton Python scripts

R3 The provided skeleton serves as a default scaffold for typical ML tasks, offering a high-level structure without enforcing task- or data-specific code. It enables agents to complete the template using theoretically any specialized modules by following the TODO list. We believe this approach is significantly more flexible than the data-specific templates used in the DS-Agent framework. Typically, the only component that requires adjustment for new tasks is the evaluation metric.

Instruction fine-tuning in Prompt Agent

R4 For entirely new problem classes, we could extend the existing model through additional fine-tuning or by providing a few in-context examples. EvoInstruct can be used to generate instruction–response pairs in the new domain to support this adaptation.

To provide some context, instruction fine-tuning was initially necessary to ensure precise parsing of user inputs for interoperability between agents, as this work began before OpenAI supported accurate structured outputs. With recent updates to GPT models, we can now directly use a JSON schema, eliminating the need for fine-tuning in many cases. The Prompt Agent can accurately parse user queries by referencing the schema explicitly. The schema is available at /prompt_agent/schema.json.

Potential use of PaperQA [9]

R5 We agree that a tool like [9] could improve the literature retrieval component. Incorporating PaperQA-style scientific querying could enhance citation relevance and support more context-aware prompt generation, ultimately leading to better planning decisions. We will acknowledge this direction in the paper and consider it for future iterations.

In-context examples for Retrieval-Augmented Planning

R6 The in-context examples (plan_knowledge) are automatically generated on the fly. They are extracted, summarized, and organized from raw retrieved documents and texts (e.g., search results, arXiv papers, Kaggle notebooks) before being passed to the Agent Manager for planning. We have updated the GitHub repository to include examples at /example_plans/plan_knowledge.md.

"if no dataset is found...".

R7 When we mention relying on the LLM’s inherent knowledge, we are referring to a fallback mechanism intended to produce the most plausible output given limited context. However, in the absence of actual data, the pipeline will ultimately fail during final implementation verification due to runtime errors. Our approach assumes that a dataset is either provided (e.g., image classification) or retrievable via search (e.g., node classification). If no dataset is found, the agent will prompt the user to supply one when in interactive mode. We will revise the text to clarify this assumption.

Writing suggestions

R8 We have addressed all the mentioned issues, including a table with standardized details (see anonymous GitHub). This will be included in the final version.

Thank you for recognizing our contributions and for your thoughtful, detailed feedback. Below, we respond to your specific concerns.

Scalability concerns

R1 This is an important point. A key motivation behind AutoML-Agent is to reduce the computational overhead of the search process. Our framework achieves this by relying solely on LLM inferences for agent communication and leveraging retrieval-augmented knowledge for planning, avoiding expensive training-feedback loops.

Figure 4c and Table 11 demonstrate the scalability of our approach. AutoML-Agent has a time usage standard deviation of only 1 minute, compared to SELA’s 14 minutes, across datasets ranging from ~1K to 143K instances. As a training-free method, AutoML-Agent maintains stable search time, whereas training-based methods like SELA exhibit fluctuations depending on dataset and model size. This suggests our framework scales efficiently without performance degradation.

More precisely, our computational complexity is approximately $O(1 + m)$, compared to $O(s + m)$ for training-based methods, where $s$ denotes the search-time training feedback and $m$ the model training time before deployment.

Lack of a statistical significance test

R2 Currently, Tables 5, 6, and 7 report the results with standard deviations over five independent runs. While we did not include formal statistical tests, reporting standard deviations is a common practice to convey the reliability of performance differences. Although we believe this is sufficient, please let us know if there is a specific test you would like to see.

Comparisons with TabPFN

R3 TabPFN is indeed a highly relevant and efficient method for tabular classification. We would like to clarify that TabPFN is already included in our baselines under the Human Models category ($\S$C.3). For your convenience, we summarize the results below.

Lack of real-world deployment testing

R4 We acknowledge the importance of real-world deployment testing. To this end, we designed our framework with a deployment stage via a Gradio API, showing a proof-of-concept using benchmark datasets. While full deployment in production environments involves additional considerations (e.g., infrastructure, latency, data privacy, and so on), we view this as a promising direction for future work.

Extension to RL tasks

R5 AutoML-Agent’s modularity allows for extension to RL tasks. While our current implementation focuses on (un-)supervised pipelines, the framework is not fundamentally limited to these settings. Extending AutoML-Agent to RL would require incorporating domain-specific modules—for example, agents for environment interaction, action space design, and reward handling (see https://www.automl.org/autorl-survey). As noted in the paper, tasks such as RL or RecSys would benefit from additional agents to manage actor-environment interactions and reward modeling.

Thanks to its modular design, one could integrate an “RL agent” that interfaces with an environment (e.g., OpenAI Gym), while the planner generates RL-specific steps such as policy training and evaluation. Although this extension is certainly feasible, it could be substantial enough that an AutoML-Agent for RL—perhaps termed AutoRL-Agent—would merit contributions worthy of a separate paper.

What are the main failure cases of AutoML-Agent?

R6 Please kindly refer to Reviewer mnKA-R3.

Edge cases in code generation

R7 We appreciate the reviewer’s concern regarding LLM hallucinations in code generation. Our framework is specifically designed to mitigate this issue through two core mechanisms: RAP and multi-stage verification.

• RAP grounds the planning agent’s decisions in real-world information by retrieving external resources, rather than relying solely on the LLM’s internal memory, which can produce hallucinated content. This ensures that generated plan steps are more likely to be valid and based on established approaches.
• Our multi-stage verification process acts as a safety net. If the LLM produces incorrect or nonsensical code, the implementation stage will flag it, triggering a corrective attempt.

This dual mechanism—prevention through retrieval and correction through verification—has proven effective (please refer to Reviewer EaoR-R5 for robustness experiments on noisy information). To further reduce risk, we provide agents with structured pipeline skeletons, which help constrain generation.

We thank the reviewer for the insightful and encouraging review. We greatly appreciate your positive assessment of our paper’s novelty, empirical rigor, and technical soundness. Below, we address your thoughtful concerns.

High computational cost

R1 We would like to clarify that, apart from the final implementation verification step, AutoML-Agent relies solely on inference throughout the entire search process, including planning. Unlike many prior works, this design ensures efficiency, as runtime does not scale with dataset size or model complexity. Our modular design also enables parallel execution of sub-tasks, further reducing wall-clock time. That said, while computational demands remain low (as discussed in Reviewer Son5-R1), using high-performing LLMs for each agent may still incur monetary costs (as noted in Reviewer mnKA-R3).

Dependence on external APIs and models

R2 We agree that relying on a proprietary model like GPT-4o entails cost and reliability implications. However, our framework is model-agnostic—agents can operate with any sufficiently capable LLM. We used GPT-4o to demonstrate state-of-the-art performance, but open-source or local models can be substituted to eliminate external API dependencies, albeit with some performance trade-offs. Given the rapid advancements in LLMs, we expect costs to decrease and become increasingly justified by the resulting performance gains.

Computational and monetary cost implications in practical deployments

R3 In practical deployments, the computational and monetary costs of running AutoML-Agent primarily depend on the number of planning iterations (typically 3–5), the complexity of the downstream task (e.g., data modality, task type), and the size and access method of the LLM (e.g., API vs. self-hosted). After pipeline generation, the cost shifts to standard training and inference of the downstream models, which depends on the specific components selected.

Explicit discussion of limitations and potential failure modes

R4 We have revised the limitations section to include a more explicit discussion of potential failure modes beyond the evaluated tasks. For specific failure cases, please also refer to our response to Reviewer mnKA-R3.

Robustness to noisy or outdated information

R5 Our design does incorporate measures to be robust to bad information. During RAP, the agent cross-verifies information by retrieving from multiple sources against user's requirements before deriving insight knowledge, and the multi-stage verification will catch issues if a retrieved piece of instruction leads to an error (since the code won’t execute or will produce a poor result, prompting a fix). As the inclusion of outdated information is highly unlikely due to retrieval constraints, we focused our evaluation on robustness to noisy information. We simulated two scenarios:

• Pre-Summary Injection: Injecting unrelated or low-quality examples before insight extraction and planning.
• Post-Summary Injection: Injecting noisy examples after insight extraction but before planning, i.e., mixing noisy inputs with the useful insights.

Given the user requirements, these noises are generated by an extra agent (i.e., adversarial agent) prompted to create 'unhelpful' and 'fake' insights that hinder the planning process.

Our results show that AutoML-Agent is robust to such noise. Its built-in error correction and multi-stage verification mechanisms significantly mitigate the impact of noisy inputs, ensuring that the final model performance remains largely unaffected. Thanks to this suggestion, we observed that noise injection can even lead to better performance in particular cases. We conjecture that this may be because the Agent Manager is implicitly forced to further distinguish between useful and non-useful information.

Due to the space limit, the generated plans can be found at /example_plans/plan_with_noises.md in the anonymous GitHub.

Adaptability to other ML tasks (e.g., RL or RecSys)

R6 Please kindly refer to Reviewer Son5-R5.

We thank the reviewer for the positive comments on the novelty, motivation, and empirical rigor of our work. We are glad to address your concerns below.

Method complexity

R1 Thank you for pointing this out. In the final version, which permits an extra page, we will move key details currently located in the appendix, such as pseudocode summaries, into the main text. We hope these changes will make the framework easier to understand for a broader audience.

Is each agent important? How do these workers contribute to the final score?

R2 Yes, each agent is essential. Without even one of them, the entire framework would not function properly, considering the current implementation. This is due to a tight coupling between each agent, integrated tool use, and its corresponding step in the pipeline at the code level for stable interoperability. As a result, removing any single agent would cause the system to fail at runtime.

Specifically, the Prompt Agent ensures that user instructions are correctly interpreted and converted into structured input, enabling reliable downstream planning and execution. The Data Agent performs crucial data-related tasks that inform model search with domain-aware characteristics, directly enhancing model quality. The Model Agent drives performance by executing training-free model search, hyperparameter tuning, and profiling to select optimal candidates, thereby raising the normalized performance score. The Operation Agent ensures that high-performing models are correctly implemented and executable, completing the pipeline with valid, deployable code—vital for success rate metrics. Finally, the Agent Manager orchestrates the entire process, ensuring that all steps execute in the correct sequence and that each agent’s output is validated before proceeding, thereby enforcing the correctness of the entire pipeline.

What are the errors or detailed analyses of these workers?

R3 In this submission, we provide intermediate outputs at each stage ($\S$E) to illustrate what each agent produces—for example, the parsed JSON from the Prompt Agent and outputs from the Data and Model Agents.

Building on this, we will expand our analysis to include typical failure modes. For instance, the Data Agent may mishandle uncommon data formats (e.g., failing to impute missing values or referencing incorrect column names), while the Model Agent may select suboptimal models under constrained search spaces. Our multi-stage verification process detects and corrects most of these issues. As shown in $\S4.3$, certain variants fail to produce runnable code without specific modules, whereas the full system identifies and fixes such errors via feedback. The Agent Manager flags plans that violate user constraints (e.g., low accuracy), while implementation verification catches runtime errors (e.g., incomplete code or incorrect references) from the Operation Agent and requests fixes.

Notably, we also encountered consistent challenges with smaller or less capable models. As discussed in $\S$B.4, models like LLaMA-2-7B, Qwen1.5-14B, and even GPT3.5 often failed at complex planning or executable code generation. Common issues included incomplete code, altered comments without proper continuation, or unchanged code outputs. These patterns align with prior findings in DS-Agent and Data Interpreter, suggesting such limitations are systemic to smaller models rather than specific to our framework. We aim to reassure the reviewer that we have carefully studied “how it makes errors” and that the final results are reliable.

How did you design the prompt for each agent?

R4 We deliberately design prompts to optimize agent behavior with specific instructional steps based on role:

• Creative or interpretive agents (Agent Manager, Prompt Agent) benefit from low-pressure, neutral framing to avoid overconfidence. The Prompt Agent adopts a neutral “assistant project manager” persona to support coordination and interpretation.
• Execution-oriented agents (Data, Model, Operation Agents) respond better to strong, authoritative personas that promote precision and task adherence. The Data Agent is framed as “the world’s best data scientist” to encourage confident, detailed analysis and emphasize responsibilities.

These specialized prompts guide the LLM toward more accurate outputs compared to using a single, general-purpose prompt. Persona-driven system prompts reliably shape agent behavior. As suggested by Reviewer L7kx, we tested five prompt variations for each agent, differing in tone and task specificity. Due to space limitations, the experimental results with example outputs can be found at /example_plans/prompt_sensitivity.md in our anonymous GitHub repository. Overall, agents are not highly sensitive to exact phrasing as long as their roles were clearly defined, which is also reinforced through the user prompts—making the framework robust to variations in system prompts.