Response to Reviewer 875E

Discussion of Novelty Compared to Existing Methods

While autonomous scientific discovery has been explored by several existing frameworks as detailed in Section A.11.2, our AI-Researcher introduces several key innovations that distinguish it from prior work. Unlike existing systems that focus on specific components of the research pipeline, our framework provides the first truly end-to-end automated scientific research system capable of generating publication-ready papers from initial concepts without human intervention. While AI Scientist pioneered automated research idea generation and experimentation, and CycleResearcher demonstrated open-source LLM viability, these systems lack the comprehensive integration and systematic evaluation framework that our work provides.

Our key contributions include: (1) A complete end-to-end automation pipeline that produces publication-quality manuscripts, surpassing the partial automation offered by existing frameworks; (2) Scientist-Bench, a systematic evaluation framework containing 22 research papers across multiple domains, enabling standardized assessment of autonomous research capabilities at both guided (Level-1) and open-ended (Level-2) task levels; (3) An innovative Code-Advisor Agent iterative optimization paradigm that significantly improves algorithm implementation accuracy through structured feedback cycles, addressing a critical limitation in existing autonomous research systems. These innovations collectively enable more reliable and comprehensive autonomous scientific discovery compared to current approaches.

Comparison with AI Scientist Frameworks and Different LLM Backbones

Thank you for pointing out these important missing comparisons. To address both concerns, we conducted additional experiments comparing our AI-Researcher system against existing frameworks and with different LLM backbones.

Different LLM Backbones: We evaluated our framework using different LLM series on Level-1 tasks in the vector quantization domain. As shown in Table 1, our system achieves best performance with Claude-series (claude-3-5-sonnet-20241022 + claude-3-5-haiku-20241022, 3.22), followed by Gemini-series (gemini-2.5-pro + gemini-2.5-flash, 3.06), DeepSeek-series (deepseek-r1-0528 + deepseek-v3-0324, 2.39), and 4o-series (gpt-4o-2024-08-06 + gpt-4o-mini-2024-07-18, 2.0). Gemini-series occasionally encounters agent loop stagnation issues, while DeepSeek-series and 4o-series suffer from implementation hallucinations where agents incorrectly believe they have successfully implemented solutions.

AI Scientist Framework Comparison: Our AI-Researcher system (using Claude-series) significantly outperforms the AI-Scientist framework (using Claude-series) (2.89 correctness score) on the same tasks. This superior performance stems from our explicit code-advisor agent paradigm, which enables reflection and iterative correction in each loop, substantially improving implementation accuracy compared to systems without structured feedback mechanisms.

These results demonstrate both the robustness of our framework across different foundation models and its superiority over existing autonomous research systems.

Evaluation Protocol and Score Interpretation

The evaluation dimensions (research motivation, methodology, innovation, and experimental validation) are embedded within our ICLR-based review guidelines that guide the LLM review agents' assessment process. The evaluation is conducted entirely through LLM-based scoring, where review agents perform pairwise comparisons between AI-generated and human papers using these guidelines. The numerical scores represent comparative ratings on a 7-point scale (-3 to +3), where negative values indicate AI papers are inferior to human papers, zero indicates equivalence, and positive values indicate superiority. Each score reflects the aggregated assessment across all four evaluation dimensions rather than separate scoring per dimension.

Response to Reviewer 875E's Concerns

Thank you very much for your thoughtful feedback and for indicating your willingness to raise your score based on our responses. We greatly appreciate your constructive engagement and have carefully addressed each of your three specific concerns below. We believe our detailed responses demonstrate the novelty, rigor, and validity of our approach, and we hope they will support your consideration of a score revision.

Clarification of Novelty Compared to Existing Methods

Thank you for this question. We acknowledge that both AI-Scientist and CycleResearcher represent important foundational contributions to automated research. Building upon these valuable works, our core methodological novelty lies in three key areas:

1. True End-to-End Automation vs. CycleResearcher: CycleResearcher explicitly states that "human researchers or other agents are still needed to execute the experiments" and that "all experimental results mentioned in the generated papers were fabricated (Hallucinated)." In contrast, AI-Researcher achieves complete automation with real, verifiable experimental results.

2. Bidirectional Code-Theory Refinement vs. AI-Scientist: Our novel Code Advisor Agent Loop introduces recursive refinement between theoretical concepts and implementations through bidirectional mappings. Unlike AI-Scientist's linear pipeline, our system continuously validates mathematical formulations against code implementations, creating a feedback-driven optimization process that dramatically reduces implementation errors.

3. Atomic Resource Decomposition: Our Resource Analyst agents decompose complex research concepts into atomic components with explicit traceability, enabling modular verification and significantly reducing hallucination risks compared to existing monolithic approaches.

These innovations collectively enable the first system to conduct complete, verifiable scientific research with human-level rigor while maintaining full automation throughout the discovery lifecycle.

Comparison with AI Scientist Frameworks Using Standardized Review Agents

Thank you for requesting this more direct comparison using our review agent evaluation framework. We conducted additional experiments comparing our AI-Researcher system with the AI-Scientist framework using GPT-4o as the judge agent, applying the same Evaluation Protocols in section A.8 Experimental Settings.

As shown in Table, our AI-Researcher system demonstrates superior performance across both evaluation metrics:

• Average Rating: AI-Researcher achieves -0.55±1.00 compared to AI-Scientist's -0.92±1.70, indicating significantly better paper quality with lower variance

• Comparable Rate: 83.33% of AI-Researcher papers achieve comparable quality to human papers, substantially outperforming AI-Scientist's 61.46%

This direct comparison validates our earlier analysis of key innovations. The superior performance stems from our comprehensive design choices: (1) The Code-Advisor Agent iterative optimization paradigm provides structured feedback cycles that significantly improve implementation accuracy—a critical weakness in existing systems; (2) Our end-to-end integration enables better coherence between idea generation, implementation, and documentation phases; (3) The systematic evaluation framework (Scientist-Bench) ensures more reliable assessment of autonomous research capabilities.

These results demonstrate that our innovations translate into measurable improvements in research output quality, with both higher average ratings and greater consistency in producing human-comparable research papers compared to existing state-of-the-art autonomous research frameworks.

Response to Reviewer 6z2Z

Thank you for your valuable feedback and thoughtful suggestions. We have carefully addressed your comments and made the corresponding revisions. We sincerely hope our responses have resolved your concerns, and would appreciate it if you could consider updating your rating in light of these changes.

Comparison with AI Scientist Frameworks, Different LLM Backbones, and Ablation Studies

Thank you for pointing out these important missing comparisons. To address these concerns, we conducted additional experiments comparing our AI-Researcher system against existing frameworks, with different LLM backbones, and comprehensive ablation studies.

Different LLM Backbones: We evaluated our framework using different LLM series on Level-1 tasks in the vector quantization domain. As shown in Table 1, our system achieves best performance with Claude-series (claude-3-5-sonnet-20241022 + claude-3-5-haiku-20241022, 3.22), followed by Gemini-series (gemini-2.5-pro + gemini-2.5-flash, 3.06), DeepSeek-series (deepseek-r1-0528 + deepseek-v3-0324, 2.39), and 4o-series (gpt-4o-2024-08-06 + gpt-4o-mini-2024-07-18, 2.0). Gemini-series occasionally encounters agent loop stagnation issues, while DeepSeek-series and 4o-series suffer from implementation hallucinations.

AI Scientist Framework Comparison: We selected AI Scientist [1] as our primary comparison baseline because it represents another end-to-end automated scientific research system, making it the most comparable framework to our approach. Due to time constraints, we focused on this most relevant comparison. Our AI-Researcher system (using Claude-series) significantly outperforms the AI-Scientist framework (2.89 correctness score) on the same tasks, demonstrating the effectiveness of our structured feedback mechanisms.

Ablation Studies: Leave-one-out experiments reveal each component's contribution: removing the Resource-Analyst slightly reduces performance to 3.06, while removing the Advisor significantly drops performance to 2.94. This confirms that both components are essential, with the Advisor being particularly critical for implementation quality.

These results demonstrate the robustness of our framework across different LLM backbones, superiority over existing systems, and the necessity of each architectural component.

[1]. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024). The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292.

Human Expert Validation and LLM Judge Reliability

Thank you for raising this critical concern about the lack of human expert evaluation and potential LLM judge hallucination. We acknowledge that relying solely on LLM-based evaluation without human validation poses significant limitations to our assessment credibility.

To address this concern, we conducted a comprehensive human expert evaluation as suggested. Given that each expert needed to carefully compare 7 paper pairs (14 papers total) and perform detailed pairwise ratings, this process was extremely time-consuming and resource-intensive, leading us to recruit 7 domain experts for this validation study. Each expert performed full-text reviews following identical ICLR review guidelines used by our AI evaluators, with all papers fully anonymized and randomly ordered to eliminate any human bias in the evaluation process.

This rigorous human evaluation yielded an average score of -0.816, which closely aligns with our LLM review agents' decisions, providing strong validation that our LLM judges are not hallucinating quality assessments. The consistency between human expert scores and automated evaluation demonstrates the reliability of our evaluation framework.

While this represents a pilot study rather than a full-scale human evaluation, it provides crucial calibration for our LLM-based scoring system and confirms that our automated methodology produces assessments consistent with real scientists' judgments. We will incorporate these human validation results into the main text to strengthen our evaluation credibility.

API and Time Cost Analysis

Thank you for requesting detailed cost analysis of our system's iteration budget. We conducted comprehensive cost and time measurements for our AI-Researcher system using Claude-series models (claude-3-5-sonnet-20241022 + claude-3-5-haiku-20241022) on Level-1 tasks in the vector quantization domain.

As shown in Table 1, our end-to-end research process requires an average API cost of 39.102 us dollars and 4.796 hours of wall-clock time per complete research cycle. Each individual Code-Advisor refinement loop consumes approximately 11.428 us dollars in API costs. Since our complete research cycle involves three Code-Advisor refinement loops, this accounts for the majority of our total API costs (34.284 us dollars out of 39.102 us dollars), with the remaining costs (4.818 us dollars) attributed to other system components such as literature analysis and experiment planning.

The Code-Advisor refinement loops dominate the cost structure (87.7% of total costs) primarily because only the Code Agent utilizes the more expensive claude-3-5-sonnet-20241022 model for its superior coding capabilities, while all other system components employ the more cost-effective claude-3-5-haiku-20241022 model. This design choice optimizes both performance and cost efficiency by allocating premium computational resources specifically to the most demanding implementation tasks.

The time cost of 4.796 hours encompasses both LLM inference response time and agent experimental execution time across all system components. These measurements validate the practical feasibility of our approach while demonstrating the computational investment required for high-quality iterative code refinement in autonomous scientific research.

Domain Coverage and Title Clarification

Thank you for raising these concerns. While we acknowledge some limitations, we respectfully present our perspective on these points.

Regarding domain coverage, while our current evaluation focuses on AI research, this choice reflects both practical considerations and the inherent complexity of the AI domain itself. Importantly, our framework is fundamentally designed around the universal scientific research pipeline—literature review, idea generation, experimental planning, and validation—which represents core processes consistent across all scientific disciplines. AI research encompasses diverse computational methodologies spanning machine learning, computer vision, and algorithmic innovation, demonstrating significant technical breadth. Moreover, computational research tasks form the core of modern scientific inquiry across many disciplines, making our framework's computational focus broadly relevant. Our system demonstrates clear improvements over existing approaches (outperforming AI-Scientist and other baselines), validating the effectiveness of our innovations even within this focused domain.

Regarding the title, while we believe our framework introduces meaningful architectural innovations (Code-Advisor iterative refinement, atomic component mapping, structured multi-agent coordination) that advance autonomous research capabilities, we appreciate the reviewer's perspective on potential overstatement. We acknowledge that scientific innovation often involves novel synthesis rather than entirely unprecedented discoveries. To address this concern and better reflect the scope of our contributions, we plan to rename our work to "AI-Researcher: Autonomous Scientific Innovation for AI Systems and Beyond," which more accurately captures both our current focus and future aspirations.

Future work will expand domain coverage to further validate our framework's broader applicability across scientific disciplines.

Response to Reviewer qqqG

Thank you very much for your insightful feedback and constructive suggestions. We have carefully considered your comments and made corresponding revisions and enhancements. We hope our responses have adequately addressed your concerns, and we would be grateful if you would consider revising your rating based on these updates.

LLM Evaluator Variability and Reliability

We thank the reviewer for raising this important concern about evaluator variability. We acknowledge that LLM evaluators do exhibit fluctuation in their assessments. To mitigate single LLM evaluator bias, we deliberately employed multiple LLM backbones with different architectures and training paradigms.

Additionally, Appendix A.9 demonstrates our review agent's reliability through evaluation on ICLR submissions with human expert scores, showing high consistency between our review agents and human expert reviewers.

To further validate our approach, we conducted an additional costly human evaluation where we recruited 7 domain experts who each performed full-text reviews of 7 paper pairs following identical ICLR guidelines used by our AI evaluators, with AI-generated and human papers randomly ordered to prevent bias. This comprehensive evaluation yielded an average score of -0.81632 that closely aligns with our LLM review agents' decisions, providing strong additional confidence in our evaluation framework's reliability.

Resource Limitations in Compute-Heavy Domains

You have astutely identified this issue. To prevent the agent from building excessively large models on particularly large datasets (such as ImageNet 512x512), we provide recommended datasets for different domains including CV and GNN that are manageable within our agent environment (single RTX 3090 GPU). Additionally, we believe that adapting to current environmental constraints is also a necessary capability that research agents should possess. This approach ensures that our framework can operate effectively within realistic computational boundaries while still demonstrating meaningful scientific discovery capabilities.

Comparison with Other Autonomous Research Frameworks

Thank you for this important question about comparisons with other autonomous frameworks. We provide a comprehensive discussion of related autonomous frameworks in Section A.11.1 AI Agent Systems, where we categorize existing frameworks into three paradigms: Tool Integration Frameworks (LangChain, HuggingGPT), Multi-Agent Collaboration Frameworks (MetaGPT, CAMEL, AutoGen), and Self-Directed Agentic Task Execution Systems (AutoAgent).

Regarding generality, innovation capacity, and end-to-end automation, AI-Researcher distinguishes itself in several key ways: First, in terms of end-to-end automation, our framework provides complete autonomous scientific research capabilities from literature analysis through implementation to scholarly documentation, whereas existing frameworks focus on isolated capabilities or general problem-solving tasks. Second, regarding innovation capacity, our system is specifically designed for scientific discovery with specialized components for idea generation, theoretical-practical mapping, and rigorous validation—capabilities that transcend current general-purpose agent systems. Third, in terms of generality, while initially demonstrated in AI research, our framework's methodology is designed to extend to other scientific domains as the underlying research lifecycle (literature review, ideation, implementation, experimentation) remains consistent across scientific disciplines.

As noted in our related work analysis, existing frameworks fundamentally lack the intellectual capacity for true scientific innovation, as scientific breakthroughs demand nuanced hypothesis formation, creative experimental design, and critical knowledge synthesis—cognitive processes requiring deeper reasoning than current general-purpose systems provide.

Response to Reviewer S8cY

We sincerely appreciate the thorough feedback and valuable suggestions you have provided! We have carefully considered your input and made the necessary updates and modifications accordingly. We genuinely hope that our responses have addressed your concerns and that you would consider revising your rating score based on the changes we have made.

Github Link Issue

We sincerely apologize for the inadvertent error with the anonymous GitHub link provided in our submission.We have now corrected this issue and ensured that the link directs to the appropriate codebase. We respectfully ask that you please review the updated repository link.

Unclear Description of Scientist-Bench Dataset

We apologize for the unclear presentation of Scientist-Bench in the main text. We will move key details from Section A.7 into the main paper, including: (1) the systematic paper selection methodology across 16 AI domains, (2) a brief summary of the five-step reference extraction process identifying 15-20 influential papers per target, (3) the anonymization protocol preventing memorization shortcuts, and (4) the two-stage evaluation framework with multiple LLM evaluators. This will replace the brief description with comprehensive methodology explaining how Scientist-Bench enables standardized assessment across Level-1 (guided) and Level-2 (open-ended) tasks.

Title Ambiguity

Thank you for this excellent question. First, our system is indeed capable of serving as a generalist agent for scientific discovery. This is because research across different scientific domains follows a common iterative cycle of literature review, idea generation, planning, and experimentation. More specifically, our current system can handle computational tasks in other biomedical fields, such as data analysis in biomedical research, though it does not include hands-on experimental tasks. As an initial study, we chose the AI domain for validation because of our greater familiarity with this field. Moving forward, we plan to extend Scientist-Bench to other scientific domains. However, we apologize for the potential ambiguity in the title and plan to rename it to "AI-Researcher: Autonomous Scientific Innovation for AI Systems and Beyond." We hope this revision will eliminate such ambiguity.

Insufficient Technical Details

Thank you for pointing out this clarity issue. The "Challenges," along with "Existing Methods," "Motivation," and other items listed, are the required components that the Idea Generator must include in the final idea proposal. We acknowledge that the presentation format may have caused this misunderstanding, and we have revised the display format of these items for better clarity. Additionally, we gratefully accept your suggestion to move more technical details to the main paper. We have incorporated more experimental data and condensed prompts into the main text to enhance the technical grounding of our work.

Missing experimental description

We appreciate the reviewer's excellent suggestion to improve the clarity of our evaluation approach. We will add the following description to the introduction after presenting our contributions: "We evaluate AI-Researcher through comprehensive experiments on our Scientist-Bench benchmark, including pairwise comparisons between AI-generated and human-authored papers using multiple LLM judges following ICLR review guidelines, technical implementation validation through specialized code review agents, and performance analysis across both guided innovation tasks and open-ended research exploration scenarios."

Curation Procedure Description

Thank you for your valuable suggestion. We agree that clear Experimental Settings can help readers better understand the experimental setup, particularly the content in Appendix A.8. We have incorporated this section into the main text and streamlined the high-level method descriptions to accommodate the inclusion of detailed Experimental Settings and comprehensive technical details (such as agent definitions, tools, etc.). Additionally, regarding your concern about the curation procedure, our benchmark includes a detailed curation process outlined in A.7.2 Benchmark Construction, which consists of Step 1: Systematic Selection of AI Research Benchmark Papers, Step 2: Input Construction for AI-Researcher, and Step 3: Rigorous Anonymization to Ensure Scientific Originality. We will also summarize and incorporate this section into the main text to provide clearer experimental settings.

Scoring criteria issue

We thank the reviewer for this important clarification. Our -1.0 threshold is a reasonable and conference-aligned setting for several reasons: First, our benchmark comprises exclusively high-quality papers accepted at top-tier venues, making it extremely challenging to clearly exceed these carefully-selected contributions. Second, top conference review processes inherently involve borderline decisions where papers within narrow margins are considered comparable quality—borderline scores often determine acceptance at competitive venues like ICLR and NeurIPS. Following this established academic practice, we consider AI-generated papers scoring ≥-1.0 as achieving near-human quality since they fall within the borderline range of these exceptional benchmarks, where -1.0 represents the lower bound of papers that could potentially receive acceptance at top conferences.

We will add a clear description of the 7-point scale and our threshold rationale to the main text to improve clarity.

Human Expert Evaluation of AI-Generated Papers

Thank you for this insightful observation about Figure 9 and the suggestion for human expert evaluation. We acknowledge the limitations you identified in the example paper, such as missing error bars in experiments and limited citations in the introduction—features that indeed differ from typical human-generated research.

To address this concern and further validate our evaluation framework, we conducted an additional comprehensive human expert evaluation. Given that each expert needed to carefully review both AI-generated and human-written papers in detail, this process was extremely time-consuming. We recruited 7 domain experts who each performed full-text reviews of 7 randomly selected paper pairs (AI-generated vs. human-written), following identical ICLR review guidelines used by our AI evaluators. All papers were fully anonymized and randomly ordered to eliminate any human bias in the evaluation process. This rigorous human evaluation yielded an average score of -0.816, which closely aligns with our LLM review agents' decisions, providing strong additional confidence in our evaluation framework's reliability.

This human expert validation not only supports our automated evaluation methodology but also confirms that while AI-generated papers may exhibit certain stylistic differences (as noted in Figure 9), the overall quality assessment remains consistent between human experts and our AI evaluation system. We will incorporate discussion of these stylistic limitations and our human validation results into the main text.

Technical Clarifications

We have addressed the following technical detail concerns:

• Point 3: We have clarified that the scientific databases mentioned on Line 105 include arXiv**,** Google Scholar**,** GitHub**, HuggingFace**, and other general scientific databases used by the Knowledge Acquisition Agent for systematic literature exploration.

• Point 4, 14: We have moved the definitions of Level 1 and Level 2 tasks from Appendix A.8 to the main text. Level-1 Task (Guided Innovation) involves the agent receiving explicit research instructions alongside reference papers to develop targeted innovations, covering all 22 groundtruth papers. Level-2 Task (Autonomous Exploration) requires the agent to perform independent research exploration with only reference papers as input, testing its capacity to identify research gaps and generate novel directions without explicit guidance. We strategically selected 5 groundtruth papers across distinct research domains for this more challenging scenario.

These clarifications provide readers with the necessary technical context directly in the main paper.

Autonomous Literature Search Capability

We greatly appreciate your valuable suggestions regarding autonomous literature search capabilities in Question 2 and Limitations. Initially, we chose not to implement automatic agent-driven search in order to standardize the evaluation process for Scientist-Bench. However, inspired by excellent work such as [2] and others, we have internally developed autonomous research search agents as part of our system optimization. We will add the technical details and citations for this component. Specifically, we have provided the agent with all the workflow tools originally used for dataset construction and designed prompts to enable the agent to autonomously complete systematic literature reviews. This enhancement addresses the limitation you identified and moves toward a more fully autonomous scientific research system.

Reference:

[2] Skarlinski, M.D., Cox, S., Laurent, J.M., Braza, J.D., Hinks, M., Hammerling, M.J., Ponnapati, M., Rodriques, S.G. and White, A.D., 2024. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409.13740.

Further Response to Reviewer S8cY

We sincerely appreciate the thoughtful feedback and valuable suggestions you have provided. We have carefully addressed each of your concerns through comprehensive revisions and additional experiments. We genuinely hope that our responses have adequately addressed your concerns and that you would consider revising your rating based on the substantial improvements we have made.

Below are our detailed responses to your other feedback:

Writing and Formatting Issues

Thank you for identifying these concerns. We have systematically corrected all writing and formatting issues including citation formats, figure clarity, capitalization errors in references, typographical errors, and missing punctuation. All formatting inconsistencies have been resolved to improve manuscript presentation quality.

Missing References and Related Works

Thank you for pointing out the missing venue information for SciBench, highlighting important related works in agentic systems, and requesting clarification on the Claude models used. We have addressed all these points with comprehensive updates.

Specific updates made:

1. Added SciBench [Wang et al., 2024] (for Point 12) in A.11.2 evaluation frameworks paragraph: "These systems are supported by emerging research platforms and evaluation frameworks that enhance their capabilities and measure their effectiveness. SciBench Wang et al. [2024] provides a comprehensive benchmark for evaluating college-level scientific problem-solving abilities of large language models across mathematics, chemistry, and physics domains. Agent Laboratory Schmidgall et al. [2025] provides an end-to-end autonomous research workflow..."
2. Integrated the three agentic systems (for Point 17, 18, 19) into A.11.2 autonomous research frameworks paragraph: "Recent advances have transformed AI's role in scientific research from assistive tools to autonomous agents capable of executing complete research workflows. The AI Scientist framework Lu et al. [2024] pioneered this field as the first comprehensive system where frontier language models independently generate research ideas, conduct experiments, and produce scientific papers. Building upon this foundation, Curie Kon et al. [2025] introduces a framework specifically designed to embed rigor into automated scientific experimentation through systematic validation modules. Complementary approaches include CycleResearcher Weng et al. [2025], which demonstrated the viability of open-source LLMs for autonomous research, and the AI co-scientist AI Co-Scientist Team [2025], which employs multi-agent debate mechanics. For training and evaluation, Aviary Narayanan et al. [2024] provides an extensible platform for training language agents on challenging scientific tasks including molecular analysis and protein engineering. Robin Ghareeb et al. [2025] represents a multi-agent system capable of automating the entire scientific discovery process from literature review to experimental validation, with demonstrated applications in drug discovery."
3. Added GPTSwarm [Zhuge et al., 2024] (for Point 16) in A.11.1 Multi-Agent Collaboration Frameworks paragraph: "Multi-Agent Collaboration Frameworks. The second paradigm addresses complex problem solving through structured agent interactions. MetaGPT Hong et al. [2024] formalized human workflow patterns through Standardized Operating Procedures (SOPs), creating systematic collaboration protocols. AutoGen Microsoft AutoGen Team [2025] expanded this vision with a comprehensive programming framework for developing systems that support both autonomous operation and human collaboration. GPTSwarm Zhuge et al. [2024] introduces an innovative approach by modeling multi-agent systems as optimizable graphs with binary edge weights, enabling dynamic reconfiguration of agent collaboration patterns. AgentScope Gao et al. [2024] prioritized robust coordination through a message-exchange architecture with built-in fault tolerance. CAMEL introduced innovative role-playing techniques that facilitate autonomous agent cooperation while maintaining alignment with human..."
4. Clarified Claude model specifications and added system card citations: (for Point 6, 22) We have clarified that the Claude-series models refer specifically to claude-3-5-sonnet-20241022 and claude-3-5-haiku-20241022. We have added the system card citation from Anthropic's Claude 3.5 Model Card [Anthropic, 2024]. For cost considerations, we used claude-3-5-sonnet-20241022 exclusively for the Code Agent component due to its superior coding capabilities, while claude-3-5-haiku-20241022 was employed for all other system components to optimize computational efficiency. The complete model specifications and performance characteristics are documented in the respective system cards.

Response to Reviewer XKkH

Thank you sincerely for your detailed feedback and thoughtful suggestions. We've carefully reviewed your comments and made corresponding updates and improvements. We hope our responses have effectively addressed your concerns, and we would greatly appreciate it if you would consider updating your rating in light of these changes.

Figure Generation Automation

Yes, figure generation is fully automated in our system. We acknowledge the reviewer's concern about VLM evaluation of figures, but recent advances have significantly improved their capabilities for analyzing experimental visualizations.

Key points addressing this concern:

1. Advanced VLM capabilities: Modern models like Claude 3.5 Sonnet and Gemini 2.5 Pro have demonstrated substantial improvements in visual analysis and scientific figure understanding compared to earlier generations.
2. Iterative refinement architecture: Our Code Agent-Advisor Agent framework provides robust validation through multiple review cycles. The Code Agent generates figures, which are then systematically reviewed and critiqued by the Advisor Agent, enabling iterative refinement until quality standards are met.
3. Multi-round optimization: This iterative loop ensures that figure quality issues are identified and addressed through successive improvements, mitigating the limitations of single-pass VLM evaluation.

While VLMs may not achieve perfect human-level figure evaluation, the combination of advanced models and our iterative agent architecture provides a reliable mechanism for automated figure generation and validation.

Regarding the Knowledge Acquisition Agent's Autonomy

We greatly appreciate your question about the Knowledge Acquisition Agent's capabilities. Initially, we chose not to implement fully automatic agent-driven literature search in order to standardize the evaluation process for Scientist-Bench and ensure consistent experimental conditions across all evaluations. However, we recognize this as an important limitation and have internally developed autonomous research search agents as part of our system optimization. We will add the technical details for this component. Specifically, we have provided the agent with all the workflow tools originally used for dataset construction and designed prompts to enable the agent to autonomously gather 10-15 references and complete systematic literature reviews. This enhancement addresses the limitation you identified and moves toward a more fully autonomous scientific research system that can operate without requiring human input for the literature gathering step.

Regarding Unit Tests in Resource Agent

When the Resource Agent creates atomic components with mappings between math and code, unit tests are not included. The Resource Agent focuses on extracting and establishing conceptual connections between mathematical formulations and code implementations rather than executing code. This process is more about constructing relationships and mappings between theoretical concepts and their computational representations. After establishing multiple theory-code connections, the agent can form a subsequent implementation plan for practical execution. Since the Resource Agent operates at the conceptual mapping level rather than the code execution level, unit tests are not necessary at this stage.

Regarding Idea Generator Prerequisites

You are absolutely correct. The Idea Generator is indeed conditioned on the 10-15 references provided by human researchers, and we acknowledge that this was not explicitly stated in Section 3.1.2. We have added a clear explanation of the Idea Generator's prerequisites and dependencies in the revised version to ensure this important detail is properly communicated to readers. Thank you for pointing out this omission.

Clarification on Recursive Refinement Mechanism

Thank you for pointing out this unclear description. We have clarified this concept in the revised version. The "recursive refinement mechanism" refers to our system's iterative process where theoretical concepts and their practical implementations continuously inform and improve each other. Specifically, when our Resource Analyst agents create bidirectional mappings between mathematical formulations and code implementations, the system can refine theoretical understanding based on implementation results, while simultaneously using theoretical insights to improve code implementations. This process is further enhanced through iterative collaboration between the Code Agent and Advisor Agent, where the Code Agent's implementation attempts are continuously refined based on the Advisor Agent's feedback, and the Advisor Agent's guidance evolves based on observed implementation outcomes. This creates a multi-layered feedback loop where each iteration of theory-to-code, code-to-theory, and agent-to-agent refinement enhances all components, similar to how human researchers iteratively refine their theoretical models based on experimental results and mentor-student interactions. We have added a more detailed explanation with concrete examples to make this mechanism clearer for readers.

Clarification on Structured Feedback Cycles

Thank you for requesting clarification on "structured feedback cycles." The structured feedback refers specifically to the systematic format of feedback provided by the Advisor Agent to the Code Agent. This structured feedback includes: (1) assessment of implementation correctness, and (2) detailed analysis and experimental suggestions for improvement. Based on this structured feedback, the system determines whether to continue experimentation (if implementation is incorrect and requires further refinement based on the analysis and suggestions) or to proceed (if implementation is correct). This structured approach ensures consistent and actionable communication between agents, mirroring the systematic mentor-student feedback relationship in academic research. We have added references to specific examples in the appendix to illustrate these structured feedback formats and cycles.

• Line 166: "This approach increases implementation success rates with test-time scaling capabilities." -- Is this statement supported by empirical evidence? I couldn’t find any test-time scaling experiments in the paper.

Verification of Faithful Translation of Concepts into Code

Thank you for this important question about our verification mechanism. We ensure faithful translation of academic concepts into working code through our Expert Validation Framework, specifically via the Advisor Agent's systematic review process. The Advisor Agent carefully examines the Code Agent's implementations by systematically comparing the code against the atomic research ideas extracted during analysis. When the Code Agent's implementation contains errors or deviations from the theoretical concepts, the Advisor Agent can promptly identify these issues and provide specific, actionable modification recommendations. This iterative validation process continues until the implementation accurately reflects the intended academic concepts, ensuring fidelity between theory and code throughout the development cycle.

Regarding Specialized Navigation Tools

You are absolutely correct. The "specialized navigation tools" refer to the set of function calls defined in the appendix. We have added a reference to A.1 Definitions of Tools in the main paper. Specifically, these tools include gen_code_tree_structure, read_file, list_files, and other functions that enable the agent to navigate and read workspace file contents effectively. These tools allow the Advisor Agent to systematically examine implementations and provide comprehensive feedback based on direct analysis of the codebase.

Examples of "Unfeasible" Classification Failure Cases

Thank you for your curiosity about failure cases. We actually discuss some failure cases in Section 4.1 Dual-Metric Evaluation Framework: Quantifying Implementation Quality (RQ1) under "Performance Comparison between LLMs in Scientific Implementation." The examples include cases where models frequently generated code with persistent tensor dimension mismatches and training instabilities (NaN losses) that remained unresolved despite multiple debugging attempts. Additional failure cases include Out-of-Memory (OOM) errors when implementations involve models that are too large to complete within the allocated computational resources. These persistent issues, despite multiple refinement cycles, result in the "unfeasible" classification as the system recognizes that further attempts are unlikely to succeed under the given constraints.

Clarification on "Validation Studies"

Thank you for asking about the specific meaning of "validation studies." In this context, validation studies refer to supplementary experiments recommended by the Advisor Agent to verify the correctness and effectiveness of the implemented research concepts. These may include testing the implementation on different datasets, conducting ablation studies to validate specific components, or performing sensitivity analyses to ensure the implementation faithfully reproduces the intended research outcomes. These validation studies serve as additional verification steps beyond the initial prototype testing to comprehensively validate the scientific validity and robustness of the implemented algorithms.

Further Response to Reviewer XKkH

We sincerely thank the reviewer for the follow-up questions and for engaging deeply with our work. We greatly appreciate your continued interest and thoughtful observations. Below, we provide detailed responses to each of your points.

API and Time Cost Analysis

Thank you for requesting detailed cost analysis of our system's budget. We conducted comprehensive cost and time measurements for our AI-Researcher system using Claude-series models (claude-3-5-sonnet-20241022 + claude-3-5-haiku-20241022) on Level-1 tasks in the vector quantization domain.

As shown in Table 1, our end-to-end research process requires an average API cost of 39.102 us dollars and 4.796 hours of wall-clock time per complete research cycle. Each individual Code-Advisor refinement loop consumes approximately 11.428 us dollars in API costs. Since our complete research cycle involves three Code-Advisor refinement loops, this accounts for the majority of our total API costs (34.284 us dollars out of 39.102 us dollars), with the remaining costs (4.818 us dollars) attributed to other system components such as literature analysis and experiment planning.

The Code-Advisor refinement loops dominate the cost structure (87.7% of total costs) primarily because only the Code Agent utilizes the more expensive claude-3-5-sonnet-20241022 model for its superior coding capabilities, while all other system components employ the more cost-effective claude-3-5-haiku-20241022 model. This design choice optimizes both performance and cost efficiency by allocating premium computational resources specifically to the most demanding implementation tasks.

The time cost of 4.796 hours encompasses both LLM inference response time and agent experimental execution time across all system components. These measurements validate the practical feasibility of our approach while demonstrating the computational investment required for high-quality iterative code refinement in autonomous scientific research.

Test-Time Scaling Capabilities

We apologize for overlooking this important question about our test-time scaling claim. Thank you for pointing out the lack of empirical support for this statement.

To address this concern, we conducted additional experiments on test-time scaling using our AI-Researcher system with Claude-series models on Level-1 tasks in the vector quantization domain. We varied the number of Code-Advisor iterative loops as our scaling metric.

As shown in Table 1, our results demonstrate clear test-time scaling capabilities: performance increases progressively with more iterative refinement cycles, from 2.44 (1 iteration) to 2.89 (2 iterations) to 3.22 (3 iterations). This empirical evidence supports our claim that additional computational investment at test-time through iterative refinement cycles leads to measurably improved implementation quality.

This scaling behavior validates our architectural design where each additional Code-Advisor loop provides structured feedback and refinement opportunities, enabling the system to progressively correct implementation errors and improve code quality. The consistent performance gains across iterations demonstrate that our framework can effectively leverage additional test-time computation for enhanced research outcomes.

We will incorporate these test-time scaling results into the main paper to provide proper empirical support for our claims.

Distinguishing Execution Failures from Scientific Quality Issues

Thank you for the thoughtful follow-up. We appreciate your interest in how the system handles cases where the output, while executable, may not meet scientific or methodological expectations.

In our dual-metric evaluation framework (Section 4.1), we make a clear distinction between execution feasibility and implementation quality. Specifically, “Unfeasible” refers to cases where the agent fails to produce runnable code after multiple refinement cycles—commonly due to persistent issues such as tensor shape mismatches, NaN losses, or out-of-memory (OOM) errors.

In contrast, cases where the implementation completes successfully but exhibits scientific flaws, conceptual inconsistencies, or inadequate hypothesis validation fall under the Correctness dimension. These are not marked as “Unfeasible” because the task has technically been completed. Instead, the Advisor Agent analyzes such outcomes and flags potential weaknesses in reasoning or methodology, and the Judge Agent reflects these issues in a lower correctness score on a 5-point scale.

In summary, while "Unfeasible" denotes hard execution failure, deeper quality issues—including but not limited to “failed hypotheses”—are systematically captured by our Correctness evaluation process.