Automated Hypothesis Validation with Agentic Sequential Falsifications

We thank the reviewer for their positive feedback! We respond to the specific comments below:

“Can authors conduct analysis on how POPPER outperforms the baselines?”

We appreciate the reviewer’s insightful suggestion. Following the reviewer’s recommendation, we conducted a detailed manual examination and identified three main reasons for POPPER's superior performance:

1. The sequential experiments employed by POPPER significantly improves power over methods like CodeGen, ReAct, and Self-Refine, which uses only 1-2 experiments. In particular, many of the hypotheses cannot be directly observed, and therefore a direct attempt at validating the hypothesis might often fail, whereas Popper allows a sequence of carefully designed implication tests that can find more meaningful results.

2. Self-refine and relevance checker also help refine the experiment design, whereas the CodeGen, ReAct, and Self-Refine uses the most obvious experiment, which could contain bias and lacks rigor.

3. With its rigorous e-value-based approach, POPPER addresses the sequential dependencies of the falsification experiments and safely aggregates p-values from multiple experiments. This enables POPPER to achieve better Type I error control compared to the Fisher combined test and the LLM-likelihood. The Fisher combined test is not well-calibrated, and LLM-estimated scores often exhibit bias.

We will add these insights to our revised manuscript.

“The design agent is presented with the details of how to conduct the experiment in a given domain. Can authors elaborate on the extent of such assistance provided to the agent?”

Thank you for this important clarification request. To clarify, we did not provide explicit experimental details or instructions to the agent for each domain. The only domain-specific information provided was: “You are an expert specialized in the field of {domain},” as outlined in Supplementary Notes Listing 2. Consequently, the LLM agent designs experiments solely based on its internal world knowledge without additional domain-specific guidance.

We thank the reviewer for their positive feedback! We address each point in detail below:

“The claim "POPPER compares with human experts" should be appropriately caveated with limitations.”

We appreciate the reviewer highlighting this important point. We agree and will explicitly note the caveats related to the small sample size and differences from real-world complex settings in the revised manuscript.

“The authors may consider including literature on LLMs for automated ML experimentation, such as RE-Bench (Wijk et al.) and MLE-bench (Chan et al.).”

We thank the reviewer for suggesting additional relevant references. We will incorporate these references in our revised version.

“The main paper sounds like POPPER can be treated as a "solve all" method for automating science, but as I understand from the appendix, there are important caveats to be mindful of.”

We thank the reviewer for this valuable feedback. We agree that clearly presenting the limitations is crucial to ensure responsible usage and prevent potential misuse. We will add detailed discussions on failure modes and Type I error considerations in the main text in the revised manuscript.

We greatly appreciate the reviewer's thoughtful feedback and acknowledgment of our work's value in falsifying natural-language hypotheses. Below, we respond in detail to the specific points raised:

"Proposed framework claims to design and execute any type of experiments (laboratory, simulations, data analyses)"

We appreciate the reviewer highlighting this point. Our theoretical framework is indeed valid across various experiment types. Since for wet-lab experiments, the data is collected on the fly, so it naturally satisfy the assumption stated in Section 2. We thus emphasize the broad scope of our framework. However, due to practical constraints (e.g., cost and time), we instantiated our approach specifically through data analysis experiments in our large-scale evaluation. We will revise the text to explicitly acknowledge this practical limitation.

"Power determined analytically or via Monte Carlo simulations that converge toward true power values"

Thank you for raising this point. Currently, our power analysis is conducted using Monte Carlo simulations with five random seeds, and we reported the mean and standard deviations in our result tables. While we acknowledge that additional runs could further reduce variance and improve convergence toward the true power, scaling up significantly is not feasible due to computational constraints.

"Disagreement with exclusion of methods lacking proper Type I error control"

We respectfully clarify that since methods lacking proper Type I error control can inflate false positives, including them in power analysis would be an unfair comparison. We also stress the importance of valid error control as the first-order criterion. We welcome additional elaboration from the reviewer on the disagreement and are eager to discuss this further.

"Concern about novelty regarding the theorem presented"

As explicitly stated in our original manuscript, “Theorem 4 is a standard result following Grunwald et al. (2020), included in Appendix A.2 for completeness.” Our novelty claim does not rest on the safe testing framework itself but on leveraging this framework to instantiate a sequential falsification framework to enable practical and rigorous validation of abstract, free-form hypotheses in LLM-driven experiments. We will clarify this distinction explicitly in the revision.

"Extent to which evaluated hypotheses might be present in LLM pre-training data is unclear"

We appreciate this insightful comment. Our validation approach strictly relies on aggregated e-values derived from statistical analyses grounded in data. In our experiments, as detailed in section 4, we used data permutations for controlling Type I error, which ensures the resulting data is independent of the pre-training data. This experiment setup enforces that any discovery must be purely data-driven and not reliant on the agent's prior knowledge. We will clarify this critical detail in the revision.

"Major weakness is substantial overclaiming"

We thank the reviewer for this important feedback. Although we initially aimed to mitigate overclaiming by carefully delineating assumptions and providing comprehensive failure-mode analyses, we recognize the need for further clarity. In the revision, we will rigorously address ambiguous claims and explicitly highlight caveats to avoid potential misunderstandings. In particular, we will emphasize that the error control relies heavily on our assumptions, and several design components, such as relevance checker and using LLMs with strong capabilities, are intended to make sure the assumptions are satisfied. In practice, users should be careful in judging whether their system is sufficiently powerful to obey these assumptions.

"Intrinsic analyses of individual components to determine sources of failure would be valuable"

Thank you for raising this important point. In section 4.2 and Supplementary Section G, we conducted human annotation on the quality of falsification experiment subhypotheses and the relevance checker’s performance. In Supplementary Section D, we conducted an extensive intrinsic failure-mode analysis by examining detailed logs from 128 failure cases, categorized into 10 distinct failure modes. Additionally, we performed trajectory analysis documented in Supplementary Section E. We will ensure this comprehensive analysis is more prominently highlighted in the manuscript.

"Distinction between hypothesis validation and hypothesis testing?"

We appreciate this clarifying question. We used the terms interchangeably, recognizing that "hypothesis validation" tends to resonate more within scientific domains, whereas "hypothesis testing" is predominantly used in statistical contexts.

"Clarify the two power lines in Figure 4(2)"

The upper line represents statistical power, and the lower line represents the Type I error rate, both plotted against the number of maximum tests conducted.

We sincerely thank the reviewer for the constructive feedback and for recognizing our efforts to introduce rigor in the context of LLMs as bold and commendable. We address the thoughtful suggestions raised by the reviewer in detail below:

"Tune down the claim and specify the assumptions"

We appreciate this suggestion and fully agree. We have explicitly highlighted all underlying assumptions in Section 2, discussed Popper's reliance on the base LLM's reasoning capabilities in Section 4, and addressed limitations and failure cases through detailed error analysis in the appendices of our initial submission. We will further clarify ambiguous statements and ensure all claims are appropriately moderated in the revised manuscript. In particular, we will emphasize that the error control relies heavily on our assumptions, and several design components, such as relevance checker and using LLMs with strong capabilities, are intended to make sure the assumptions are satisfied. In practice, users should be careful in judging whether their system is sufficiently powerful to obey these assumptions.

Prompts in the provided code are indicative of challenges in using LLMs to process outputs of numerical methods.

We appreciate the detailed feedback. We acknowledge that p-hacking and misreporting p-values were indeed an issue in the initial experiments, especially when the base model is weaker. However, with the additional self-refine and other prompting mechanisms, we were able to consistently control the type I error rate with Claude 3.5 Sonnet. We discussed the performance variations across different backbone LLMs in section 4.1.

"Assess more than 30 examples of hypothesis validation failure modes"

Thank you for highlighting this aspect. To clarify, we initially analyzed 20 failed experiments to derive 10 distinct failure-mode categories. Subsequently, we expanded this analysis using a comprehensive set of 128 failure cases collected from benchmark runs across TargetVal-IFNG, TargetVal-IL2, and DiscoveryBench, as stated in Appendix D. Indeed, these include all the failure cases from one run of our experiments. Therefore, our analysis already provides extensive coverage, which we will clearly emphasize in the revision.

"Run a synthetic experiment with known ground truth and simulated expression data"

We thank the reviewer for this insightful suggestion. We would like to clarify that we indeed design the experiment setup for Type I error rate estimation to precisely address this concern. We simulated a null scenario by permuting rows of the dataset (e.g., shuffling gene names or expression values), thereby disrupting any real associations between variables. After permutation, all hypotheses become null—including those that may have been true positives in the original data—ensuring that our evaluation of the type-I error is faithful. Under this known null ground truth condition, POPPER consistently refrained from rejecting most null hypotheses, effectively controlling the Type I error rate. We will clarify this explicitly in the revised manuscript.