Strategic Hypothesis Testing

Thank you for your review and insightful comments. We are glad that you found the problem interesting and well-motivated, and the results technically interesting. We hope our work can lead to further empirical and technical works exploring strategic behaviour for hypothesis testing. Below, we include a detailed response to your question:

---

Uncertainties about cost and revenue parameters:

Indeed, both the agent participation decision and the principal’s optimal p-value lower-bound $\hat{\alpha}$ depend on the cost and revenue parameters; in fact, these are the only parameters required for this decision. In drug development, past data on trials of similar drugs/genres can be used to estimate costs, whereas projected revenue must be computed for any business decision prior to development and trials. That said, you are correct that in practice, these parameters can only be calculated approximately/with some error. Below is a way to extend our work to such scenarios.

In the most general case, we can consider the agent having access to parameters $\hat{c_0}, \hat{c}, \hat{R}$ that are within $\epsilon$ of the true values (note that epsilon can be different for each of the three; this is just for ease of exposition); similarly, the principal has access to $\tilde{c_0}, \tilde{c}, \tilde{R}$ that are within $\delta$ of the true values.

1. Agent-side approximation error: We can first ask how different (due to $\epsilon$ error) the agent's decision would be upon observing the effectiveness $\mu_0$. In generalizing Theorem 3.1, we note that the analysis depends on the properties of $u(n; \alpha)$ and that it can be partitioned by indices $n_1, n_2$ such that within each partition, the function is convex or concave. As we see in equation 8 in Appendix B, the indices $n_1, n_2$ are not affected by $\epsilon$; neither is the decision in the convex region since the minimum is at the endpoint. For the concave region, the optimal corresponds to the x-intercept of $\partial u/\partial n$. The error $\epsilon$ will change the value of the utility function derivative by $O(\epsilon)$; thus the change in x-intercept (and thus the optimal choice of $n$) will shift by at most $O(\epsilon)/L$, where $L$ is the inverse Lipschitz constant of $\partial u/\partial n$ (by inverse lipschitz constant, we mean $|f(x_1) - f(x_2)| \geq L|x_1 - x_2|$, allowing us to bound change in the domain given change in range). That is, $|\hat{n} (\alpha; \mu_0) - n(\alpha; \mu_0)| \leq \frac{O(\epsilon)}{L}$. For the function $\partial u/\partial n$ specifically, this inverse lipschitz constant can be expressed in terms of $n_1, n_2, n_{min}, n_{max}$.

Uncertainty about cost and revenue also affects the participation threshold $\mu_\tau(\alpha)$, defined as the minimum belief where an agent will participate for a given p-value $\alpha$. Again, since the utility deviates by $O(\epsilon)$, the corresponding change in threshold belief can again be bounded by the inverse lipschitz constant $k$ (in terms of $\mu_0$) of the function $u(\mu_0, n(\alpha, \mu_0))$. In short, this approach can bound $\mu_{\tau}(\alpha)$ and $\hat{\mu}_{\tau}(\alpha)$.

2. Principal side approximation error: As for the principal, their perception of the cost/revenue, $\tilde{c_0}, \tilde{c}, \tilde{R}$, is different than the agents. So for any $\alpha$, they will perceive a threshold belief $\tilde{\mu}_{\tau}(\alpha)$. As the principal does not know the agent's error $\epsilon$, they can do no better than follow the conclusions of section 4 and choose a p-value of at least $\hat{\alpha}$ such that

$\tilde{\mu}_{\tau}(\hat{\alpha}) = \mu_b.$

But under this $\hat{\alpha}$, since the agent's estimated parameter is different from the principal's estimated parameter ($\epsilon \neq \delta$), we know that $\hat{\mu}_{\tau}(\hat{\alpha}) \neq \mu_b$, which means that the agent will not behave exactly as the principal expects.

However, it is possible to compute how far away $\hat{\mu}_{\tau}(\hat{\alpha})$ is from $\mu_b$ by using the inverse Lipschitz constant $k$ of function $u(\mu_0, n(\alpha, \mu_0))$ mentioned above.

Importantly, note that the principal's uncertainty error depends only on how far their cost/revenue estimate $\tilde{c_0}, \tilde{c}, \tilde{R}$ was from the agent's estimate $\hat{c_0}, \hat{c}, \hat{R}$ (and not the true value), since this is what the agent uses to make their decision. The agent's uncertainty error, on the other hand, only depends on how far their estimate $\hat{c_0}, \hat{c}, \hat{R}$ was from the true values $c_0, c, R$. One can analyze them separately.

Thank you for your thoughtful review and comments. We are glad you found the work well-motivated and interesting. Indeed, given the breadth and scope of hypothesis testing and statistical decision-making in the wild (e.g., food, drugs, manufacturing), we hope this work opens up a broader research direction on designing optimal decision rules that account for agent incentives and strategic manipulation. Please see below detailed answers to questions/comments in the review:

---

Regarding the optimal p-value:

You are right that the value $\hat{\alpha}$ we rigorously characterize is not the global optimum for the general problem; rather, it is a lower bound on the optimal p-value threshold $\hat{\alpha} \: \alpha^* \geq \hat{\alpha}$.

1. Practicality: In most settings (like drug approvals), the natural desire is to choose a low p-value as this bounds the false positive rate. As a result, bounding the optimal p-value from below may be sufficient in practice, as regulators/principals are generally adverse to higher p-values. Moreover, since our problem is always constant-dimensional (effectiveness is a scalar), if computing the exact $\alpha^*$ is desired, this can be easily done empirically, by plotting the cumulative loss and noting the minimum (see Figure 1 for example). We include code for this computation.
2. Interpretability: As our analysis shows, $\hat{\alpha}$ dominates any p-value below it in the sense that both FN and FP are monotonically worse for such values. Above $\hat{\alpha}$, the relative strengths of FN and FP dictate this trade-off. Since the value of $\hat{\alpha}$ is agnostic to both the underlying distribution $q$ and the hyper-parameters $\lambda_{fn}$ and $\lambda_{fp}$ (parameters which may be noisy/heuristic driven), this is a much easier decision to justify. Note that the global optimal $\alpha^*$ will depend on all these parameters – a formal analysis of this would compare the derivatives $\frac{d}{d\alpha} \text{FN}(\alpha)$ and $\frac{d}{d\alpha} \text{FP}(\alpha)$ which depend on parameters (q, $\lambda_{fn}$, $\lambda_{fp}$).

In the revised version, we will highlight these operational advantages and interpretability of $\hat{\alpha}$ and comment on the properties and comparisons to $\alpha^*$.

---

Regarding the technical contribution:

Our goal at the onset was to propose a realistic and principled model of strategic interaction in the hypothesis testing framework. Obtaining clear and interpretable conclusions seemed difficult here; unlike standard settings, the principal must control both statistical error and agent participation using only the p-value threshold $\alpha$. The challenge is further compounded by the rich interplay among the prior, passing probabilities, sample sizes, and agents’ strategic behavior. Some of these relations cannot be explicitly characterized and are not smooth. Yet a sequence of subtle observations and technical insights results in a set of clean structural and practically relevant results. The clarity and easy interpretability of such results are, in our view, a key strength of the framework, especially since decision-makers in these contexts, such as regulators, must justify their choices to a broad audience.

Moreover, while the results are interpretable, the conclusions can be surprising. The fact that a lower bound on the optimal p-value emerges from our work is distinct from the classic thinking of upper-bounding $\alpha$ to control Type I errors. Our analysis is also flexible, highlighting that if context-dependent p-values are unpalatable, regulators can instead subsidize the fixed and marginal costs to achieve similar results. We will highlight the latter point more in the revision.

Thank you for your detailed review and comments, which gave us new insights into our contributions and setting! We will fix all typos in the revision; please see below a discussion on the more specific points raised:

---

Noisy and imperfect observations of distribution $q$ and effectiveness $\mu_0$

We presently assume that the effectiveness of a product (drug) $\mu_0 \sim q$, where $\mu_0$ is the privately observed effectiveness and $q$ is the originating distribution. $q$ can be seen as the historical performance of drugs in a given category, data that the FDA typically maintains. As for $\mu_0$, we interpret this as the evidence the manufacturer has based on internal testing and trials, which typically precede a public trial. You are absolutely right that there can be noise and uncertainties about these parameters in practice. However, we note a key strength of our work/model: neither the agent participation decision nor the p-value threshold $\hat{\alpha}$ requires access to q. Specifically,

1. Knowledge of q is not required by the agent: the agent observes $\mu_0 \sim q$, and then reacts (decides to participate and chooses samples if so) based on this realization.
2. Our core result, the optimal p-value lower bound $\hat{\alpha}$, is agnostic to the distribution $q$. This is of major operational importance since it means the principal also does not need to know $q$ to choose $\hat{\alpha}$. That said, $q$ does affect the loss incurred by the principal at this chosen p-value. We detail this below.

Quantify the impact of uncertainty due to estimation errors in $q$ on loss calculations

Computing the FN/FP loss at any chosen p-value, including $\hat{\alpha}$, requires access to $q$. Suppose the principal has an erroneous version of the distribution $q$ called $q_{\text{approx}}$. Then for any $\alpha$, we can bound the difference between the true FN/FP as calculated under $q$, and the approximate one calculated under $q_\text{approx}$ using TV distance. In particular, if distributions $q$ and $q_{\text{approx}}$​ differ by at most $\delta$ in TV distance, then for any bounded measurable function $f \in [0,1]$, we can show the difference in conditional expectations over any event with non-negligible probability (at least $\gamma$) is bounded by: $|E_{\mu \sim q}[f(\mu)|A] - E_{\mu \sim q_{approx}}[f(\mu)|A]|\leq \frac{2\delta}{\gamma}.$ Since $\gamma$ is purely from the agent's decision, it is unaffected by the noise level $\delta$. This allows us to quantify the change in FN and FP losses due to errors in $q$. In particular, both the FN and FP losses take the form of conditional expectations over events defined by thresholds (e.g.,$\mu_0 \leq \mu_b$​), multiplied by ratios of marginal probabilities. We can show that both terms change by at most $\frac{2\delta}{\gamma}$ if the total variation distance between $q$ and $q_{\mathrm{approx}}$​ is at most $\delta$, and the relevant conditioning events have probability at least $\gamma$ under both distributions. In short, we can give an approximation error bound of the form ($\text{Loss}$ can be either FP or FN): $|\text{Loss}(q) - \text{Loss}(q_{\text{approx}})| \leq \frac{4\delta}{\gamma}$. We will include the formal proof of this result and a discussion on the effect of noisy/imperfect observations in the revised version.

Agent's imperfect observation of $\mu_0$

As for the agent (drug manufacturer), suppose that instead of exactly observing $\mu_0$, they have some unbiased estimator of it, $\hat{\mu}$. In extending our results to this unbiased estimate setting, the agent participation decision is no longer a threshold/deterministic; instead of observing $\mu_0$ and taking the participation decision $1[\mu_0 \geq \mu_{\tau}(\alpha)]$, now they observe $\hat{\mu}$ and their participation decision is given by $P[\hat{\mu} > \mu_{\tau}(\alpha)]$ – a softer decision boundary. While key properties like monotonicity in $\mu_0$ still hold if the estimator, the analysis needs to be re-done with this change, and will now include parameters relating to the distribution of the estimator (like variance).

---

Interpreting our conclusion on cost and revenue dependent p-value threshold

Our results highlight that when agents strategically choose their trial samples and participation based on private beliefs and cost-revenue calculus, the p-value decision must also consider such dynamics. There are two ways to interpret this:

1. As pointed out in the review, one perspective is to set the p-value based on the expected revenue-cost calculus. This implies that for low-revenue high-development-cost drugs, the p-value threshold should be more generous; otherwise, it is not in the best interest of developers to participate. This can be interpreted as setting a domain/market-specific p-value.
2. In many cases, the first interpretation may not be palatable, and agencies may be reluctant to change p-values. In such cases, our model suggests that the principal can still derive optimal results against strategic agents by adjusting the other parameters. Specifically, we imagine regulators providing subsidies for development costs and trials to ensure optimal participation on the desired p-value. This is not far-fetched since governments often subsidize crucial drug developments (the COVID-19 vaccine, for example). From this perspective, our result can be interpreted as giving an accurate and interpretable optimal “cost subsidy” needed for drugs to pass under a given confidence interval and charge a given price.

Thank you for bringing this up, and we think the latter interpretation is especially insightful. We aim to discuss this in the revised version.

---

Additional work to convince regulatory bodies

Our initial results suggest that the current pricing/revenue of drugs in the US under the FDA p-value of 0.05 is consistent with our model of agent strategic behaviour. That said, from a regulatory perspective, more precise and fine-grained empirical work would be needed to further substantiate this.

A more detailed discussion on the mitigation strategies, whether through adjusting p-values or subsidizing development costs (as mentioned above), is also warranted. Policy decisions like this naturally have broader externalities, hidden or explicit, and they need to be carefully assessed. As for the welfare of such strategies, it is possible to define a Price of Anarchy for Stackelberg games where the optimal welfare corresponds to maximizing the utility for all agents. Conceptually, however, this may be a bit odd since it is not clear why any objective beyond minimizing total error (the current principal objective being optimized) should be socially desirable. It may, however, be prescient to consider whether subsidizing development costs to encourage worthy participation is indeed optimal if there are budget constraints, or what the right trade-offs here are.

Thank you for your helpful review and feedback! We present a natural, tractable model, grounded in real‑world concerns, for studying hypothesis testing in the presence of strategic participants. Our analysis builds on a sequence of subtle observations to produce clear, interpretable results that give valuable insights on incentives and behaviours in this setting.

Depending on the perspective, the resulting conclusions can be surprising. The fact that a lower bound on the optimal p-value emerges from our work is distinct from the classic thinking of upper-bounding the p-value to control Type I errors. Our analysis is also flexible, highlighting that if context dependent p-values are unpalatable, regulators can instead subsidize the fixed and marginal costs to achieve similar results. We will highlight the latter point more in the revision.

We view our work as part of the broader research direction on data-driven decision-making under strategic behavior. Strategic classification, strategic regression, and incentive-aware machine learning have received significant attention in the past decade, with many contributions appearing in NeurIPS and other leading machine learning venues. We hope our work encourages further exploration into strategic aspects of statistical analysis, an area that remains relatively under-explored.