Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs

We thank the reviewer for their constructive and thoughtful feedback. Our responses to each of the concerns are provided below.

1. Extension to continuous hyperparameter spaces:

As mentioned in lines 88–89 of the draft, a pre-selection step using methods such as LLM judges, Bayesian optimization, and Hyperband can be used to construct a discrete candidate set from a continuous hyperparameter space. For example, Bayesian optimization can efficiently explore the continuous domain using acquisition functions, and the most promising configurations can be selected to form the input to RG-PT [Snoek et al., 2012]. Similarly, Hyperband adaptively allocates resources to a large pool of randomly sampled configurations from a continuous domain, ultimately yielding a high-performing discrete subset [Li et al., 2017].

2. On the impact and selection of prior parameters (e.g., $D$, $n_p$):

We agree that these parameters influence the power (though not the statistical validity) of RG-PT. We have provided a detailed and principled procedure for selecting both the DAG depth $D$ and the prior weight $n_p$ in our response to Reviewer T6Dk.

3. Computational complexity:

A detailed complexity analysis and comparison with LTT and PT is already provided in Appendix C of the paper. In practice, we found RG-PT to run efficiently even on large candidate sets (e.g., 100+ configurations), with runtimes under 5 minutes on a standard Apple MacBook Pro (M1 chip). We are happy to include empirical runtime comparisons in the revised appendix to complement the theoretical analysis in Appendix C.

4. Sample efficiency and finite-sample behavior:

As demonstrated through various experiments in [Ramdas et al., 2017], DAG-structured testing yields higher power compared to linear testing when the graph encodes true logical dependencies among hypotheses. To illustrate this more rigorously, we provide next a straightforward analytical comparison in the context of fixed sequence testing (FST), which can be seen as a special case of graph-based (DAGGER) testing with a chain graph.

Suppose we wish to test $K$ hypotheses $\mathcal{H}_1, \ldots, \mathcal{H}_K$. Let $m < K$ of these hypotheses correspond to non-null hypotheses, i.e., reliable hyperparameters, and the remaining $K - m$ be true nulls, i.e., unreliable hyperparameters. Assume all $K$ $p$-values $\\{p_i\\}_{i=1}^{K}$, one per hypothesis $\mathcal{H}_i$, are mutually independent and identically distributed. In particular, let $F_0(p) = p$ be the CDF of all p-values under the null, and $F_A(p)$ be the CDF under the alternative, where $F_A(\alpha) = \beta > \alpha$ reflects the higher concentration of small $p$-values under the alternative hypothesis. Accordingly, for a testing threshold given by $\alpha$ (i.e., $p_i \leq \alpha$ rejects the null), the type-I probability (of incorrectly rejecting the null) is $\alpha$, while the probability of correctly detecting a non-null hypothesis is $\beta$.

Assume now a well-informed testing order whereby FST is applied in the order from $\mathcal{H}\_1$ to $\mathcal{H}\_K$. FST proceeds through these hypotheses, deciding for the alternative (reliable hyperparameter) if $p_i \leq \alpha$. The probability of correctly rejecting all $m$ non-null hypotheses is given by $\prod_{i=1}^m \mathbb{P}(p_i \leq \alpha) = \beta^m$ .

Now consider a misspecified testing order in which $r \geq 1$ null hypotheses appear among the first $m + r$ positions. Let us pessimistically assume that these $r$ nulls occur before all $m$ non-nulls. The test may stop at the first null encountered if its $p$-value exceeds $\alpha$, which happens with probability $1 - \alpha$. Therefore, to reach and reject all $m$ non-nulls, we must reject all $r$ preceding nulls (each with probability $\alpha$) and then reject all $m$ non-nulls (each with probability $\beta$). Thus, the probability of correctly rejecting all $m$ non-nulls is $\alpha^r \cdot \beta^m.$

This derivation shows that even one null placed early in the sequence reduces power by a factor of $\alpha$. In the worst case, with multiple early nulls, the power degradation becomes exponential in $r$.

Similar calculations can be done by considering a more complex structure in the hypothesis space via a DAG. In this case, when the DAG is well specified, the set of true null hypotheses (reliable hyperparameters) corresponds to a rooted subtree of the DAG.

5. On extracting domain knowledge from the learned DAGs:

We agree with the reviewer that the learned reliability graphs may encode meaningful domain-specific structure, especially in rich settings such as prompt engineering. For instance, the DAG could implicitly capture relationships between prompt length, linguistic specificity, or formatting cues and their impact on performance. Extracting such patterns could indeed yield actionable design principles for future prompt development.

That said, our focus in this paper is on statistical reliability and multi-objective selection, rather than semantic analysis or pattern mining. A thorough analysis of graph interpretability across domains would require additional infrastructure (e.g., prompt annotation, linguistic taxonomy alignment), which is beyond the current scope. Nonetheless, we fully agree that this is a compelling direction for future work and will explicitly mention it in the conclusion section.

We thank the reviewer once again for the constructive feedback, and we hope that these additions and clarifications improve their view of the work and alleviate their concerns.

References:

• Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2951–2959.

• Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(1), 6765–6816.

• Ramdas, A., Barber, R. F., Wainwright, M. J., & Jordan, M. I. (2017). A unified treatment of multiple testing with prior knowledge using the p-filter. Annals of Statistics, 47(5), 2790–2821.

We thank the reviewer for the thoughtful and detailed comments. We address each concern below and outline the specific clarifications and additions we will make in the revised manuscript.

1. On the choice of graph depth $D$:

As noted by Reviewer T6Dk as well, we agree that this parameter plays a meaningful role in shaping the power of RG-PT. In our rebuttal to Reviewer T6Dk, we have now proposed and adopted a principled data-driven procedure for choosing $D$.

2. On the semantics of edges inferred via non-negative Lasso:

We appreciate the reviewer’s insightful question. Indeed, our goal is to encode reliability dominance in the DAG edges, not merely predictive correlation. To clarify this, we now provide a structural motivation and explain why a linear model is a reasonable and practical choice.

Let us denote the RG as $G = (V, E)$, where $V = \\{\lambda_1, \dots, \lambda_{|\Lambda_{\text{OPT}}|}\\}$ is the set of candidate hyperparameters and $E \subseteq V \times V$ is the set of directed edges. An edge $(\lambda_i \rightarrow \lambda_j) \in E$ indicates that the reliability of $\lambda_j$ is dependent on that of $\lambda_i$, i.e., $\lambda_i$ is a ``parent" of $\lambda_j$ in the DAG.

We assume that each configuration $\lambda_j \in V$ has an unobserved binary reliability label $R_j \in \\{0, 1\\}$, where $R_j = 1$ indicates that $\lambda_j$ is reliable. We can model the relationship between these reliability variables using a stochastic Boolean structural equation

$$R_j = f_j\big(\\{R_i : (\lambda_i \rightarrow \lambda_j) \in E\\}\big) + \varepsilon_j,$$

where $f_j$ is an unknown binary-valued function, and $\varepsilon_j$ is a noise term, which is generally correlated with the function $f_j(\cdot)$'s output. The function $f_j(\cdot)$ may, for instance, encode that hyperparameter $\lambda_j$ is likely to be reliable as long as a sufficiently large number of parent nodes are. Moreover, the noise term captures the possibility that a child configuration is unreliable ($R_j=0$) due to random effects not captured by function $f_j(\cdot)$.

Although we do not observe the true reliability labels $\\{R_j\\}$, we do observe scalar-valued performance vectors $Y(\lambda_j) \in \mathbb{R}^T$, such as per-task or per-example risks, which serve as continuous surrogates for reliability. We then approximate the above relationship using a linear regression model:

$$Y(\lambda_j) = \sum_{i \neq j} \beta_{ij} Y(\lambda_i) + \varepsilon_j,$$

with non-negativity constraints $\beta_{ij} \geq 0$, which preserve the monotonicity of influence. This leads to the non-negative Lasso objective we solve to infer the edge set $E$.

This formulation does not claim that reliability is fundamentally linear in nature. Rather, it is a linearization of an unknown monotonic Boolean process, which may be justified as done in the literature on influence modeling [Wainright & Chickering, 2006]. The sparsity induced by Lasso ensures that only the most predictive and interpretable relationships are retained. The non-negativity constraint encodes the domain prior that improving the parent should not reduce the reliability of the child.

Ultimately, the FDR control of RG-PT remains valid regardless of any modeling choice made for the construction of the DAG. The DAG only affects the power of the procedure, not its validity.

3. On missing baselines:

We thank the reviewer for highlighting the importance of benchmarking against simpler, heuristic, or non-statistical baselines. We note that this comparison has already been conducted in prior work: both LTT and PT evaluate their performance against non-risk-controlling baselines (e.g., "$\alpha$-constrained'' and "$(\alpha, \delta)$-constrained'') and demonstrate improved power under statistical control. Since our method consistently outperforms LTT and PT across all benchmarks, we implicitly improve over these naive baselines as well. Nevertheless, we appreciate the reviewer’s suggestion and have added additional heuristic baselines in the object detection experiment.

• Chain graph PT: We use our proposed graph learning method to construct a chain graph by setting $D = |\Lambda_{\text{OPT}}|$, and then apply PT using this learned chain as the ordering.

• Randomly shuffled PT: We run PT on a chain graph where the hyperparameters have been randomly shuffled, effectively removing prior structural information.

• Uniform Sampling: We uniformly sample $k$ hyperparameters (without replacement) and return them as the selected subset, mimicking random selection without statistical guarantees.

The median object recall threshold ($\lambda_1$) achieved by chain graph PT, randomly shuffled PT, and uniform sampling was 0.61, 0.52, and 0.49, respectively, while for PT and RG-PT it was 0.56 and 0.68, respectively. The results show that randomly shuffled PT underperforms compared to standard PT, while chain graph PT offers a modest improvement over PT but remains weaker than RG-PT, highlighting the benefits of using a learned DAG structure. Finally, uniform sampling performs erratically, the selected hyperparameters span the full performance range (reflecting the lack of selection precision), and the method exhibits a high average FDR of 0.54, confirming its lack of statistical reliability.

4. On structural sensitivity of the DAG:

We agree with the reviewer that the quality of the learned DAG influences the power of the procedure. That said, the FDR guarantee remains intact regardless of DAG structure, as DAGGER is valid under arbitrary acyclic graphs.

Our approach assumes that there exists an unknown latent DAG that reflects true reliability dependencies. The goal of the learning procedure is to approximate this latent structure from available signals (BT scores, prior edges, and observed risk vectors). There are indeed many ways to learn such a DAG. We proposed a computationally tractable pipeline based on convex surrogates and empirical priors. Our ablations (e.g., on graph depth $D$) already explore aspects of structural variation. We will consider expanding the ablation scope (e.g., edge pruning and alternative clustering methods) in future work, though we note that defining ground truth reliability DAGs remains challenging in real-world datasets.

5. Minor points: We will enlarge the font size in Figure 1(a) and add a table of notations to the Appendix for reader convenience.

We thank the reviewer once again for the constructive feedback, and we hope that these additions and clarifications improve their view of the work and alleviate their concerns.

References:

• Wainwright, M. J., & Chickering, M. (2006). Estimating the "wrong" graphical model: Benefits in the computation-limited setting. Journal of Machine Learning Research, 7(9), 1829–1859.

We thank the reviewer for their helpful review. We would like to first clarify that the theoretical FDR guarantee of RG-PT holds regardless of the choices of the parameters $D$, $n_p$, and $\tau$. These parameters affect the construction of the RG and hence influence statistical power, but not the validity of the testing procedure.

That said, we agree that practical guidance is important for users, and we plan to include in the revised manuscript a description of a specific procedure for selecting these hyperparameters. This is the method we employed in our experiments, and we will include it in the appendix of the revised version to aid future applications.

Graph Depth $D$: We select $D$ by computing the Silhouette score [Rousseeuw, 1987] for each candidate value $D \in \\{2,3,\dots,D\_{\max}\\}$ and select the value that maximizes the average Silhouette score. The maximum depth can be set to $|\mathcal{Z}\_\text{OPT}|$, or a fixed, smaller, value such as setting $D\_{\max} = 20$ for computational efficiency. The Silhouette score quantifies both intra-cluster cohesion and inter-cluster separation, and is a well-established criterion for model-free cluster number selection [Kaufman & Rousseeuw, 2009]. This approach is fully data-driven, and fits naturally with the structure of RG-PT, where each cluster corresponds to a DAG level.

Prior Weight $n_p$: We set $n_p$ using a predictive empirical Bayes procedure, which calibrates the trust in the prior information $\eta\_{ij}$ based on its ability to generalize to held-out data. Specifically, we follow the steps below:

• Create a set of candidate values for $n_p$, e.g., $\\{0.25|\mathcal{Z}\_\text{opt}|, 0.5|\mathcal{Z}\_\text{opt}|, 0.75|\mathcal{Z}\_\text{opt}|, |\mathcal{Z}\_\text{opt}|\\}$

• For each candidate $n_p$, compute the BT scores $s(\lambda)$ for the hyperparameters in $\lambda \in\Lambda\_\text{OPT}$.

• Evaluate the predictive log-likelihood of the pairwise outcomes on a held-out validation dataset $\mathcal{Z}\_\text{val}$ as $\mathcal{L}\_{\text{val}}(n_p) = \sum\_{\lambda_i \in \Lambda_{\text{OPT}}} \sum\_{\lambda_j \in \Lambda\_{\text{OPT}}} \left[ y_{ij} \log \frac{e^{s(\lambda_i)}}{e^{s(\lambda_i)} + e^{s(\lambda_j)}} + (1 - y\_{ij}) \log \frac{e^{s(\lambda_j)}}{e^{s(\lambda_i)} + e^{s(\lambda_j)}} \right]$ where $y\_{ij} = 1$ if $\lambda_i$ outperformed $\lambda_j$ in $\mathcal{Z}\_\text{val}$, and 0 otherwise.

• Choose the $n_p$ that maximizes $\mathcal{L}\_{\text{val}}(n_p)$.

This procedure reflects standard practice in empirical Bayes model selection [Efron and Morris, 1973] and ensures that the prior contributes proportionally to its predictive usefulness.

Lasso Regularization $\tau$: We select the Lasso regularization parameter $\tau$ using stability selection [Meinshausen & Bühlmann, 2010], a principled technique for tuning regularization parameters. Our goal is to obtain a DAG structure that is both sparse (to maximize testing power and interpretability) and stable (i.e., robust to data perturbations). Stability here refers to the reproducibility of selected edges under subsampling. The procedure is as follows.

• We generate multiple random subsamples of the calibration dataset $\mathcal{Z}\_{\text{OPT}}$, each containing a fixed proportion of the data (e.g., 50%) as in standard stability selection [Meinshausen & Bühlmann, 2010].

• For each candidate value of $\tau$, we fit the non-negative Lasso independently on each subsample, and follow the steps to create the RG.

• For each edge $(\lambda_i \rightarrow \lambda_j)$, we record its selection frequency, the fraction of subsamples in which it appears in the inferred DAG.

• We then compute the edge stability score for a given $\tau$ as the average selection frequency over all edges. This metric reflects the reproducibility of the graph structure.

• Among all candidate $\tau$ values, we select the one that maximizes stability, subject to a sparsity constraint on graph complexity (e.g., average node degree $\leq 3$), to prevent overfitting and preserve statistical power [Shah & Samworth, 2013].

We will include these procedures in the appendix of the final version. This ensures that RG-PT can be applied in a fully automated and reproducible manner, without compromising its theoretical guarantees.

References:

• Rousseeuw, P. J. (1987). Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53–65.

• Kaufman, L., & Rousseeuw, P. J. (2009). Finding groups in data: an introduction to cluster analysis. John Wiley & Sons.

• Efron, B., & Morris, C. (1973). Stein’s estimation rule and its competitors—an empirical Bayes approach. Journal of the American Statistical Association, 68(341), 117–130.

• Meinshausen, N., & Bühlmann, P. (2010). Stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(4), 417–473.

• Shah, R. D., & Samworth, R. J. (2013). Variable selection with error control: another look at stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 75(1), 55–80.

We thank the reviewer for the thoughtful and detailed comments. We address each concern below and outline the specific clarifications and additions we will make in the revised manuscript.

1. Lack of theoretical insight into why RG-PT outperforms LTT and PT

Please see point 4 of our response to reviewer LYAN.

2. Clarification on claim that BO and MAB “lack statistical guarantees”

We will revise Line 59 to clarify that BO and MAB methods lack post-selection error rate control such as FDR or FWER guarantees. Under assumptions such as the correct specification of kernel models, BO and MAB can provide bounds on cumulative or simple average regret. These do not offer the type of assumption-free guarantees on the risk of the selected parameters that can be instead provided by FWER- or FDR-controlling methods.

To validate the practical impact of the different theoretical guarantees by BO and MAB, we have now implemented BO and MAB on the seq2seq task, comparing the achieved FDR with LTT, PT, and RG-PT. For BO, we used the scikit-optimize implementation with the UCB acquisition function. For MAB, we used a standard stochastic bandit algorithm, namely Thompson Sampling.

It was observed that only RG-PT, LTT, and PT satisfy the target FDR level $\alpha = 0.1$, while the achieved FDR on the test dataset for BO and MAB was 0.14 and 0.21, respectively.

3. Terminology: “selected” vs “rejected” in Line 117

Thank you for this suggestion. While “rejected” aligns with classical hypothesis testing, we intentionally used “selected” to match the hyperparameter selection framing. We will clarify this distinction in the revised text for consistency with standard terminology.

4. Derivation of the $p$-value in Equation (6) and (7)

Equation (6) is derived using Hoeffding's inequality, applied to the empirical mean of bounded loss values in $[0,1]$, similar to the approach in LTT [Angelopoulos et al., 2021] and PT [Laufer et al., 2022]. Additionally, the proof that eq. (7) forms a valid p-value can be found in Appendix A.2 of the PT paper [Laufer et al, 2022]. We will include a short derivation and cite these works explicitly to make the paper more self-contained.

5. Interpretation of Equation (10) as a “data-driven probability”

We agree that this terminology is unclear and will revise it. This construction belongs to a broader class of metrics used in paired comparison and choice models [Cattelan, 2011]. A function $f: \mathbb{R} \to [0,1]$ is called a link function if it satisfies the symmetry condition

$$f(x) + f(-x) = 1,$$

and is increasing in $x$. In pairwise comparison models, such a link function is used to compare two items with respective scores $p_i$ and $p_j$ (with higher values indicating better performance) by evaluating $f(p_i - p_j)$. This quantity can then be interpreted as the relative likelihood that item $i$ is preferable to item $j$. Examples of commonly used link functions include [Cattelan, 2011]:

• Thurstone model [Thurstone, 1927]: $\displaystyle f(p_i- p_j) = \Phi(p_i - p_j)$, where $\Phi(\cdot)$ is the normal CDF.
• BT model [Bradley & Terry, 1952]: $\displaystyle f(p_i- p_j) = \frac{1}{1 + e^{-(p_i - p_j)}}$,
• Transformed BT model (our choice) [Bradley & Terry, 1952]: $\displaystyle f(\log p_i- \log p_j) = \frac{1}{1 + e^{-(\log p_i - \log p_j)}} = \frac{p_i}{p_i + p_j}$.

We chose the linear ratio (transformed BT model) form for its simplicity, numerical stability, and effective integration with the Lasso-based DAG construction. In the revised manuscript, we will clearly state this design choice and emphasize that alternative transformations from this family could be substituted without affecting the theoretical guarantees of RG-PT.

6. Justification of Equation (11)'s weighting scheme

Equation (11) defines a convex combination of prior and data-driven comparisons to estimate the pairwise reliability strength between hyperparameters $\lambda_i$ and $\lambda_j$. The ratio $n_p / |\mathcal{Z}_\text{opt}|$ encodes our confidence in the prior information relative to the observed data, where $n_p$ serves as a pseudocount analogue.

This formulation is not ad hoc, but is directly inspired by the theory of conjugate priors in exponential family models [Diaconis and Ylvisaker, 1979]. In such models, the natural parameters of the posterior distribution are formed by additive combinations of prior and empirical sufficient statistics. For example, in the Beta-Binomial model, the posterior mean of a Bernoulli parameter $\theta$ is

$$\mathbb{E}[\theta \mid \text{data}] = \frac{\alpha + s}{\alpha + \beta + n},$$

where $(\alpha, \beta)$ are prior pseudocounts and $(s, n-s)$ are observed successes and failures. This additive structure generalizes to Dirichlet-Multinomial models (of which the BT model is a special case), where prior and empirical pairwise counts are combined to produce regularized estimates of comparison probabilities.

In our setting, the constructed count $w_{ij}$ can be viewed as a smoothed estimate of the relative reliability of $\lambda_i$ over $\lambda_j$, constructed analogously to the posterior update in exponential family models with conjugate priors, where the prior and observed sufficient statistics combine additively to determine the posterior natural parameter. The weighting in Equation (11) thus mimics a Bayesian update, where the prior and data each contribute proportionally to the final score. This principle is well-established in the Bayesian literature [Bernardo and Smith, 1994; Diaconis and Ylvisaker, 1979; Gelman et al., 1995], and we will make this connection explicit in the revised manuscript.

7. Behavior when there is no structure among hyperparameters

In scenarios where no prior structural knowledge is available, we explicitly set $n_p = 0$, resulting in an RG that is learned entirely from held-out data via the BT model and non-negative Lasso. This setting is described in Lines 211 and 221 of the manuscript. Even in this purely data-driven regime, RG-PT retains the formal FDR control guarantees of DAGGER.

Clearly, if the hypothesis space is completely unstructured, one cannot hope to learn useful information through this approach. That said, our experiments confirm that RG-PT still exhibits a larger power compared to LTT and PT even when ordering among hypotheses are not hard-coded into the testing procedure. In particular, in the object detection task, we did not assume access to any prior reliability information and set $n_p = 0$. Despite this, RG-PT still outperformed both LTT and PT, demonstrating its ability to discover and exploit latent structure purely from empirical data. We will clarify this point more explicitly in the revised version.

8. Edge case: all non-negative Lasso coefficients are positive

Yes, it is possible that a node has all previous-layer nodes as parents if all Lasso coefficients are positive. In such cases, the node will be tested only if all these parents are declared reliable. For example, the bottom node in Fig. 1 has all the nodes from the previous level as a parent, illustrating this case.

9. Justification for the reliability threshold $\alpha$ and FDR level $\delta$

The threshold $\alpha$ defines what it means for a hyperparameter to be considered “reliable,” i.e., having risk below $\alpha$. Its choice depends on the downstream application and the desired level of caution. In our experiments, we selected $\alpha$ via preliminary validation sweeps to identify values that effectively separate low-risk configurations while maintaining adequate statistical power. For example, values of $\alpha$ in the range $[0.1, 0.3]$ are commonly used in the applications considered in the LTT paper [Angelopoulos et al., 2021].

The parameter $\delta$ sets the desired control level for the FDR, specifying the maximum acceptable expected proportion of selected hyperparameters whose risk exceeds the target level $\alpha$. Following common practice in the MHT literature, including the LTT and PT papers, we set $\delta = 0.1$, corresponding to a confidence level of 90% for the reliability of the selected hyperparameters. We will clarify this rationale and include appropriate citations (e.g., [Benjamini and Hochberg, 1995]) in the revised manuscript.

10. Report achieved FDR for LTT and PT

The empirical FDR achieved by RG-PT for all the tasks is already reported in Table 2 of the Appendix. In the revised manuscript, we will extend this table to also include the achieved FDR values for LTT and PT as well. The results for LTT and PT were originally excluded, as the FDR control of these two methods has already been validated experimentally in their respective papers. Importantly, the experiments all demonstrate that LTT, PT, and RG-PT all satisfy the FDR constraint, as backed up by theory.

We thank the reviewer again for the constructive feedback. We believe the above clarifications and additions will significantly improve the clarity and rigor of the paper.

References:

• Cattelan, M. (2011). Models for paired comparison data: A review with emphasis on dependent data. Statistical Science, 26(3), 412–433.

• Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34(4), 273–286.

• Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345.

• Diaconis, P., & Ylvisaker, D. (1979). Conjugate priors for exponential families. The Annals of Statistics, 7(2), 269–281.

• Bernardo, J. M., & Smith, A. F. M. (1994). Bayesian Theory. Wiley.

• Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (1995). Bayesian Data Analysis. Chapman and Hall/CRC.

• Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.