We thank the reviewer for their useful comments. We have included a set of new experiments, described at the end of this review.

Reviewer Comments

1. On the KS test performance: The KS tests for the Exponential and Gaussian models do reveal discrepancies. Both reference priors have asymptotes at $\theta=0$. Our failure to fully capture the shape of the reference prior results from having to constrain the output of the variational prior to a subset of the real line. Currently, the final layer of the normalising flows we use is a sigmoid layer, and it must therefore be the case that the flow's density $\to 0$ at the boundaries of the region (since it cannot assign mass to infinitely low values in logit space), meaning the current network design cannot capture an asymptote accurately. This can passing a KS test hard. Yet, aside from the asymptote, the learned and true priors are similar over the majority of the support (see updated Fig. 2 in pdf below). Note that this problem would persist even when modelling $\log(\sigma)$: either we'd leave the space unconstrained, in which case we'd fail to fully capture the uniform, improper prior on the unconstrained space, or we'd constrain to a bounded domain through a sigmoid layer again, leading to the same problem as above. We will discuss this in the revision, along with the use of alternative variational families that can better capture asymptotes.
2. On learning priors in unconstrained spaces and learning flat priors over the reals: Please see previous bullet point, which we hope addresses this.
3. On experiments being low dimensional: To demonstrate our methods' abilities to perform well for higher-dimensional parameters, we include in the pdf below (Fig. 1) new experiments on the SLCP model (description below, and further discussion in our reply to Reviewer V6ZV), which is higher-dimensional. Both methods recover expected priors; GED is less successful since it does posterior estimation, and SLCP often has complex posteriors. We'll discuss this in the revision.
4. On the additional references: Thank you for these suggestions. These are relevant to the problem of determining prior distributions but focus on eliciting expert priors (arguably the opposite of our problem).
5. On applications & practical implications of reference priors in SBI: It is possible for certain regions of parameter spaces to produce unreasonable/nonsensical outputs from simulators, and that reference priors might assign significant mass to such regions. However, the modeller can still restrict attention to regions of the parameter space $\Theta$ that do not produce absurd behaviours even when using our methods; we simply address the problem of how to learn a reference prior on a parameter space once it is given (which may be a subset of $\Theta$). We provide examples in the new experiments (Figures 2 and 3 of accompanying pdf) on doing SBI using the learned reference priors (and posteriors), and we can further point to our answer to Question 2 for reviewer RgwU, where we provide additional discussion on the use of reference priors for SBI.

New experiments

We have included a set of new experiments here, https://github.com/ICML-7582/rebuttal_plots/blob/main/plots.pdf, including:

• Fig. 1: Experiments on a new simulator (SLCP-D [1]) that possesses a higher dimensional parameter space. The SLCP-D model has five parameters which parameterise the mean $\mu(\theta)$ and covariance $\Sigma(\theta)$ of a 2D Gaussian. The output of SLCP-D consists of four iid samples from said Gaussian and 48 distraction vectors sampled from a mixture of Student's $t$-distributions that are independent of the parameters. SLCP-D has a relatively simple likelihood that produces complex posteriors. The true reference prior should be uniform in $\mu(\theta)$, and for the determinant $|\Sigma(\theta)|$ should decay rapidly in $|\Sigma(\theta)|$. VLB & GED recover this behaviour. This also highlights that our methods are approximately invariant to reparameterisation: we learn reference priors for $\theta$ but recover the correct behaviour of $\mu (\theta)$ and $|\Sigma(\theta)|$. VLB outperforms GED for SLCP-D; intuitively this is because GED approximates a (complex) posterior during training, while VLB methods do not.
• Figs. 2 & 3: SBI experiments using reference priors from InfoNCE (a VLB method) & GED. Fig. 2: NRE posterior vs. ratio estimator trained during the VLB reference prior training; Fig. 3: NPE posterior vs. posterior density estimator trained during GED training. Inferences are almost identical, demonstrating our point about being able to perform SBI with no additional cost.
• Fig. 4: New plots for SIR model outputs when only the infection data is used to find reference prior.
• Fig. 5: An updated version of Figure 2 in the paper.
• Fig. 6: An updated plot of the KS test.

[1] Lueckmann et al., "Benchmarking Simulation-Based Inference", AISTATS 2021.

Thank you for your helpful feedback. We have included a set of new experiments here, and see reply to Reviewer UJGD for a detailed explanation.

Methods and Evaluation Criteria

• On inconsistencies between methods: It is reasonable that there may be inconsistencies between methods since there may be multiple local optima that give approximately the same mutual information between the model parameters and output but correspond to different priors. We will use some of the extra space to discuss that a sensible approach in practice may be an "ensembling" approach, namely to perform multiple runs of the prior learning procedure to account for any such variation. Such ensembling approaches are common in SBI to account for variations of this kind, see e.g. [1, 2] below.
• On choosing between methods: Please see our discussion on this question in our response to Reviewer RgwU ('Question 3'). We will add this discussion to the final version.
• On the performance of GED given that it learns a posterior as well as a prior: Indeed, estimating the posterior is not straightforward for some simulators. This is a possible disadvantage of GED. However, the method has produced reference priors in almost all cases matching either the ground truth or the other approaches. Other approaches might be preferable depending on problem specifications. We believe our methods (the first to address approximating reference priors in likelihood-free settings) constitute a useful baseline upon which future work can likely improve.
• On lines 196--198: We'll use some of the extra space to give further details on the underlying methodology: we'll move the algorithm from Appendix A.6 into the main body, and specify that taking the prior and posterior estimators to be of the same model class is not necessary (the main reason for this was to reduce tuning and method complexity).
• On higher-dimensional parameter spaces: Please see discussion on SLCP below.
• On metrics: We have now used these metrics and present them in the tables below (format: mean (standard dev) from 5 repeats). We'll use some of the extra space to discuss in the main body that we perform generally better but often similarly to/a bit worse than a likelihood-based method ("Berger") from [3].

Wasserstein:

C2ST:

• On Equations 2, 3, 9, and 10: Thank you; we'll derive these relationships in the appendix for clarity.
• On performing SBI: We include a demonstration of how SBI can be performed at no additional cost using our methods, and a comparison against the results of running NPE and NRE in Figs. 2 & 3 of the rebuttal pdf. We'll use the extra space to include these results in the revised paper.

Experimental Designs or Analyses

1. Please see comment above on the new test metrics. For further comments and discussion on the use of the KS test, please see reply 1. to Reviewer UJGD below.
2. We have included new experimental results for models with a higher number of parameters (SLCP, for description see reply to UJGD). The true reference prior for the parameters should be uniform in $\mu(\theta)$ whilst the prior density for the determinant $|\Sigma(\theta)|$ should decay rapidly in $|\Sigma(\theta)|$. The priors learned by VLB and GED recover this behaviour. This highlights that our methods are approximately invariant to reparameterisation, since we learn reference priors in terms of $\theta$ but recover the correct reference prior in terms of $\mu (\theta)$ & $\Sigma(\theta)$. VLB methods outperform GED; GED approximates a (complex) posterior during training, while VLB methods don't. More generally, the tradeoffs between VLB and GED are analogous to those between NRE and NPE.
3. Please see our responses above on how to decide between/combine methods.

[1] Cannon et al., "Investigating the impact of model misspecification in neural simulation-based inference", arXiv preprint (2022)

[2] Elsemüller et al., "Sensitivity-aware amortized Bayesian inference", TMLR (2024)

[3] Berger et al., "The Formal Definition of Reference Priors", The Annals of Statistics (2009)

Thank you for your feedback. We have included a set of new experiments here, and see reply to Reviewer UJGD for a detailed explanation.

Figure 2

• The bumps in panels a) and b) were an artefact of the network architecture. Sigmoid layers in normalizing flows seem to induce unexpected tail behaviours in the resulting densities. Having experimented further, they have now disappeared, see Fig. 5 in the rebuttal pdf.
• The stopping point of $\theta = 100$ was largely arbitrary, but not going higher can be justified by the fact that the prior falls off as a power law for panels a) and b).
• Neither the reference prior theoretical foundations nor the VLB or GED methods require compact parameter sets. It happens to be that most simulators chosen require parameters to be in some compact set.
• The question of where to truncate the prior for a potentially unbounded parameter space is important, but is a general problem when specifying priors and not specific to our methods. We instead address the problem of "given a parameter space, how could a minimally informative (proper, continuous, positive) prior be constructed?"
• The y axes in panels c) and d) must necessarily be truncated due to the presence of asymptotes at the boundaries. We considered these plots to already adequately show that our methods have imperfections, such as InfoNCE and SMILE slightly overestimating densities close to the boundary in panel c), and GED exhibiting a slight left-skew in panel d).
• Re. question on SBI effects: We now include a gold standard baseline for the reference priors in Fig. 2 of the accompanying pdf using the algorithm in [1] which uses knowledge of the tractable likelihood functions of these models. Plots and KS test statistics for these can be seen in Figure 6 in the rebuttal plots, and table of additional metrics requested by Reviewer V6ZV are given in our response to them below. In general we perform better, but in some cases similarly.

Figure 4

• We agree that diversity in one statistic $\neq$ diversity in another. We chose $x$ to be curves of the proportions of individuals who are susceptible, infected, and recovered through time. A good reference prior should -- since the MI is a difference between the entropies of the marginal likelihood $H(x)$, and the likelihood function in expectation over the prior, $H(x\mid\theta)$ -- maximise $H(x)$ (encouraging diversity in statistic $x$) while minimising $H(x\mid\theta)$ (discouraging diversity in statistic $x$ from individual likelihood functions, on average). Panels a) and b) in Fig. 4 aim to show this. In the accompanying rebuttal plots we include similar plots obtained by finding reference priors from GED and VLB with $x$ as the infection curve only. In this case, the infection curves are much more diverse.
• Panel d): We agree that the features we expect are approximately, but not perfectly, recovered from our prior learning methods. In the revision, we will use some of the extra space to discuss reasons why we believe our methods have struggled for this g-and-k simulator. Our other experiments, in addition to our new SLCP experiments (see Fig. 1 in the rebuttal pdf), do however demonstrate that our approaches can learn useful approximations to reference priors.

Methods and Evaluation Criteria

Please see our discussion of our comparison to the method from [1] and about diversity above.

Relation to Broader Literature

We'll use some of the extra space to give more details on Nalisnick & Smith (2017) & how we differ from this prior work.

Other Strengths and Weaknesses

We hope our new experiments that compare the performance of our methods to the gold standard tractable-likelihood approach from [1] (see above & response to Reviewer V6ZV) and demonstrate the ability of VLB & GED to perform SBI accurately at no extra cost (see Fig. 2 & 3 in accompanying pdf) address your concerns.

Other Comments

We'll fix the typos. We'll also: say "minimally informative", not "uninformative", in abstract; use "Jeffreys' prior"; write "optimise an estimate of, or a lower bound on, the MI..."; ensure "VLB" is defined on first use; and write $c$ in Eq. 18 and state that $c=0.8$ is a common choice that we also use. We'll explain the (missing) narrative link to Jeffreys' prior; the point of this, and Eq. 7 in particular, was to highlight that continuous, positive distributions satisfying Eq. 2 also (asymptotically & under regularity conditions) satisfy an alternative notion of "minimally informative" given in Eq. 7. Finally, we'll state these regularity conditions.

Questions

Our use of a prior and posterior estimator that are of the same model class is just for ease of hyperparameter tuning, and not a requirement of GED. On the comment re. continuous approximations: Please see "Other comments" above.

[1] Berger et al., "The Formal Definition of Reference Priors", The Annals of Statistics (2009)

Thank you for the valuable feedback. We have included a set of new experiments, see the reply to Reviewer UJGD for a detailed explanation.

Questions

1. We will include explicit technical definitions for, and a discussion of, proper and improper priors. Briefly: an improper prior is an "unnormalisable" prior, i.e. one with infinite integral over the support, while a proper prior can be normalised to have integral 1; improper priors do not strictly correspond to probability measures for this reason, but can sometimes still produce proper Bayesian posterior distributions.
2. We will use some of the additional space to include a discussion on these 4 points at the end of our revised paper. Briefly:
   • The first of these points we discuss in bullet point 3. directly below this one.
   • Using a user-specified (e.g., Uniform) prior may be cheaper since it involves no learning of a prior; however, we are not recommending that our learned reference priors replace any subjective priors the modeller believes in, but rather that they complement such subjective priors by enabling a prior sensitivity analyses. Our methods allow modellers to demonstrate the degree to which their inferences differ from the case where "minimal" information (as measured by the MI between $\theta$ and $x$) is built in to the prior, and so we envision that our methods generate priors that are used alongside the modeller's subjective prior.
   • To assess whether a reference prior is sensible (whether it serves the purpose it intends to serve) the modeller can estimate the MI between $\theta$ and $x$ using the different networks trained in the VLB and GED methods, or e.g. via sample-based entropy estimators [1]. However, note that estimating MI from finite data is a challenging problem (see [2], [3]). Alternatively, one can measure divergences between prior and posterior distributions as an estimate of the MI to assess these.
   • We outline what we believe is the main practical and conceptual benefit in the second bullet point above, and will further emphasise this in the revision.

Question 3: Differences between VLB and GED

We will include detailed heuristics in the paper. VLB methods maximise a variational lower bound for the MI which relies on learning critic networks to (essentially) construct density ratio estimates; GED methods construct density estimates from samples of the marginal and conditional distributions of $(\theta, x)$ and maximise directly the estimated MI.

• GED methods do not require differentiability so they are preferred for problems with non-differentiable data collection, but (for equally computationally complex simulators) VLB methods are less computationally expensive than GED methods (in the sense that they only need to learn one density estimator instead of two).
• GED methods rely on constructing an estimate for $\pi(\theta \mid x)$. In problems prone to yield very complex posterior distributions, this may hinder the efficacy of the method. In these, VLB methods might be preferred (when applicable), since this entails only one density estimation task on $\Theta$ (i.e., learning the prior), while GED involves two (i.e., both the prior and posterior).
• When successfully learning a reference prior $\pi$ via GED, we also get (at no extra cost) an estimator for the posterior $\pi(\theta \mid x)$ under $\pi$, which we can directly use for e.g. SBI. We did this in the new rebuttal experiments (see Figures 2 and 3).

Weaknesses

1. We will add detail to Figure 1 in the revision for clarity and place it earlier.
2. We agree that practitioners will often have prior constraints, and we support the incorporation of such beliefs into their Bayesian analysis. As we discuss above, we do not argue against the use of such subjective priors, but instead intend to provide practitioners with tools to assess how much information they have built into their prior, by equipping them with methods that let them find "minimally informative" priors. Our methods can of course also be used when no strong subjective prior beliefs exist, or in situations in which the practitioner prefers to minimise their own influence on the Bayesian analysis for whatever reason.

General comments

• On the connection to GBI: We’ll use some of the extra space to briefly discuss connections to GBI.
• Line 198: Thank you, we'll correct this typo.

References

[1] Kraskov, Alexander, Harald Stögbauer, and Peter Grassberger. "Estimating mutual information." Physical Review E—Statistical, Nonlinear, and Soft Matter Physics 69.6 (2004): 066138.

[2] Song, Jiaming, and Stefano Ermon. "Understanding the limitations of variational mutual information estimators." arXiv preprint arXiv:1910.06222 (2019).

[3] McAllester, David, and Karl Stratos. "Formal limitations on the measurement of mutual information." International Conference on Artificial Intelligence and Statistics. PMLR, 2020.