Addressing Misspecification in Simulation-based Inference through Data-driven Calibration

We thank you for the very valuable feedback. We hope our answers below address the major concerns you have.

The main weakness ... consider recommending acceptance.

We would like to point you to the second paragraph in the introduction, which motivates the use of a calibration set and explains in what real-world situations we can expect to have labeled real-world observations, together with examples. As we wrote, for such real-world applications, where the result of the inference must be accurate, there will most likely be a labeled dataset so the practitioner can verify that the pipeline works and build trust in the result.

As an expert in SBI ... OT to understand what is going on.

Thank you for raising this point - we will further expand this section to make it easier for non-expert readers. Do you have any more specific suggestions of what background is missing for you to better understand this part?

The applied metrics in the experiments are reasonable ... SD or some other easy to understand quantities. Once we acknowledge model misspecification, we lose access to a simulator of the “true” generative model of the real-world data and thus also to the “true” posterior distribution. For instance, for the Task E and F, relying on a limited set of real-world data, we do not have access to the “true” posterior. Thus we chose to rely on two complementary metrics that can be used in practical settings, have clear interpretations and are only optimal for the “true” posterior, if we would know it. These metrics, the ACAUC and LPP, highlight if the credible intervals cover the true parameter value at the designated frequency and the informativeness (i.e., how much uncertainty reduction do we get by using the learnt posterior distribution) to infer the parameter of the posterior respectively. To contextualise the LPP numbers, we provide the “SBI” and “prior” baselines that represent an upper bound and a lower bound on the performance of well-calibrated models, when the independence assumption made in equation 3 holds. Nevertheless, we appreciate the suggestion for synthetic benchmarks, for which it is possible to compute the mean and SD of “true” posterior distribution. We are planning to add those results as a table in the appendix.

Curse of dimensionality ... in more detail here if possible.

It appears non-trivial to make any formal statement on the scalability of the method as N, the number of parameters of interest, increases in general. We believe this will depend strongly on the exact simulator and real-world data considered. Nevertheless, we can reason about some corner cases. In case the parameters are really independent, the worst we can do is to apply the method N times and it should lead to reasonably good results (as a function of the degree of misspecification and C, the size of the calibration set). We could also apply RoPE jointly and hope to gain data efficiency as the sources of misspecification could be common to multiple parameters and the finetuning step would thus rely on strictly more information than if it only considered one parameter at a time. A toy example would be $\theta := \mu \in \mathbb{R}^k$ and $x_s \in \mathbb{R}^{k\times N} \sim \mathcal{N}(\mu, 1) \times \dots \times \mathcal{N}(\mu, 1)$ while $x_o \in \mathbb{R}^{k\times N} \sim \mathcal{N}(\mu + c\mathbb{I}, 1) \times \dots \times \mathcal{N}(\mu + c\mathbb{I}, 1)$. This example highlights that, in certain cases, increasing the dimensionality of the parameters can help as this also implies that the calibration set is strictly more informative on the misspecification.

Unfortunately, in other interesting cases there could be dependence between parameters (given the observation) that we would like to model as well as possible. In such cases, if these dependencies are poorly described by the simulator, we would expect the curse of dimensionality to hurt us and recovering these dependencies might necessitate exponentially larger calibration sets. Nevertheless, if the dependencies are well described by the simulator, we might again benefit from the more informative calibration set.

Thank you for your follow-up questions. We have addressed them below and hope our responses provide the necessary clarifications. We believe the points raised can be effectively addressed through minor revisions to the current manuscript. However, if further clarification is needed, we would be happy to provide it.

I understand your use cases where you argue for the existence of some labeled real data set. The application I work with (e.g., in psychology) have parameters (which would be the "labels" in your case) which are fundamentally unobservable. So where we would not get any real-world labled dataset. But I do see that they may be other use-cases.

We absolutely agree that there are many applications of SBI where the inference parameters are unobservable. Our goal was to explore an alternative approach for the case when they are, because there are interesting real-world applications where this is the case where SBI could prove very valuable. Is this clear from the second paragraph in the introduction?

The notation used in the definition of mathcal{B}_0 after the condition operator is unclear to me. As a result also the notation in EQ2 only makes sense half-way.

Thank you for pointing this out, the notation is indeed a bit confusing. Here’s an alternative expression for line 179 $\mathcal{B}\_o = \{P\in\mathbb{R}^{N\_o\times N\_s}\_+ : \textstyle\sum\_{j=1}^{N\_s} P\_{ij} = \frac{1}{N\_o} \;\forall i=1,...,N\_o \}$

For equation 2 we will keep the notation but add an explanation, i.e.

$P^\star = \arg\min\_{P\in\mathcal{B}\_o} \langle P, C\,\rangle + \rho\mathop{KL}\left(P^T\mathbf{1}\_{N\_o} \| \frac{\mathbf{1}\_{N_s}}{N\_s}\right) + \gamma \langle P, \log P\rangle,$ where $\mathbf{1}\_{n}$ is a vector of ones with size $n$ and $\mathop{KL}\left(P^T\mathbf{1}\_{N\_o} \| \frac{\mathbf{1}\_{N\_s}}{N\_s}\right)$ is the Kullback-Leibler divergence between the marginal distribution over the simulated observations implied by the transport matrix $P$ and the uniform distribution (where each observation has the same probability).

What do you think? Would this be clearer?

I see that of course we don't have a ground truth in the real world datasets. But what about the other datasets? There we could apply other (ground truth) metrics too. Thank you for planning to add them.

We also don’t have ground truth for synthetic benchmarks because we don’t have a closed-form expression for the posterior; we can approximate it using NPE (which is the posterior estimated by the “SBI” baseline in our experiments). If you still feel this is valuable, we can add (in the Appendix) some additional analysis (e.g., metrics & visualizations) using this proxy posterior.

Thank you for your explanation on the curse of dimensionality. I see why it is hard to reason about it formally but what abut empirically? This could be studied in the experiments too, right?

We fully agree with your point that developing algorithms capable of robustly predicting posterior distributions over high-dimensional parameter spaces, even in the presence of model misspecification, is a crucial research direction. We also acknowledge that this is not something RoPE explicitly addresses in its current form, nor has it been empirically demonstrated yet. We hope RoPE can be a good source of inspiration for paper tackling such settings. Nevertheless, we believe this falls outside the scope of our paper, which focuses on addressing SBI under model misspecification in its more common form—namely, for low-dimensional parameter spaces—an area explored by state-of-the-art papers [1-2]. By introducing two real-world benchmarks and evaluating population-level metrics, we believe our work advances the experimental validation further than previous studies. Furthermore, we argue that any high-dimensional experiment should be grounded in a real-world problem, using realistic data to reflect genuine rather than idealized model misspecification. In such cases, evaluating the "correctness" of posterior distributions becomes inherently complex and application-dependent, requiring careful consideration of the specific scenario to ensure meaningful results. While we agree that this is an important and interesting challenge—relevant, for example, to climate and weather modeling or gravitational lensing—we believe it warrants a dedicated study in its own right. We also clearly acknowledge that applying RoPE to such setting may likely requires certain modification to the algorithm proposed.

[1]:Huang, Daolang, et al. "Learning robust statistics for simulation-based inference under model misspecification." Advances in Neural Information Processing Systems 36 (2024).

[2]:Ward, Daniel, et al. "Robust neural posterior estimation and statistical model criticism." Advances in Neural Information Processing Systems 35 (2022): 33845-33859.

Dear authors,

The rebuttal period is almost over and you have not yet responded to my review and the open questions. While some of my questions have been answered in the other responses, I would still appreciate a response.

Best wishes, Your reviewer

I have two concerns ... good ablations were considered.

We appreciate your concern about scalability. However, our current focus is on settings where the test set size does not consist of “millions” of observations, although we agree this could be an interesting follow-up work. Optimal Transport is a very active research area and we believe adapting this part of RoPE to more specific setups, such as for very large test sets, should be valuable. For instance, in Appendix H, we briefly discuss how RoPE could be extended to handle larger sets of observations. This goes beyond the scope of our paper which rather aims to provide a simple and robust framework to handle model misspecification and to demonstrate that this strategy significantly outperforms others on a set of representative benchmarks.

We use the simulated samples to (1) train the NPE, and (2) have a pool of samples to perform the OT matching during inference. Regarding (1), there is no more constraint than in the standard SBI settings there, as we aim to use as many simulations as possible to learn a posterior distribution that approximates the one implicitly defined by the simulator and prior distribution as well as possible. Regarding (2), in our experiments we observed that using a simulation set of similar size as the test set yielded good results on the benchmarks considered. Nevertheless, we agree that there might be marginal gains by increasing the size of that simulated dataset and that would increase the computational complexity of running RoPE. In that case, exploring a version of RoPE based on amortized OT transport is a promising avenue.

Regarding the dimensionality of the considered model we would like to clarify two points. First, our empirical validation is significantly more extended than concurrent work (e.g., [4] and [5]) aiming to solve the issue of model misspecification in SBI, which typically consists of synthetic settings with one or two target parameters. Second, as it is common in SBI, we explicitly mention the parameters of interest, but the simulator (e.g. task F) often depends on a much larger set of nuisance parameters that are not of direct interest but impact the simulations and add complexity to the modeled generative process. We would like to emphasize that there exist many applications where the goal is to accurately infer a low dimensional parameter while quantifying uncertainty. For instance, when assessing the cardiovascular health of a subject, doctors only check a few vital parameters (e.g, central blood pressure, arterial stiffness or cardiac output, see related work in introduction). We believe that providing tools to create new measurement techniques to infer such low-dimensional quantities, from imperfect models and labeled calibration sets, is of great value for many domains of science and engineering. This low dimensionality also aligns well with the most prominent applications of SBI where, while the forward model is often very complex, the parameters of interest are much lower dimensional (e.g, neuronal activity tracking [1], Higgs discovery [2], or Gravitational analysis [3])

We hope this addresses your concerns, but please let us know which claims you feel are still over the top.

[1]Gonçalves, Pedro J., et al. "Training deep neural density estimators to identify mechanistic models of neural dynamics." Elife 9 (2020): e56261.

[2]Brehmer, Johann, et al. "Mining gold from implicit models to improve likelihood-free inference." Proceedings of the National Academy of Sciences 117.10 (2020): 5242-5249.

[3]Dax, Maximilian, et al. "Real-time gravitational wave science with neural posterior estimation." Physical review letters127.24 (2021): 241103.

[4]:Huang, Daolang, et al. "Learning robust statistics for simulation-based inference under model misspecification." Advances in Neural Information Processing Systems 36 (2024).

[5]:Ward, Daniel, et al. "Robust neural posterior estimation and statistical model criticism." Advances in Neural Information Processing Systems 35 (2022): 33845-33859.

We are sincerely thankful for your review and we understand some of your concerns. We hope our discussion below will convince you that those concerns can be addressed in the scope of small revisions. We also believe, that some of your criticism is subjective and does not directly concern our contribution but the types of real-world problems we versus you are interested in solving (see lines 44-78). We hope to find common ground where our contribution can be seen as a sound and original step towards better solutions to address model misspecification in the context of simulation-based inference.

The basic premise of the work ... without any guarantees that is finite.

We agree with you there are settings–such as the ones you describe–for which our solution would not be of direct use. This is why we explicitly acknowledge this limitation in the second paragraph. Nevertheless, SBI is also often used in a context where 1) we have access to a model in the form of a simulator, 2) the model’s parameters exist beyond the scope of the considered model (e.g., they represent physical quantities), and 3) the goal is to infer their value given the simulated measurement. We also agree that the “true prior” and “posterior” are artificial objects that are, again, a model of the real world. Nevertheless, we use these objects to represent an oracle solution that would be optimal, which represents the best inference we would be able to do if we had infinite access to the real-world generative process. We believe it is important to make a certain distinction between the “prior” and the “likelihood model” in our context as 1) these two objects are instantiated very differently in the context of SBI (one is just a marginal distribution over parameters, the other is a black-box computer program) and 2) the parameter $\theta$ and measurements $x$ have very concrete interpretations. The (true) prior defines the marginal distribution of parameters in the real world, which can be estimated, e.g., by a large set of labels, but does not necessarily require paired measurement. In contrast, the likelihood model is represented by the computer program (as a stochastic map from parameter to observations).

The authors are quite selective ... coupling-based normalizing flows.

We agree [1-7] are relevant work, as they also discuss and aim to address the issue of model misspecification, that are worth being discussed in the scope of our paper. However, we would like to emphasize that in many situations where SBI is relevant, the parameters are interpretable (e.g., these parameters are given physical units). It may be expected that the practitioner would like a solution to model misspecification that does not hurt this interpretation. In our paper, we suggest that it is possible to keep this interpretability with a calibration set where this interpretability is respected. In contrast, we believe the solutions proposed in these works are not viable in the case where the model’s parameters are of direct interest, as they do enforce in any way that this interpretation is kept. We are doubtful there exists a non-data-driven way of addressing model misspecification when keeping the parameter interpretation as needed in the settings we define in lines 44-78. We will add a discussion on these papers and believe this could indeed help the reader to better understand our contribution.

While we agree it is worth adding [9], we do not believe reference [8] is misleading. It introduces monotonic transformations parameterized by neural networks and demonstrates that these networks are universal approximators of continuous monotonic functions. Following this, authors can conclude that autoregressive normalizing flows that rely on these transformations can represent arbitrarily expressive classes of continuous density approximators (see “Universality” paragraph before section 4). To the best of our knowledge, this is the first work to claim that NF are universal density estimators of continuous random variables. We also rely on autoregressive NF, so results about coupling-based NF are less relevant.

The presentation of the results ... from task E).

We appreciate your feedback but believe the results section may be hard to follow because our evaluation is extremely complete and we try to set high standards for future works aiming to address model misspecification in SBI. We aimed to present our results objectively by contextualizing with several contenders, ablations and baselines. We believe this picture, while dense in information, also clearly highlights that RoPE systematically outperforms alternative strategies, providing informative posterior distributions that are well-calibrated. We also believe plotting the difference between simulated and real-world data is important as it provides some context to the reader to understand that we do not only consider “toy” or “naive” misspecification. Do you have any specific advice to make our results easier to read without losing information?

What do you mean by “sushi” plot?

Can the authors explain the insensitivity of performance to the size of the calibration set?

We believe these results demonstrate that combining a supervised fine-tuning step together with an OT step, which ensures the observed and simulated marginal distributions of the data are matched to each other, is an effective way to exploit both supervised and unsupervised data. Nevertheless, RoPE is not perfect and there may be multiple reasons for the performance to quickly saturate. Indeed, if the simulator is extremely good or bad, it may respectively make the issue of misspecification very easy or impossible to address by RoPE. In this case, we would observe a plateau. In addition, the RoPE fine-tuning strategy is simple and may be suboptimal in cases where the calibration set size is larger, explaining why performance plateaus on some benchmarks. Nevertheless, RoPE systematically outperforms baselines, yielding well-calibrated and informative posterior distributions. The RoPE framework could be improved for particular use cases by adapting the fine-tuning step and ensuring it extracts as much information as possible from the real data, while aligning them with the sufficient statistics from the simulated data.

Can the method be extended to incorporate a less-than-perfect calibration set (e.g., as coming from a well-calibrated posterior estimator from another data set)?

Yes, we believe that exploring different fine-tuning strategies, tailored to specific scenarios, are interesting future directions. For instance, the current loss could be adapted to use samples from the well-calibrated posterior instead of exact values. A potentially reasonable fine-tuning loss could be: $\mathcal{L}(\varphi;\mathcal{C}, p^\star) := \sum\_{i=1}^{N_c} \lvert \mathbf{g}\_{\varphi}(\mathbf{x}\_o^i) - \mathbb{E}\_{\theta \sim p^\star(\theta \mid (\mathbf{x}\_o^i)}\mathbb{E}\_{\varepsilon \sim \mathcal{U}[0, 1]}[ \mathbf{h}\_{\omega^\star}\left(S(\theta, \varepsilon)\right) ] \rvert\_2,$, where $\mathcal{C}$ only contains real-world observations and $p^\star$ is this well-calibrated posterior.

We are thankful for your careful feedback. We provide below clarifications and address the concerns you raised.

I have concerns ... Please let me know if I am missing something.

We would like to emphasise that the goal of our paper is to study and propose a novel method for addressing the issue of model misspecification in SBI and is not about addressing the issue of posterior calibration as studied in [1-3]. While we agree the latter issue is important and has not been entirely solved, there exist reasonable solutions to cope with this problem (when ignoring model misspecification), as proposed in [2], [3], or by increasing the simulation budget and optimizing the neural network hyperparameters. Moreover, the miscalibration of a surrogate posterior model can be directly measured and potentially addressed with the aforementioned strategy. Handling model misspecification is a distinct issue as it can only be observed via the gathering of real-world data. Thus our main goal is not to obtain a perfectly calibrated posterior distribution, as we agree this may be arbitrarily difficult even in the well-specified SBI setting. Instead, we aim to create predictive models that 1) exploit the information present in the simulator, even if it is misspecified, 2) are informative (i.e., do not simply output the prior distribution), and 3) are reasonably well calibrated, such that the predicted uncertainty is not overly overconfident.

As a remark, in Appendix F, we do not assume the SBI model is calibrated but “marginally calibrated”, which simply means the expectation of the posterior distribution over the marginal distribution of the observation matches the prior (as defined by eq 9). This property is arguably easier to check and might also be easier to ensure than the standardly checked average “per-observation calibration”. From your question, we understand our phrasing might have been misleading and wrongly interpreted as a claim to solve the issue of calibration in SBI. We will ensure to add the necessary context for the reader to understand why we highlight this “self-calibration” property.

[1] Hermans et al. (2022). A Crisis In Simulation-Based Inference? Beware, Your Posterior Approximations Can Be Unfaithful. TMLR.

[2] Falkiewicz et al. (2023). Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probability. NeurIPS.

[3] Delaunoy, Arnaud, et al. "Towards reliable simulation-based inference with balanced neural ratio estimation." Advances in Neural Information Processing Systems 35 (2022): 20025-20037.

Looking at how the authors ... instead of NPE-RS and NNPE.

We believe your confusion might stem from some language issue caused by the double use of the same word “calibration” that we make along the paper and refers to the calibration of the posterior distribution and is also used to denote the set of labelled data RoPE uses to address misspecification. Nevertheless, we believe our work is about addressing the issue of model misspecification, instead of posterior calibration. Indeed, the posterior is a direct by-product of the model, which we see as defining the joint distribution between parameters and observations, and when we define misspecification as a gap between two posterior distributions, this can be equivalently seen as a gap between two generative processes: the one described by the simulator with a prior distribution and the one observed in the real world. Applying calibration methods such as introduced in [1] and [2] would not solve the problem we are interested in as these methods only focus on the simulator and do not consider potential gap between real-world and simulated samples. Our experiments also focus on such “misspecification scenarios” as exemplified by Task E, where the data simulated and real-world measurements are drastically different. We agree that under no model misspecification, it could be interesting to study whether RoPE can help improve the calibration of NPE; however, this is not the goal we target in the paper. We believe the second paragraph of the introduction provides an explicit mention of use cases we are interested in and why these are related to model misspecification. We do not believe [2] will provide a viable solution in our context as the method relies on Monte Carlo estimate of model calibration, which itself necessitates large sets of labelled data. [1] and [2] use the simulator to generate such labelled data. In contrast, we can only real-world labelled data, which are more expensive to collect. Extending these methods to handle model misspecification as we define it is an open research direction.

The equation in line 190 ... and the implications for when it does not hold.

The main difference is that NNPE is very restrictive on the class of model misspecification, whereas RoPE can theoretically learn to be robust to any model misspecification that respects eq 3. The spike and slab model considered in NNPE is very restrictive and does not provide a viable solution for many real-world use cases where a practitioner would like to exploit domain knowledge in the form of a simulator to infer the properties of a real-world system. We observed this weakness in our empirical results (solid orange line in in Figure 2). It is, for instance, pretty clear that x_s and x_o in Task E are not related to each others via a spike and slab model. The equation in line 190 is correct if and only if eq. 3 is satisfied. Indeed, $p(\theta \mid x_o) = \int p(\theta, x_s \mid x_o) dx_s = \int p(\theta \mid x_s) p(x_s \mid x_o) dxs$

$\iff p(\theta \mid x_s) = p(\theta \mid x_s, x_o)$

$\iff p(x_o, \theta \mid x_s) = p(x_o \mid x_s) p(\theta \mid x_s, x_o) = p(x_o \mid x_s) p(\theta \mid x_s)$

$\iff x_o \perp \theta \mid x_s$ by definition of conditional independence and following Bayes rule. Thus NNPE, although it does not explicitly state it, makes the same assumption. Thus this modelling assumption is necessary to make the factorization we use correct. If the assumption is wrong, there might be additional information flowing from $\theta$ directly to x_o that our predictive model would not be able to exploit. This assumption is a modelling assumption and its relevance is thus problem dependent. Overall we believe that a well-designed simulator should represent a robust relationship between the parameters and the observation and thus using this assumption should be beneficial to make predictions that are robust to environment / distribution shifts, as we explain in lines 192 to 200.

The notation is unclear at times ... which is not introduced.

In lines 81-90 we use the notation $\mathbb{P}^\star$ to introduce the standard definition of model misspecification, as is usually formalized in the literature. From line 90 on we are introducing our extended notation for model misspecification, whose necessity we justify also with the example in Appendix B. $p^\star(\theta \mid x_o)$ is the density of the “true” posterior distribution of the parameter given an observation in the real world. This follows implicitly from $p^\star(\theta)$ and $p^\star(x_o \mid \theta)$ that are introduced to represent the “true” marginal distribution of parameters and “true” conditional distribution of observation given the parameter value.

In section 2.2 ... to understand every detail.

We could add the following intuition: An optimal transport (OT) coupling is a mathematical object that represents the most efficient way to associate two probability distributions while minimizing a cost function that measures the "distance" between samples drawn from one distribution and their counterparts in the other.

Minor comments:

Indeed, thank you for noticing these typos.

Model misspecification & introducing $p^\star(\theta \mid x\_o)$

Thanks a lot for your follow up thoughts, they really helped us understand your very valid concerns. We have reformulated our definition of model misspecification (lines 73-78) in an equivalent way that illuminates these points better. We write our new definition here, and address your comments below. [New definition start] First, we assume the calibration set $\{(\theta^i, \mathbf{x}\_o^i)\}\_{i=1}^{N\_c}$ of real-world observations $\mathbf{x}\_o \in \mathcal{X}$ and their corresponding labels $\theta \in \Theta$ are sampled i.i.d. from a joint distribution given by the density $p^\star(\theta,\mathbf{x}\_o)$. Let $p^\star(\theta)$ be the marginal density of the underlying parameters $\theta$ in the real world, and $p^\star(\mathbf{x}\_o) := \int_\Theta p^\star(\theta) p^\star(\mathbf{x}\_o \mid \theta) d\theta$ be the marginal density of the real-world observations, where $p^\star(\mathbf{x}\_o \mid \theta)$ is the unknown process which is modeled by the simulator $p(\mathbf{x}\_s\mid \theta)$. We say the simulator is misspecified if $\exists \mathcal{S} \subseteq \Theta\times\mathcal{X}$ with $\iint\_\mathcal{S} p^\star(\theta, \mathbf{x}) d\theta d\mathbf{x} > 0$ such that $p^\star(\mathbf{x}\_o\mid\theta) \neq p(\mathbf{x}\_s=\mathbf{x}\_o\mid\theta)$. [New definition end] This is equivalent to the old definition, if we had explicitly introduced $p^\star(\theta \mid x\_o)$ and $p(\theta \mid x\_o)$ (like you suggested) as: $p^\star(\theta \mid \mathbf{x}\_o) := \frac{p^\star(\theta)p^\star(\mathbf{x}\_o \mid \theta)}{p^\star(\mathbf{x}\_o)}$ and $p(\theta \mid \mathbf{x}\_o):= \frac{p^\star(\theta) p(\mathbf{x}\_s = \mathbf{x}\_o \mid \theta)}{p^\star(\mathbf{x}\_o)},$ which we agree should have been more explicit in our original definition.

In light of this reformulated definition, let us discuss the reasons for misspecification (1-3) that you outline. Regarding reason (1): note that our definition is stronger: even when the true data generating process does belong in the family of distributions induced by the simulator, the simulator may still be misspecified under our definition (as we showed in the example in Appendix B), because we require that for a fixed value of $\theta$---instead of “some” value of $\theta$---the simulator matches the distribution of the real data. Is this clearer now?

Regarding point (2): under this definition, if the simulator is misspecified it will produce the wrong distribution even when the prior used for inference is the true prior.

Regarding point (3): indeed, even without misspecification, the estimated posterior can be misscalibrated due to the inference tool (e.g., NPE) that one employs, as shown in the references you suggested. However, this point is orthogonal to the above definition, i.e., it is a problem of the estimator even when the simulator is correctly specified. In our definition, we do not talk about the estimation of the posterior but consider its “oracle” form as a by-product of the simulator and the prior distribution. We understand your confusion and would like to clarify this more explicitly in the revised version of the manuscript. Do you think this is reasonable?

Thanks for the further questions and remarks.

Typo in the numerator:

You are correct; there was a typo in our previous response regarding the numerator for the posterior's simulator. It should read: $p(\theta \mid x\_o) := \frac{p^\star(\theta) p(x\_s = x\_o \mid \theta)}{\int p^\star(\theta) p(x\_s = x\_o \mid \theta) \text{d}\theta}$ Apologies for the confusion. As a note, this typo is not present in the updated manuscript.

Reasons to use a different definition:

You are right to point out that $\theta$ plays a key role in our definition. Unlike the misspecification definition that Ward et al. 2023 and Huang et al. 2023 use, we assign a clear interpretation to $\theta$ (often as a physical quantity) and aim to preserve it even under misspecification. We achieve this by assuming the calibration set provides trustworthy labels, as is realistic for many real-world scenarios. Appendix A2 provides an example highlighting this distinction. For instance, if the true generative model is $\mathcal{N}(\theta, 1)$ but the simulator assumes $\mathcal{N}(\theta + \alpha, 1)$ (where $\alpha$ is the misspecification), traditional definitions might consider this model well-specified, whereas our approach accounts for the semantic importance of $\theta$. As you noted in your last comment, we agree that it is possible to reconcile the two definitions by re-stating what are the model parameters and observations.

The goal of re-stating the definition is to clarify the settings we are interested in: predicting $\theta$ accurately (so that it can be used for downstream applications/decisions that assumes $\theta$ has a certain meaning), even when the simulator is misspecified. With our definition, we also discard prior misspecification and instead focus on issues with the simulator (though we also evaluate robustness of our algorithm in presence of prior misspecification). In contrast, eq. (3) in the paper is a useful modeling tool—even if imperfect—that motivates RoPE and whose positive and negative consequences are discussed in the paper.

Distinguishing assumptions:

The first assumption, as long as $p^\star(\theta)$ and $p(x\_o \mid \theta)$ are properly defined, is inherently valid. In contrast, the second assumption (no direct dependence between $\theta$ and $x\_o$ beyond what is modeled by the simulator) is a practical modeling simplification. We can provide an example where this assumption holds and another where it does not, to clarify its scope:

Example where eq 3 is respected:

$\theta \in \mathbb{R} \sim p^\star(\theta) \quad x\_s \in \mathbb{R} \sim \mathcal{N}(\theta, 1) \quad x\_o = x\_s - \alpha \iff x_o \in \mathbb{R} \sim \mathcal{N}(\theta - \alpha, 1)$

Example where eq 3 is not respected:

$\theta \in \mathbb{R} \sim p^\star(\theta) \quad x\_s \in \mathbb{R}^2 \sim \mathcal{N}([\theta, 0]^T, I) \quad z \sim \mathcal{0, 1} \quad x_o = x_s + [0, \theta * z]^T - \alpha \in \mathbb{R}^2 \iff x_o \sim \mathcal{N}([\theta, \theta] - \alpha, 1)$

Reconciling perspectives:

This is a great point and we also considered presenting the perspective of treating $(x\_o, \theta)$ as a joint observation to reconcile the two definitions. However, we believe this perspective obscures the distinct roles of $x$ (observed at test time) and $\theta$ (the quantity to predict) that is important in our setting. We were afraid bringing this into the discussion might be more confusing and this is why we avoided it but we also appreciate this dual view and we would be glad to incorporate this perspective in Appendix A2 if you think it would be helpful.

Please let us know if you have further questions! Thanks a lot for the very constructive discussion.

Thank you for the quick response. I will summarize my thoughts now and won't take up more of your time.

We are very grateful for this interaction, for having spent so much time reading our rebuttal, and overall, for your interest in this discussion.

I think anything that helps the reader understand your setting better would be nice to include, including these simple cases, dual view, etc. that we have been discussing. Otherwise you get readers like me who just get thrown off by the definition an can't move past it.

We seized the opportunity given by this rebuttal to improve, through what we believe are a few minor cosmetic modifications, the clarity of our paper. This is in part thanks to you, and the time you have spent reading, understanding and criticizing our paper.

Unfortunately, I won't be increasing my score, as I don't think the paper is ready yet (our entire back and forth about the definition is a signal for this).

We of course respect your opinion. We have benefited from this interaction nonetheless, and we believe the paper is better now than it was when first submitted.

The paper does not properly motivate why using optimal transport is the right thing to do. Given your setting/definition/assumptions, there could potentially be many possible solutions to this problem. Why is the proposed solution the right one? I understand that you can model misspecification with OT, and doing so helps in the examples you have in Section 4. But the paper directly jumps from the problem setup to the solution without telling where the solution came from.

We acknowledge this criticism, and agree that a proper justification on why OT is an effective solution only arrives fairly late in the paper (Section 3.2). We understand this concern and have clarified the motivation for using OT earlier in the text (L.94–98 and L.187–188) to guide the reader more effectively, before they reach section 3.2.

There needs to be more discussion regarding the hyperparameter and how to set it in practice. At best, you should provide an algorithm, or at the least some recommended guidelines/heuristics based on analyzing its effect.

We argue that RoPE has a fairly limited set of hyperparameters (entropic regularization $\gamma$ and unbalancedness regularization $\tau$), which are very classic in all regularized OT works (see for instance [1-5]). Our parameter choice for both has been guided by the extensive literature and software implementations that exist (here, in particular, OTT-JAX [6]). We use the same fixed values of $\gamma$ and $\tau$ throughout the 6 benchmarks (L.406-410). With these settings, RoPE performs consistently well across all benchmarks and outperforms NNPE and RNPE. We believe that this is more convincing, from a practitioner's perspective, than a hyperparameter search that would need to be tuned for each experiment.

Furthermore, we encourage you to check in more detail the several experiments in which we study the effect of these parameters (lines 463–508, as well as Figure 3 (right panel) and Figure 4). To assist practitioners, we explicitly provide advice on how to set these parameters within the manuscript: L.470–473 for $\gamma$ and L.485–508 for $\tau$. We hope this addresses your concern and offers useful guidance for practical applications.

Again, regardless of your final score, we would like to thank you for the time you have spent reading, criticizing and helping us improve the submission.

[1] Peyré, Gabriel, and Marco Cuturi. "Computational optimal transport: With applications to data science." Foundations and Trends® in Machine Learning 11.5-6 (2019): 355-607.

[2] Genevay, Aude, Gabriel Peyré, and Marco Cuturi. "Learning generative models with sinkhorn divergences." International Conference on Artificial Intelligence and Statistics. PMLR, 2018.

[3] Fatras, Kilian, et al. "Unbalanced minibatch optimal transport; applications to domain adaptation." International Conference on Machine Learning. PMLR, 2021.

[4] Pham, Khiem, et al. "On unbalanced optimal transport: An analysis of sinkhorn algorithm." International Conference on Machine Learning. PMLR, 2020.

[5] Chizat, Lenaic, et al. "Unbalanced optimal transport: Dynamic and Kantorovich formulations." Journal of Functional Analysis274.11 (2018): 3090-3123.

[6]: Cuturi, Marco, et al. "Optimal transport tools (ott): A jax toolbox for all things wasserstein." arXiv preprint arXiv:2201.12324(2022).