Addressing Misspecification in Simulation-based Inference through Data-driven Calibration

We sincerely appreciate your review and your recognition of our contribution to the literature on simulation-based inference (SBI) under model misspecification.

Regarding your concern about the availability of labeled real datasets, we acknowledge that such datasets are not always accessible. However, as discussed with Reviewer x5KJ, moving away from labeled calibration data in method papers introduces the challenge of defining alternative evaluation metrics. Without access to true parameters, validating the reliability of a method becomes significantly more difficult. That said, we agree that exploring approaches leveraging unlabeled real datasets is an interesting direction for future work.

We will incorporate the suggested references on increasing robustness in SBI and check whether they contain additional relevant citations on misspecification.

Thank you again for your thoughtful feedback and your time.

Thank you for the thoughtful and constructive feedback. We appreciate your insights and will use them to improve the final version of our manuscript. Below, we detail the modifications we plan to make in response to your comments.

Figure 1 is overly dense

We acknowledge that Figure 1 contains too much information and will revise it to improve readability. We will use the additional space to split the figure in two. This will allow us to make the individual plots larger and have space to add the method labels directly on the plots, making them easier to parse. We will also implement the suggested changes.

Clarification on NLTP and LPP

We confirm that NLTP and LPP are exactly the same metric. We will make this explicit in the text to avoid confusion.

ACAUC and joint calibration

ACAUC quantifies marginal calibration by measuring the gap between empirical and expected coverage, averaged over confidence levels. However, it does not capture joint calibration, as dependencies between parameters are not explicitly assessed. Alternative dependence-sensitive metrics may require larger test sets to be stable. Given this limitation, we will explicitly discuss the trade-offs of ACAUC in the revised manuscript.

SBI baseline details

There is a formal description of the SBI baseline in the appendix (lines 965-970). Do you think it should be moved to the main text? The SBI baseline is trained and evaluated purely on simulated data. Since RoPE assumes a conditional independence structure, the SBI baseline should provide an upper bound on RoPE’s performance in terms of LPP.

Performance of the MLP baseline

The strong performance of the MLP baseline is likely due to the unimodal nature of the benchmark posteriors, where a Gaussian is a reasonable approximation. However, this baseline only performs well when given the largest calibration budget. For smaller budgets (200 and below), RoPE consistently outperforms it. Extending benchmarking tasks for SBI under misspecification with multi-modal posterior would be an interesting direction for future work.

Performance variations of RoPE

RoPE consistently returns calibrated and informative posteriors, as indicated by its LPP scores being higher than those of the prior. The results in Figure 1 are based on a fixed entropy regularization parameter, but in practice, this parameter would be tuned. With the chosen setting, RoPE can sometimes be slightly underconfident while remaining well-calibrated across all tasks. For Task E, using a smaller regularization value ($\gamma = 0.1$ or lower) resulted in stronger performance than the one we reported in the manuscript (with $\gamma=0.5$). We expect RoPE to match SBI closely on standard benchmarks when $gamma$ is small. Investigating further the effects of $\gamma$, parameter dimensionality, and test set size further would be interesting but probably goes beyond the scope of this work. Finally, we emphasize that when a simulator is well-specified, standard SBI methods should be preferred over RoPE, as RoPE is specifically designed for misspecified settings.

The need for a labeled calibration set

RoPE relies on a calibration set with ground-truth parameters, but as shown in our experiments, it can also be used without the fine-tuning step. In most benchmarks, this "OT-only" version already produces well-calibrated posteriors. However, the labeled test set was crucial in identifying that OT-only was insufficient for Task E. In a misspecified setting, learning a new model is necessary, and empirical data is required to validate its reliability. This justifies our assumption that a small set of real observations could be used for training. We nevertheless agree with you that labeled calibration data is not always available in real-world applications. In such cases, practitioners must define metrics that assess whether OT-only provides sufficient performance. In some applications, a fine-tuning step similar to RoPE’s could still be justified without labeled data, but this would require application-specific validation strategies. We will clarify these points in the final version of the paper.

We sincerely thank the reviewer for carefully reading our paper and appreciate the very constructive feedback regarding the plausibility of finding a calibration set sampled from $p^\star$ in practical settings. We now discuss this question and other comments in detail.

Source distribution of the calibration set

We agree with the reviewer that in some settings, assuming the calibration set consists of samples from the distribution $p^\star(\theta, x_o)$ may be unrealistic. Nevertheless, in important practical scenarios, it may sometimes be possible to influence the calibration set collection process to ensure it is representative of the test case scenario. In the following discussion, we denote an “imperfect” calibration set distribution as $p^c(\theta, x_o)$.

Case 1: $p^c(\theta, x_o)$ and $p^*(\theta, x_o)$ have the same support

In this case, as the size of the calibration set increases, the fine-tuning step should still be able to correct the estimated sufficient statistic for all observations in the test set. If the calibration set is large, one could even perform importance sampling to ensure that the fine-tuning step optimizes the expected loss on the ideal calibration set distribution. However, for smaller calibration sets, the sensitivity of RoPE depends on whether the fine-tuned neural statistic estimator (NSE) can generalize to unseen observations. This is highly dependent on the application, particularly regarding the non-linearity of the model misspecification and the inductive bias of the NSE. If the NSE struggles to generalize, we expect the entropy-regularized optimal transport (OT) step can bring some additional robustness. Specifically, observations where the fine-tuned NPE performs well will naturally be matched to the appropriate simulated statistics, leading to accurate posterior estimates. In contrast, for observations where the sufficient statistic does not generalize well, their representation in the embedding space may become unstructured. As a result, applying OT to these embeddings may lead to random matches with the simulated statistics, causing the posterior to revert to the prior distribution, filtered by the parameter space where the NPE is reliable.

Case 2: $p^c(\theta, x_o)$ and $p^*(\theta, x_o)$ have disjoint support

In this more extreme case, even an arbitrarily large calibration set may fail to provide RoPE with relevant training examples for test observations. Consequently, the fine-tuning step relies entirely on out-of-distribution generalization, which is highly problem-dependent. If the fine-tuned NPE cannot generalize at all to the test set, the fine-tuning process may be counterproductive. However, even in this situation, we expect the OT step to highlight this issue. Since real and synthetic observations will not match meaningfully, the transport matrix should approach a uniform and the posterior will revert to the prior.

While this second scenario is particularly challenging to study, we propose to approximate "Case 1" in a controlled experiment on the Light Tunnel task. Specifically, we will construct a calibration set using samples from Prior B (as in Figure 2) and evaluate performance on a test set drawn from Prior A.

If the paper is accepted, we plan to include this experiment in the appendix and add a discussion in Section 6 to highlight RoPE’s sensitivity to non-representative calibration sets, following the considerations outlined above.

On the failure of RoPE when the simulator is increasingly simple

We agree that further investigating the trade-offs between model complexity and robustness to misspecification is an important research direction but goes beyond the scope of our paper. In many applications, there exists a trade-off:

• Simplistic simulators may ignore complex relationships between parameters and observations but have structural properties that generalize well to real-world data. SBI models trained on these simulators may be less sensitive
• Highly complex simulators may capture intricate relationships but risk overfitting to simulation-specific artifacts, failing to generalize under slight misspecifications.

For RoPE, because it assumes conditional independence $x_o \perp \theta \mid x_s$, it is better suited for handling the latter case rather than highly simplistic simulators that fail to encode meaningful parameter-observation relationships. The conditional independence assumption acts as a backstop to shortcut learning, but for the same reason, RoPE’s posterior informativeness is inherently limited by the informativeness of the posterior learned from simulations, as transparently discussed in Section 3, Paragraph 2.

Minor comments & typos

We appreciate the reviewer’s attention to detail in pointing out minor phrasing inconsistencies and imprecisions. We will carefully edit the manuscript to address these comments.

We sincerely thank the reviewer for their thoughtful and constructive feedback. We greatly appreciate the careful assessment of our work and the valuable insights provided. Below, we address the key points raised.

Additional References

We completely agree that RoPE is closely related to semi-supervised learning and Sim2Real literature. If accepted, we will use part of the additional page to elaborate on these connections.

Setting Limitations

As demonstrated in our experiments, in the absence of a calibration set, the version of RoPE without fine-tuning (OT-only) provides a viable way to handle model misspecification in some benchmarks. Nevertheless, as noted in the introduction, when posterior estimates need to be reliable, a labeled dataset is crucial for empirical validation and it appears thus natural to save a subset of this dataset for training purposes. We also recognize that settings where no labeled data is available exist. However, evaluating methods in such scenarios is highly application-dependent as it requires appropriate unsupervised metrics and we believe it may be difficult to create one generic method to handle these diverse set of problems. While we hope that RoPE’s fine-tuning step can be adapted to leverage unsupervised data and metrics specific to these applications, we acknowledge that this remains an open question. We believe future work focused on adapting RoPE to these real-world use cases could be highly impactful.

Regarding scalability, as acknowledged in Section 5, we have not yet explored RoPE’s performance in medium- or high-dimensional parameter spaces, and its ability to correct for misspecification in those settings remains an open challenge. As the reviewer pointed out, model misspecification in SBI is a complex issue, and it is unlikely that any single method can address all possible cases.

Conditional Independence

We appreciate the reviewer’s careful consideration of RoPE’s assumptions. As noted in Section 3, the independence assumption is indeed a strong constraint. It limits RoPE’s ability to uncover dependencies between observations and parameters that are not explicitly captured by the simulator. If this assumption does not hold, RoPE cannot recover the true posterior, even with an arbitrarily large calibration set. However, RoPE will approximate the posterior by leveraging the simulator’s known dependencies, which may contribute to robustness and interpretability. Furthermore, please note that this conditional independence assumption does not hold for the two real-data testbeds (tasks E and F), but RoPE still returns informative posterior estimates.