Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation

Thank you for the pertinence of your review. Your feedback is very valuable, especially since your background gives us different perspectives on our work.

W1 & W3 (Insights) Indeed, our empirical findings raise many questions, which prompt the need for theoretical explanations. We believe that this is precisely the goal of empirical research, and hence a strength of our work. Nevertheless, we agree that the lack of explanation could leave the reader puzzled. We provide two non-exclusive hypotheses that could explain our findings:

1. As explained in Section 5, many models, while accurate for short-term prediction, exhibit long-term instabilities as errors accumulate, pushing the predictions out of the training data distribution. Generative models, particularly diffusion models, partially address this problem [18, 19, 22–25] as they produce statistically plausible states, even when they diverge from the ground-truth solution.

   However, distribution shift doesn't explain everything. At the first prediction step, before distribution shift can take effect, diffusion models are already slightly better than deterministic neural solvers. This could be explained by the iterative nature of DM sampling which allows the model to refine its prediction.

2. For many physics, and especially fluids, dynamics are mainly explained by large-scale/low-frequency content, while small-scale/high-frequency content influences the (very) long-term evolution of the system. Learning to process small-scale content and predict its influence on long-term dynamics might require very large networks, as well as huge amounts of data and compute. When the latter are limited, it becomes more tractable to focus on larger scales. Therefore, by discarding some of the small-scale content, latent diffusion models are able to learn and reproduce the dynamics more easily than pixel-space models.

Validating these hypotheses is an interesting avenue for future research. We would appreciate your feedback on these hypotheses and whether we should include them in the manuscript.

W2 (Link to Video Generation) There are definitely links with the image and video generation literatures. We already make connections in the discussion, especially regarding latent image generation which works much better than in pixel-space. Thank you for the blog post, we read it with attention. Note that we already refer to a blog post by Dieleman [56], although a different one. We will further look into the video diffusion model literature regarding the effects of compression.

However, there is one very important difference between dynamical systems and videos. Dynamical systems are Markovian, meaning that knowing the current state is enough to predict the future (or the past). Therefore, there is no point in remembering the history, unlike for videos where it is very important.

W4.1 (Color map) Thank you for your feedback. We will try to use a color map that changes in intensity.

W4.2 (Well Calibrated) We are not sure to understand what you find confusing, but we are appreciate this opportunity to clarify our manuscript. Could you explain your perspective more precisely?

From our perspective, the spread-skill ratio is a measure of calibration. Spread represents the uncertainty of the model. Skill represents the error of the model. For a well calibrated model, spread should be equal to skill (and the ratio be 1), meaning that the model should be as uncertain as it is wrong.

W4.3 (Equations Readability) We will make sure to improve the readability of the equations in the camera-ready version.

W4.4 (Latent Saturation) Yes, this is quite arbitrary. We tried a few saturating functions (which you can find in the code) in our preliminary experiments and all worked similarly. We used this one as it has a smooth derivative and saturates very slowly, which prevents vanishing gradients. $B = 5$ is chosen such that the values range between $-5$ and $5$, which is close to the range of a standard Gaussian. With these choices the mean and variance of the latents are close to 0 and 1 (empirically). Therefore, we do not need to re-scale the latents before training the diffusion model. Conversely, DCAE does not use this saturating function, but relies on rescaling the latents to train the latent diffusion model.

W4.5 (Figures) Thank you again for your feedback. Indeed, these figures could be misleading. We will improve them in the camera-ready version, but cannot provide them in this rebuttal, as per NeurIPS guidelines.

In most of our experiments, the latent emulators are only conditioned on a single past frame. However thanks to our masking strategy, we are able to test whether conditioning on several past frames would help. We already do so in Table 7, where we vary the number of context frames from 1 to 3. We find that context length has little to no impact on emulation accuracy with our latent models.

This is likely due to dynamical systems being Markovian, as explained previously. In a Markov chain, knowing the current state is enough to predict the future. One could argue, however, that in the latent space, the system is not Markovian anymore as some information is lost. The Markov chain becomes a hidden Markov model (HMM). However, the fact that emulation accuracy is robust to compression and independent from context length, contradicts this argument.

Q1 (Applications) There exists many applications for fast emulators. Common ones are weather forecasting and aerodynamic design (cars, airplanes, ...), for which traditional numerical solvers are extremely slow and expensive. Anything relying on simulations for real-time decision making too.

We would like to note, however, that neural networks have other advantages than being fast emulators. They enable other tasks which are impossible with traditional numerical solvers. Notably, neural networks are differentiable, which can be very useful in the case of inverse problems. Diffusion-based emulators can also be guided with partial/noisy observations. Perhaps most importantly, data-driven emulators do not require knowledge of the underlying physics/PDEs, which is extremely valuable in real-world settings.

Q2 (Upscaling) This is called "downscaling" in numerical weather forecasting. This generally doesn't work well because for most fluid systems, numerical solvers fail at low resolution. Therefore "spatially-coarse-grained but accurate" numerical simulations typically do not exist. Besides, upscaling with a diffusion model could lead to "good looking" states, but the additional high-frequencies would not be linked to the past states.

Q3 (Generalization) Whether our models generate to out-of-distribution tasks is very relevant. We propose to add an experiment where the models are used to emulate with parameters $\theta$ that are not present in the training distribution. Our preliminary results suggest that the models generalize to new parameters $\theta$ when these are close to or in between parameters seen during training. However, they fail for completely new parameters, including new types of boundary conditions.

Regarding time extrapolation, unrolling the trajectory past the training length ($i > L$) works quite well qualitatively, although we are not able to assess it quantitatively as we don't have access to data past this length.

We hope that our answers address your concerns. If so, we kindly ask you to reconsider your score.

Thank you for the quality of your review. We greatly appreciate your interest in our findings. We are thankful for the opportunity to improve and clarify our manuscript.

W1 (Interpretability) We believe that the lack of interpretability is a limitation of data-driven emulators, rather than a weakness of our work. Whether interpretability is necessary for emulators, is an open debate, which we believe is outside of the scope of this work. An example where interpretability is not necessary is weather forecasting, where speed and accuracy are much more relevant for consumers.

W2 & Q1 (Missing Baselines) At the time of submission, no emulation results for the datasets of TheWell were available in the literature, other than the preliminary baselines from TheWell's paper [38]. Training our own competing baselines would be unreasonably expensive and lead to fairness concerns. Nevertheless, we agree that a more comprehensive set of baselines would help the reader assess the significance of our results within the context of data-driven emulators. Fortunately, in the past months, several studies have published results for TheWell datasets, which we compile here and will add to the manuscript.

Table A. Average VRMSE results from different studies using TheWell datasets. Even though our latent diffusion models (LDMs) outperform most published baselines, we emphasize that we are not making any claims regarding the novelty of our method and do not position our models as state-of-the-art, notably due to the discrepancies in parameters count, training and evaluation.

Thank you for the PINN reference you provided, we will read it with attention. PINNs are very interesting and a developing area of research. However, we believe that it is unfair to compare PINNs with purely data-driven emulators, which do not require any knowledge of the physics.

Q2 & Q4 (Insights) Indeed, our empirical findings raise many questions, which prompt the need for theoretical explanations. We believe that this is precisely the goal of empirical research, and hence a strength of our work. Nevertheless, we agree that the lack of explanation could leave the reader puzzled. We provide two non-exclusive hypotheses that could explain our findings:

1. As explained in Section 5, many models, while accurate for short-term prediction, exhibit long-term instabilities as errors accumulate, pushing the predictions out of the training data distribution. Generative models, particularly diffusion models, partially address this problem [18, 19, 22–25] as they produce statistically plausible states, even when they diverge from the ground-truth solution.

   However, distribution shift doesn't explain everything. At the first prediction step, before distribution shift can take effect, diffusion models are already slightly better than deterministic neural solvers. This could be explained by the iterative nature of DM sampling which allows the model to refine its prediction.

2. For many physics, and especially fluids, dynamics are mainly explained by large-scale/low-frequency content, while small-scale/high-frequency content influences the (very) long-term evolution of the system. Learning to process small-scale content and predict its influence on long-term dynamics might require very large networks, as well as huge amounts of data and compute. When the latter are limited, it becomes more tractable to focus on larger scales. Therefore, by discarding some of the small-scale content, latent diffusion models are able to learn and reproduce the dynamics more easily than pixel-space models.

Validating these hypotheses is an interesting avenue for future research. We would appreciate your feedback on these hypotheses and whether we should include them in the manuscript.

Q3 (Hallucinations) There are two ways our models can "hallucinate": either the decoder introduces artifacts that are not plausible or the latent emulator generates implausible dynamics.

1. Our decoder is not trained with a perceptual loss. We therefore expect the decoding to have a "smoothing" behavior on the fields, which we observe in practice (high-frequency cutoff). Due to this degradation of small-scale features, some physicality is indeed lost.

   To address this issue, we considered training a generative decoder during this project. However, this would not increase the amount of information compressed in the latent space. The additional small-scale details would be hallucinated by the decoder, without contributing to emulation accuracy, which would be misleading for humans. We never implemented this idea.

   We did not consider introducing physics-based objectives during the training of the auto-encoder, as we assume that the physics are unknown. It is however an interesting research direction.

2. Our latent diffusion-based emulators are not perfect and often diverge from the ground-truth solution. In this light, it can be considered that individual trajectories are "hallucinated". However, our evaluation (using the spread-skill ratio) demonstrates that, as an ensemble, the trajectories are well calibrated, meaning that they generally cover plausible scenarios.

Q5 (Generalization) You are perfectly right, we observed that (L)DMs need to be trained for a long time, and keep improving in the later training stages. Whether these models generate to out-of-distribution tasks is therefore very relevant. We propose to add an experiment where the models are used to emulate with parameters $\theta$ that are not present in the training distribution. Our preliminary results suggest that the models generalize to new parameters $\theta$ when these are close to or in between parameters seen during training. However, they fail for completely new parameters, including new types of boundary conditions.

Q6 (Repeatability) Although diffusion models are generative models, there exist deterministic sampling strategies for them, which always lead to the same output when starting from the same initial noise. In fact, our sampling scheme (Adams-Bashforth, see Section 3.3) is one such strategy.

We believe that this rebuttal addresses most of your concerns and, therefore, kindly ask you to reconsider your score.

Thank you for your time and effort during this rebuttal period. We appreciate your suggestion for an additional experiment. We would like to be sure to understand your concern and what you propose.

Let $x_a^{1:L}$ and $x_b^{1:L}$ be two trajectories with different physical parameters $\theta_a$ and $\theta_b$ (e.g. low vs high viscosity), but starting at the same initial state $x_a^1 = x_b^1$. It seems that you are concerned that the latents $z_a^1 = E_\psi(x_a^1)$ and $z_b^1 = E_\psi(x_b^1)$ would be equal, which is correct. Without knowing $\theta_a$ and $\theta_b$, the emulator would therefore produce the same latent trajectories.

However, in our models, the emulator is conditioned on the parameters $\theta$ and can therefore produce different trajectories with different parameters $\theta_a$ and $\theta_b$ from a single initial state $z^1$.

In this light, we are not sure how to conduct the experiment you suggest. Our framework is not compatible with the framework of the provided reference (Vafa et al., 2025), where the parameters (e.g. mass of planets) are considered to be part of the state.

In addition, there is also a more practical issue with your suggested experiment. The systems you propose do not have the same state representations and (number of) parameters than Euler. We therefore cannot apply it without modifying the architecture of our models and retraining them, at which point we are not sure the experiment makes sense anymore.

Could you please clarify what you expect from us? Would you be satisfied if we took a state $x^i$, and compared the latents $E_\psi(x^i)$ and $E_\psi(x^i + \Delta)$ to show that they are different? Or maybe you are interested in the latents produced by the latent emulator starting from the same initial state $z^1$, but with different parameters $\theta$?

I thank the authors for the discussion. My main point, as mentioned in the paper I cited, is that even if you have a good metric, the model may fail to recover the actual physical law, which is my primary concern. I believe the validation should be more thorough to demonstrate that the model accurately captures the physical laws.

I propose running the model with this group of equations to obtain the proposed metrics and demonstrate that the latents of these equations (as a trajectory) are not close to each other, indicating that the encoder is not mixing physical phenomena.

The equations I want tested are Euler, Incompressible NS under low and high viscosity, and Compressible NS with low and high $\mu$ (dynamic viscosity). Also, add unrelated equations such as KdV, Burgers, and Shallow Water with varying parameters. If you can show that the model latents can distinguish between parameters of the equation, I would be convinced and raise my score to a full score.

Thank you for your patience. Here are the results of the first experiment we suggested.

We take one random trajectory $x^{1:L}$ from the test split of the Euler dataset. We compute the inital latent state $z^1 = E_\psi(x^1)$ for the $\div_{80}$ auto-encoder.

Then, for each heat capacity $\gamma$ in $\{1.2, 1.3, 1.4, 1.5, 1.6\}$, we generate one latent trajectory $z^{1:L}$ with the diffusion-based emulator. Afterwards, we compute the distance $||z_a^i - z_b^i||$ for each pair of heat capacities $\gamma_a$ and $\gamma_b$. We report the results for $i \in \{4, 20, 100\}$. We expect the distance between latents to grow with the difference of heat capacity.

We note that we cannot use the EMD in this case. We were able to use the EMD for reviewer's 52YY experiment as the pressure is a non-negative scalar field that we transformed into a density field. The latents are vectors taking real values.

We hope that these new results address your concerns. We remain available if you have any questions.

Thank you for the quality of your review and interest in our work. We greatly appreciate that you read our manuscript with such attention to details. Your concerns are legitimate and we will address them here and in the manuscript.

Q1 & L1 (Dataset Diversity) We follow your suggestion and add an additional 3D dataset to our study. This dataset (turbulence_gravity_cooling from TheWell [38]) models the interstellar medium as a turbulent fluid subject to gravity and radiative cooling. The 3d state of the system is discretized on a 64x64x64 grid. As for the other datasets, we train several autoencoders with increasing compression rates (from 48 to 768) and latent emulators. The results (see table below) are consistent with the previous experiments and strengthen our conclusions: latent diffusion emulators are robust to compression. Latent neural solvers, conversely, are not as robust.

Table B. Average VRMSE of latent-space emulation at different compression rates (÷) relative to the auto-encoded states $D_\psi(E_\psi(x^i))$ for the TGC dataset.

Unfortunately, we cannot provide plots during the rebuttal, as per NeurIPS guidelines. Further results will be provided in the camera-ready version.

We would also like to mention that the Rayleigh-Bénard dataset models a viscous fluid.

Q2 (Time-Resolution Constraints) We apologize for the confusion and thank you for pointing this out. The time stride was introduced strictly for convenience: with larger strides, emulation is faster. We conduct the new experiment (see above) without time stride ($\Delta = 1$). We also started to retrain our models with $\Delta = 1$. At the time of this rebuttal, only some Rayleigh-Bénard results are available.

Table C. Average VRMSE of latent-space emulation at different compression rates (÷) for the Rayleigh-Bénard dataset and with a time stride $\Delta = 1$. The results are very similar to $\Delta = 4$ with $\Delta = 1$, despite the larger ($\times 4$) number of rollout steps.

Q3 & L2 (Metrics) (V)RMSE and power spectrum analysis are standard metrics in the data-driven emulation literature. The spread-skill ratio is also standard in Earth sciences. We are not aware of other widespread metrics compatible with the datasets we consider. We would appreciate if you could provide downstream metrics that we could compute in our datasets.

Regarding the high-frequency content, this is definitely true. The use of a deterministic decoder for the auto-encoder leads to a lack of small-scale features. We further evaluate our latent models relative to the auto-encoded states $D_\psi(E_\psi(x^i))$ to demonstrate that this lack is mainly due to the decoder, and not the latent emulation.

We see many applications for latent (compressed) emulation. In weather forecasting, as the lead time increases, large-scale trends and features become more relevant for meteorologist. In engineering/manufacturing, large deformations are typically more important than small ones. In astrophysics, you could care more about total light flux than flux variations over a star surface.

Q4 (Patch-size Discrepancy) This discrepancy comes from our desire to give strictly more capacity to the pixel-space models than latent-space models. With 16x16 patches, the pixel-space model has more tokens ($\frac{HW}{16^2}$ vs $\frac{HW}{32^2}$) and more dimensions per token (2048 vs 1024). We also train the pixel-space models with double the compute and for up to 2 times longer. As such, one cannot argue in good faith that our assessment was unfair against the pixel-space baseline.

We would also like to point out that ViTs typically perform better with smaller patches, as demonstrated by Wang et al. (2025) and Mukhopadhyay et al. (2025).

• Wang et al. (2025). "Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More"
• Mukhopadhyay et al. (2025). "Controllable Patching for Compute-Adaptive Surrogate Modeling of Partial Differential Equations"

Q5 (Binary Mask) The binary mask is necessary to perform multi-task conditioning. In our case, the tasks are forward and backward temporal prediction, with $c \geq 1$ context states. We vary $c$ in our experiments to determine whether context length has an impact on emulation accuracy. The results, in Table 7, indicate that it does not.

Regarding the benefits, our claim comes from our preliminary experiments and several references that report similar benefits while using randomized/variable masking strategies:

• Zheng et al. (2023). "Fast Training of Diffusion Models with Masked Transformers"
• Voleti et al. (2022). "MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation"
• Harvey et al. (2022). "Flexible Diffusion Modeling of Long Videos"

Q7 (Inpainting) We do not train for this in this study, but masked conditioning can be used for inpainting. However, we note that this task can also be tackled without any additional training in the case of diffusion models. As mentioned in Section 4, it is possible to guide the model with partial observations, which includes spatial and temporal "inpainting". We will provide more examples of guidance (e.g. inpainting, super resolution) in the camera-ready version.

Q6 (Sampling Strategy) We apologize for the confusion. The Adams-Bashfort sampling method is not novel. It was proposed and extensively tested by Zhang et al. [74]. This method is also equivalent to the widespread "linear multi-step" method proposed by Katherine Crowson in their k-diffusion repository. We believe that this sampler is sufficiently established that we don't need to evaluate it against others samplers (notably DDPM, DDIM and Heun).

Q8 (Pixel-space Diffusion) We do not train pixel-space diffusion baselines as they are too expensive (tens of thousand of GPU hours) to train and evaluate at the resolution and scale of our datasets. We agree that this is a limitation of our work and will discuss it explicitly in the manuscript. We note that our code (which we provided with the submission and will be released publicly) already supports training a pixel-space diffusion model. With enough resources, a third party could easily train these models.

L3 (Speed/Accuracy Tradeoff) Indeed we are not comparing our models against numerical solvers at different space and/or time resolutions. We agree that this would be a valuable addition to this study. Unfortunately, we do not have the expertise to implement these solvers ourselves. We will discuss this limitation in the manuscript.

We would like to note, however, that neural networks have other advantages than being fast emulators. They enable other tasks which are impossible with traditional numerical solvers. Notably, neural networks are differentiable, which can be very useful in the case of inverse problems. Diffusion-based emulators can also be guided with partial/noisy observations. Perhaps most importantly, data-driven emulators do not require knowledge of the underlying physics/PDEs, which is extremely valuable in real-world settings.

Regarding the "knob" to reduce speed and improve the accuracy, there isn't one currently. However, that is a very interesting research question. We believe this could be achieved with a tokenizer with a variable number of tokens such as FlexTok (Bachmann et al., 2025) as auto-encoder. Our methodology could then be applied on this "flexible latent space" and we would modify the computational cost by choosing the number of tokens.

• Bachmann et al., 2025. "FlexTok: Resampling Images into 1D Token Sequences of Flexible Length"

L4 (Latent Representation) As we explain in the discussion, a broader study of different latent representation strategies would be a valuable addition to this study. However, we believe that a comprehensive assessment of all latent representation methods is vastly outside of the scope of a modest academic conference paper and would require 10 to 100 times more time and compute. Therefore, in this work, we focus on the most widespread latent representation, that is the one learned by an auto-encoder.

Formatting As per the paper formatting instructions: "The authors may optionally choose to include some or all of the technical appendices in the same PDF [...]"

We hope that our answers and additional experiments address your concerns. If so, we kindly ask you to reconsider your score.

Thank you for your review and the legitimate concerns you have raised. We are thankful for the opportunity to improve and clarify our manuscript.

W1 (Novelty) We would like to emphasize that the main takeaway of our work is that the accuracy of latent diffusion-based emulation is robust to a wide range of compression rates. This result is unexpected and has practical relevance, as you acknowledge in your review. We are not making any claims regarding the novelty of our method and do not position our models as state-of-the-art.

W2 (Baselines) At the time of submission, no emulation results for the datasets of TheWell were available in the literature, other than the preliminary baselines from TheWell's paper [38]. Training our own competing baselines would be unreasonably expensive and lead to fairness concerns. Nevertheless, we agree that a more comprehensive set of baselines would help the reader assess the significance of our results within the context of data-driven emulators. Fortunately, in the past months, several studies have published results for TheWell datasets, which we compile here and will add to the manuscript.

Table A. Average VRMSE results from different studies using TheWell datasets. Even though our latent diffusion models (LDMs) outperform most published baselines, we emphasize that we are not making any claims regarding the novelty of our method and do not position our models as state-of-the-art, notably due to the discrepancies in parameters count, training and evaluation.

W3 (Explanations) Indeed, our empirical findings raise many questions, which prompt the need for theoretical explanations. We believe that this is precisely the goal of empirical research, and hence a strength of our work. Nevertheless, we agree that the lack of explanation could leave the reader puzzled. We provide two non-exclusive hypotheses that could explain our findings:

1. As explained in Section 5, many models, while accurate for short-term prediction, exhibit long-term instabilities as errors accumulate, pushing the predictions out of the training data distribution. Generative models, particularly diffusion models, partially address this problem [18, 19, 22–25] as they produce statistically plausible states, even when they diverge from the ground-truth solution.

   However, distribution shift doesn't explain everything. At the first prediction step, before distribution shift can take effect, diffusion models are already slightly better than deterministic neural solvers. This could be explained by the iterative nature of DM sampling which allows the model to refine its prediction. Validating this hypothesis is an interesting avenue for future research.

2. For many physics, and especially fluids, dynamics are mainly explained by large-scale/low-frequency content, while small-scale/high-frequency content influences the (very) long-term evolution of the system. Learning to process small-scale content and predict its influence on long-term dynamics might require very large networks, as well as huge amounts of data and compute. When the latter are limited, it becomes more tractable to focus on larger scales. Therefore, by discarding some of the small-scale content, latent diffusion models are able to learn and reproduce the dynamics more easily than pixel-space models.

We would appreciate your feedback on these hypotheses and whether we should include them in the manuscript.

W4 (Applications) We see several use cases for using (large) compression in simulation applications. First, some fields (weather, astro) are collecting a lot more data that we can currently process or store. Our findings suggest that we could store or transfer this data in a compressed form, without losing (too much) information/predictability. Second, robustness to compression also means that practitioners do not have to worry (too much) about the compression rate they use: doubling or halving the compression rate will lead to similar emulation accuracy.

W5 (Dataset Diversity) We follow your suggestion and add an additional 3D dataset to our study. This dataset (turbulence_gravity_cooling from TheWell [38]) models the interstellar medium as a turbulent fluid subject to gravity and radiative cooling. The 3d state of the system is discretized on a 64x64x64 grid. As for the other datasets, we train several auto-encoders with increasing compression rates (from 48 to 768) and latent emulators. The results (see table below) are consistent with the previous experiments and strengthen our conclusions: latent diffusion emulators are robust to compression. Latent neural solvers, conversely, are not as robust.

Table B. Average VRMSE of latent-space emulation at different compression rates (÷) relative to the auto-encoded states $D_\psi(E_\psi(x^i))$ for the TGC dataset.

Unfortunately, we cannot provide plots during the rebuttal, as per NeurIPS guidelines. Further results will be provided in the camera-ready version.

W6 (Real-World Problems) Euler and Rayleigh-Bénard are some of the most challenging physics used in the field of data-driven emulators at the moment. They are commonly used for verifying/benchmarking numerical solvers and demonstrate behaviors often found in nature. The new 3D dataset (TGC) also has real-world applications and was originally generated for a domain-science publication by Hirashima et al. (2023).

• Hirashima et al., 2023. "3D-Spatiotemporal forecasting the expansion of supernova shells using deep learning towards high-resolution galaxy simulations"

We believe that this rebuttal addresses most of your concerns and, therefore, kindly ask you to reconsider your score.

Thank you for answering our rebuttal.

Novelty: I understand this work's main takeaway; but I think the paper should go beyond the current empirical findings and justifications. The paper should have provided a generalization of the observed phenomena and theoretical justification as I wrote earlier.

Explanations: It would be beneficial if objective evidences, such as supporting references, could be provided to validate the reasonableness of the suggested hypotheses. Alternatively, building a small-scale synthetic dataset to experimentally test and confirm these hypotheses could also be helpful.

Our empirical findings are novel and valuable, as you acknowledge yourself in your review. The goal of empirical research, which the field of deep learning is largely driven by, is to raise questions, not to find theoretical justifications.

In addition, your original review asks for additional datasets and justifications why LDMS outperform deterministic solvers. We followed your suggestions and provide an additional 3D dataset as well as two hypotheses grounded in previous peer-reviewed research [18, 19, 22–25, 26, 56, 102]. In addition, there is an extensive literature on reduced-order modeling (ROM) [85-99] that provides theoretical justifications for modeling physics within reduced / compressed representations. This literature is already discussed in Section 5 (Related work).

We also believe that it is unfair to ask for other toy experiments that late in the review process.

Baseline: My original request was for experiments to be conducted using various generative models, not just LDMs. Through such comparative analysis, the paper could explore questions such as whether only LDM exhibits superior performance and, if so, the underlying reasons for this unique advantage. Including such analyses would enhance understanding the significance of the observed phenomenon, providing researchers with valuable insights.

We do not train pixel-space diffusion baselines as they are too expensive (tens of thousand of GPU hours) to train and evaluate at the resolution and scale of our datasets. Training our own baselines would also lead to fairness concerns. We believe that the additional baselines provided in the rebuttal are more than enough to demonstrate that our results are significant. In addition, baselines are not relevant to answer our research question: we are only interested in the accuracy of LDMs when compression increases. We note that our code (which we provided with the submission and will be released publicly) already supports training a pixel-space diffusion model. With enough resources, a third party could easily train these models.

Finally, we would like to note that your only remaining concerns are (a) a lack of theoretical justifications for empirical findings (even though we provide hypotheses) and (b) some missing baselines. We believe that it is unreasonable to recommend "reject" on this basis. Our submission does not have technical flaws, the evaluation is sound and extensive with very challenging datasets, and we provide the code for reproducibility. We also think that our work deserves more than "1: poor" for its originality as our findings are novel and valuable.

Thank you for taking the time to answer. We agree that applying our methodology to other generative models is interesting and valuable. However, this is outside of the scope of our work. As our title indicates, our study focuses on latent diffusion models and our findings do not need a comparison against other generative models to be valuable and relevant. The experiments your suggest would not modify our findings but provide new findings.

In addition, (latent) diffusion models are state-of-the-art generative models and practitioners are actively using them for (latent) physics emulation, as explained in our introduction. Conversely, GANs are extremely finicky to train and VAEs are poor generative models. We are not aware of any research using VAEs or GANs in the field of physics emulation. Khol et al. [2] are not comparing VAEs with DMs. They compare DMs with a transformer trained to emulate in the latent space of a VAE. The VAE plays the same role as our AEs, not as the emulator.

Therefore, we respectfully disagree with your opinion that our work is not significant and original. The very fact that our work raises so many questions, which you formulate in your own review, demonstrates its significance. Even with limited significance, your recommendation (reject) is unjustified.