Continuous Field Reconstruction from Sparse Observations with Implicit Neural Networks

We are grateful for the reviewer's constructive feedback, particularly the recommendations regarding (1) additional ablation experiments on network design; (2) optimal sensor placement; and (3) explanation of sampling rates. In the following, we address all the comments individually for clarity and completeness.

Comment $\#$ 1: The 4th line of Page 6, “we uniformly distributing” should be “we uniformly distribute”.

Response: Thank you for pointing it out. Fixed.

Comment $\#$ 2: How do the authors determine the sparse setting (Page 7) for those two datasets? The sparsity ratios for those two datasets are quite different. Also, can the authors provide an estimate or convergence analysis of the smallest number of sensors needed for reconstruction?

Response: We have conducted additional analysis to determine the minimum number of sensors required for effective reconstruction. Specifically, for each dataset, we employed proper orthogonal decomposition (POD) and sorted the computed eigenvalues in descending order. The cumulative energy was calculated, and for the simulation-based data, approximately 750 POD modes are needed to reconstruct $90\%$ of the total kinetic energy. Note this calculation is based on the snapshot POD method [1, 2]. Using this information, we defined an interval $[0.1\%, 1\%]$ and conducted a grid search to estimate the minimal sampling rate. Our findings, both quantitative and qualitative, indicate that $1\%$ is the closest approximation to the minimum sampling rate for the simulation data. Below is the MSE comparison between MMGN and POD reconstruction, with detailed results and descriptions included in the revised paper.

[1] Taira, K., Brunton, S.L., Dawson, S.T., Rowley, C.W., Colonius, T., McKeon, B.J., Schmidt, O.T., Gordeyev, S., Theofilis, V. and Ukeiley, L.S., 2017. Modal analysis of fluid flows: An overview. Aiaa Journal, 55(12), pp.4013-4041.

[2] Sirovich, L., 1987. Turbulence and the dynamics of coherent structures. I. Coherent structures. Quarterly of applied mathematics, 45(3), pp.561-571.

Comment $\#$ 3: Why not consider the ERA5 dataset for climate modeling since the recent climate modeling papers [1-3] all use this benchmark dataset?

Response: The model is designed to be data-agnostic, making it applicable to the ERA5 dataset. We plan to release the implementations as open source, enabling readers to reproduce the presented results and apply the model to datasets of their research interest.

Comment $\#$ 4: What are the spatial resolutions for the datasets used in this paper?

Response: Thank you for bringing this to our attention. For the simulation-based data, it's 1024 (time) $\times$ 192 (latitude) $\times$ 288 (longitude); and for the satellite-based data, it's 360 (time) $\times$ 901 (latitude) $\times$ 1001 (longitude). We have also accordingly revised the data description section in the paper.

Comment $\#$ 5: The dataset is not that challenging. This paper considers monthly-averaged temperature data in Simulation-based data CESM2. The dynamics are relatively smoother than daily-averaged data. Please see some of my follow-up concerns regarding the dataset in Questions.

Response: Thank you for pointing it out. In the data preprocessing step, we intentionally remove the temporal mean and seasonal cycle as they are straightforward to predict. This clarification has been included in the revised paper.

Dear reviewer, we appreciate your comments and acknowledge your concerns regarding our manuscript. We have carefully considered your feedback and made substantial revisions to enhance the clarity and coherence of our work. We have simplified the mathematical notations, providing more intuitive explanations and examples to facilitate better understanding. A simple walkthrough is provided below.

Problem Our objective is to accurately reconstruct a spatiotemporal continuous physical field, denoted as $\boldsymbol{u}$, representing quantities such as temperature, velocity, or displacement. This field, $\boldsymbol{u}$, is inherently a function of both spatial coordinates ($\boldsymbol{x}$) and time ($t$).

Prior approaches Traditional implicit neural networks (INRs) incorporate the time index ($t$) along with spatial coordinates. For instance, in 2D data, the network features three input nodes for the $x$-coordinate, $y$-coordinate, and time $t$, where the time index ($t$) is employed to direct the model to a particular time instance.

Our approach Motivated by the desire for a more context-aware indexing mechanism, we posed the question: Can the indexing process be improved? We propose utilizing measurements observed $u_t^{(1)}, u_t^{(2)}, \dots$ at time t for implicit model guidance, rather than relying on $t$ for explicit pointing. More precisely, our approach involves constructing an encoder $E_\varphi(\cdot)$ to transform the observed measurements $u_t^{(1)}, u_t^{(2)}, \dots$ into a latent code ($\boldsymbol{z}_t$). Subsequently, we employ a decoder $D_\phi(\cdot)$ that utilizes both the encoded information ($\boldsymbol{z}_t$) and spatial coordinates ($\boldsymbol{x}$) to reconstruct the underlying physical field. Below are detailed explanations of the model components we designed and the rationale behind each choice.

• Encoder: The auto-decoder is chosen as the backbone for the encoder due to its ability to accommodate free-formed observations, considering that the number and positions of available observations vary over time.
• Gabor filter: INRs often encounter challenges in learning high-frequency functions, a phenomenon known as "spectral bias." To mitigate this issue, we employ the Gabor filter to transform spatial coordinates ($\boldsymbol{x}$) before the decoding process.
• Decoder: The decoder incorporates a multiplicative filter network (MFN) as its backbone network. This choice is motivated by the recursive mechanisms embedded in MFN, enabling improved fusion of spatial coordinates ($\boldsymbol{x}$) and latent codes ($\boldsymbol{z}$).

Results We conducted comprehensive experiments to demonstrate the superior performance of the proposed model.

In response to your feedback, we have extensively revised our paper, ensuring that the updated manuscript is more accessible and comprehensible. In the meantime, we provide point-to-point response to your comments below:

Comment $\#$ 1: Problem setup: They assume that the system is governed by a PDE (1). I don't see any reason why this assumption is made, or is that assumption used somewhere?

Response: We used PDE to facilitate the transition to functional variable separation and subsequently to our proposed context-aware indexing mechanism. However, we acknowledge the potential confusion it may have caused, and as a result, we have removed this equation from the paper.

Comment $\#$ 2: Section 2.1, "Objective" (and through the paper): They use notation . What this " " stands in the set? usually is used for conditioning but I cannot see such a thing here. I would rather remove this extra part here to avoid confusion (I use '(' and ')' instead of curly brackets as this form does not seem to render those (and actually, strictly speaking, '(' and ')' would be preferred if order of measurements have meaning ))

Response: Conditioning is employed in our notation because the latent codes are derived from all available position-dependent measurements at time $t$.

Comment $\#$ 3: The section 2.1 does not talk about noise or randomness and formulations in this section would indicate that is deterministic. However, later sections seems to talk about noise and assume that is includes noise or is random (e.g. Eq (5) which does not have meaning if is deterministic). This is contradicting. I would suggest authors that authors would use separate notations for a noiseless quantity and the measurements, e.g., say something like is a vector of measurements or plus noise.

Response: Thank you for your feedback. We have incorporated the suggested changes into the revised paper, and additional details regarding the model's performance under various noise levels have been included.

I read the revised manuscript. I admit that it fixed and clarified quite many parts of the original manuscript. But there are still some parts to improve and I am not confident that if the quality is enough for publication in ICLR.

At least there points could be clarified and/or improved:

• The key in their approach is the latent model $E$ which maps measurements from time $t$ to the latent variable, which is then fed to $D$ to get prediction of $u$ for the selected point $x$. This makes me to wonder that are measurements required for all time instants for which you want to infer $u$, meaning that the approach cannot be used to interpolate or extrapolate $u$ with respect to time? I also do not see mechanism how model could adapt to temporal behaviour of the physical system. If my interpretation is true, then the method acts more like a spatial interpolator instead of predicting a space-time system. This means that the model cannot be used, for example, for weather or climate prediction. I think this is a quite big limitation and should be addressed. If this is not the case, I would wish to see an explanation that how the method is applied when $\mathcal U_t$ is empty.

• There are some missing details such as $\mathcal T$ and the loss function are not specified. Appendix seems to include training details, but this section is not referenced in the main part of paper. Appendix also has some broken references (e.g below Eq (6)). These are not big things, but improves readability and I would not like to see such in published papers.

• Numerical results look good, but baselines are mostly other INR methods. But I would as pointed out in the introduction, there are several other non-INR methods. I would wish to see those to be compared . Or if such comparison is done in another study, this could be referred.

I increased my rating as this revision was a clear improvement, but still not sure if the manuscript is ready for publication.

Dear reviewer, we are grateful for your comments and questions. We find them very helpful! Particularly those pertaining to our model designs and the proposed ablation study on the Gabor filter's effects. Overall, we have revised the methodology section to elucidate the distinctions between the current INR and our approach, providing explanations for our design choices. Please find our detailed responses below.

Comment $\#$ 1: About latent code z. The paper claims to use the trainable latent code to represent temporal information instead of using time index. What is the motivation for this? Because the recorded time index is not reliable in this application? Or the latent code can also represent other semantic information? Previous works have shown INR can represent spatial-temporal information in a good way by adding temporal index as input coordinates.

Response: Yes, the latent code has the advantage of capturing not only temporal information but also other semantic features of the underlying physical process. Previous studies have indeed demonstrated the capability of INR to effectively capture spatialtemporal information by incorporating a temporal index as additional input coordinates. But it should be noted that different from forecasting problems, the time index ($t$) is primarily used as a reference to indicate a specific time instance in the context of field reconstruction. Motivated by the desire for a more context-aware indexing mechanism, we introduce a new perspective to the modeling process by utilizing a trainable latent code. Since the latent code is learned using the available measurements at time $t$ via auto-decoding, it is a more context-aware representation, which improves the performance of the INR-based decoder. We have expanded the methodology section to elaborate on these points, providing a detailed discussion on the motivation behind our model design choices and a visual comparison between the traditional INR and our proposed approach. We sincerely invite the reviewer to check the updated paper and provide any further feedback or suggestions you may have.

Comment $\#$ 2: As described in the paper, during training, both latent code and INR are trained. While during training, INR model parameters are fixed and only latent code is optimized. Since the given information is quite sparse during testing, how reliable it can be to optimize latent code? How robust this method can be when generalized across different objects? By only changing the latent code in the input of INR decoder and freezing all other model weights, can this change the decoded image in an efficient way?

Response: Thank you for the insightful question regarding the generalizability of latent codes. To investigate this aspect, we conducted additional experiments by partitioning the simulation-based data set into two subsets, each comprising 512 instances. The temporal splitting was performed in an ascending order, with the initial 512 instances assigned to subset $\mathcal{D}_1$ and the subsequent 512 instances to subset $\mathcal{D}_2$. Consequently, due to extrapolation in the time direction, the distributions of $\mathcal{D}_1$ and $\mathcal{D}_2$ could vary significantly. We trained MMGN using $\mathcal{D}_1$. Then, keeping the decoder parameters fixed, we performed inference on $\mathcal{D}_2$, where the only trainable parameters were the newly initialized latent codes. The results indicate a marginal increase in error, demonstrating that the proposed MMGN avoids the catastrophic forgetting issue encountered by other time-index-based INR methods when re-trained using different data set.

Admittedly, one limitation of our current work is its focus on reconstructing a selected trajectory. In the future, we plan to investigate MMGN's potential for generalization to multiple trajectories or even across various climate properties, with the goal of recovering arbitrary underlying flow maps. We have incorporated this discussion into the conclusion section of the revised paper.

We appreciate the valuable insights offered by the reviewer, particularly the recommendation to conduct additional experiments involving varying levels of noise to enhance the demonstration of the model's robustness. Additionally, we are thankful for the thoughtful and practical suggestions for revising the presentation of the paper. Subsequently, we present our responses to each comment in a point-by-point manner.

Comment $\#$ 1: in experiments where you recover fields on the surface of a sphere, are you using Cartesian or spherical polar coordinates?

Response: In experiments involving the recovery of fields on the surface of a sphere, we utilize spherical polar coordinates for modeling.

Comment $\#$ 2: why not include (different levels of) random noise in observations? this is an important test of robustness.

Response: Thank you for the suggestion. We have conducted additional experiments considering the introduction of noise to the input data. The noise level in the dataset is quantified by the channelwise standard deviation specific to that dataset. Furthermore, we introduced customized noise ratios, including scenarios with noise levels set at 1%, 5%, and 10%. The quantitative results are presented below. Additionally, we made a new figure to enhance the visualization of performance distinctions among various models in the presence of noise. The detailed discussion and presentation of these findings have been incorporated into the revised paper.

Comment $\#$ 3: I am not sure that (Izacard et al., 2019) is the right reference for sparse seismic networks and small earthquakes (or even for sparse spatial coverage in scientific data; it addresses a super-resolution problem). I find the paper very hard to read. Notation and expressions in Section 3 and the last paragraph of Section 2 are confusing and it takes a long time to understand what is going on (Figure 2 helps here but the math could be clearer). Another reason is very florid prose which I'd hesitate to use in a machine learning paper. Examples: "The endeavor to tackle the conundrum of global field reconstruction from sparse observations is currently shaping two distinct paradigms of thought", "raging ocean waves", "intricate physical fields" (what are they?), "we present an innovative neural network approach", "... a singular viable option", ...

Response: We thank for the valuable feedback. We have made several improvements to the paper. Specifically, we corrected the reference in the introduction and provided a clearer explanation of the contributions. Moreover, we enhanced the methodology section by introducing the model more explicitly and illustrating the design process as well as adding a new figure.

Dear reviewers, we've incorporated feedback into our revised manuscript. Your thoughtful input has been very helpful. We kindly invite you to reassess the paper, updating your comments and scores if possible. Thank you for your time and effort.