Inference from Real-World Sparse Measurements

Dear Reviewer 7eM6,

Thank you for your review

For the spatial subsampling: For wind nowcasting, we did not apply any subsampling since the partitioning of data into time slices inherently provides this. For other datasets, random subsampling was used.

Your input on the paper's presentation is valuable, and we apologize for the overlooked typos, which will be promptly corrected.

Regarding the evaluation metrics, we actually used a set of 7 comprehensive metrics for the wind nowcasting task, and in the appendix, we benchmarked this task in several different cases: varying forecast periods and examining the impact of context-target distance on performance. Finally, while Table 1 presents MSE values for consistency, the appendix includes comparisons using metrics from related works, including relative L2 errors in some cases. While we agree that having a comprehensive evaluation of the ERA5 results would be nice to have, we think that the MSE results are enough to show that our method is able to be competitive in very different domains.

Considering the strengths you recognized and the fact that the weaknesses you list are mainly presentation-oriented, the 'reject' rating seems disproportionately severe. We are committed to enhancing the paper based on your feedback, so if you have any other concerns, we look forward to possibly addressing them during the discussion period. If this is not the case, we hope you will be open to changing your final rating, especially considering your acknowledgment of our paper's contribution to a less explored yet important domain in scientific ML.

Dear reviewer NVVV,

Many thanks for your thorough review of our paper.

Concerning the weaknesses you mentioned:

1. We agree that attention-based models seem to be used everywhere. In this work, we show that they possess properties that make them outperform competitors, for instance, the fact that their latent representations are disentangled. This leads to them being a very general type of architecture that can be applied to many problems. Regarding the literature on motion forecasting and its link to the reference example of this paper, we believe that while there are some connections—such as the fact that both processes are recorded along trajectories and that in both cases an attention-based model is used—the similarity stops there. Indeed, in our work, we don't rely on the fact that the points are recorded along trajectories; particularly, the goal is not to predict the next trajectory points. Our work mainly focuses on the fact that the data are sparsely recorded in time and space. A set of context points in the case of wind nowcasting contains only a few points per trajectory but many trajectories over the entire space. Hence, the goal here is more to be able to process data from anywhere and to extract a forecast anywhere, rather than being able to complete a trajectory. The rest of the applications, for example, do not have an inherent trajectory structure. In more detail, for the specific work you mentioned: Wayformer [1] uses a traditional transformer to condition the prediction of a trajectory given a scene, which is composed of a mix of different types of inputs, but we don't believe this work deals with sparse measurements, and the goal of the model is to decode a trajectory, not to forecast a given phenomenon anywhere in space. Agentformer [2] mixes temporal and social dimensions into a single sequence modeled by a transformer; this seems more focused on forecasting time series with no notion of being able to forecast anywhere in space. Sequential set transformer [3] again is mostly focused on forecasting different futures for agents; in our case, our primary goal is not to forecast the trajectories of planes but to extrapolate reliable wind forecasts from their measurements. However, we are open to adding a sentence in the related work section to distinguish our approach from attention-based models for trajectory predictions.

2. We are sorry that it was confusing; for the attention part, we tried to stay as close as possible to ViT notation, but additional work was necessary to specify the context and targets, and we think the confusion may have arisen here. More details on this in the answer to comments.

3. We note your remark; we think our method is general enough to be applied in many cases involving sparse data, that is why we tried to keep our description as general as possible.

On the major comments:

1. We agree that the notation is quite heavy, but we don't think there is a simpler way to describe the problem, as we need to say that the models will process a set of data that can change from sample to sample and that they should be able to generate an output conditional to a set of measurements. We propose to change the sentence you mentioned by "the models are given a set of context points at positions $c_x$ of value $c_y$, and should be able when given target positions $t_x$ to output a corresponding value $t_y$ conditioned on the context." Adding an illustration there would be too much in our opinion, but we propose linking to the one in the appendix.

2. Here equations 1-4 are referring to the way of encoding the different quantities $c_x$, $c_y$, and $t_x$ into the models. The problem is that the model processes vectors of the same dimensions without distinction between context measurements (which contain both position and values) and target positions. $\varphi$, $\psi$, $\phi$, and $\nu$ are all multilayer perceptrons.

3. No, the proposed model proposes to remove the whole decoder stack and to feed both the encoded input and encoded readout to the same transformer encoder. A traditional transformer contains both an encoder part, which consists of multi-head self-attention with full attention + feedforwards, and a decoder part with causal self-attention and cross-attention with the final layer of the encoder. The final model, therefore, does not have any decoder part and consists only of self-attention with full attention and feedforward layers.

4. As transformers are really general architectures, it's not surprising to see them used in a lot of different contexts. In [5], "A transformer neural network for predicting near-surface temperature," it seems that they use a transformer to model a 1D time series, and while it focuses on weather data as well, this work does not focus on how to forecast based on sparse data and doesn't need to encode values at different positions. [6] TENT: Tensorized Encoder Transformer for temperature forecasting seems to use an additional dimension to a transformer-encoder architecture in order to be able to deal with multivariate time series. As far as we understood the paper, each of these time series corresponds to a given city, where we have information at each time step. In our work, this is not the case as measurements are sparse and their positions can vary from example to example.

5. [6-7] are focused on spherical CNNs, which are a good way to adapt CNNs when dealing with a spherical grid and thus a good candidate for weather forecasting based on a regular spherical grid. We thought that FourcastNet, a graph net approach, was considered state-of-the-art in that regime, but we'll review if there are cases where the state of the art is reached by spherical CNNs and update the related work section if that is the case. In our work, however, we consider data that is not sampled along any (spherical) grid. GEN, our considered baseline, uses graph convolutions in its message-passing scheme, which is a way of generalizing from CNNs and spherical CNNs to irregular meshes.

6. Fourier Neural Operator is a very interesting line of work. Sadly, it relies on having information on regular 2, 3, or 4D grids in order to work, as it uses FFT extensively. This is not the case in our setup. We tried at some point to adapt FNO to our setup by using non-uniform discrete Fourier transforms (nuDFTs), which seemed to be the simplest way of adapting FNO to our case. Sadly, the results were not efficient nor really stable, so we decided not to continue in that direction.

We noted all your minor comments and will update the manuscript accordingly. For table 1, we think it is a nice way of presenting the strengths and weaknesses of the different baselines for a reader who only reads this work diagonally. We found several papers using this kind of notation.

Thank you again for your in-depth analysis of our work. We are at your disposal if any of our answers are not clear enough.

Dear Reviewer 3Jak,

First, many thanks for your kudos and for your overall nice review.

We'll answer your questions first. Q1: We thought that these notions were well-defined. The transformer encoder consists of a series of blocks consisting of self-attention mechanisms, which utilize full attention, and feed-forward networks, interleaved with residual connections and layer norms. A transformer decoder includes an additional cross-attention layer. In the decoder, the self-attention layers traditionally use causal masking, but we removed that in our implementation as there is no problem if the target positions see each other, and we use the architecture to predict all measurements in one pass, not autoregressively. For reference, consider the official PyTorch implementation or the original encoder stack from "Attention is All You Need." Q2: We noted it that way to state that in (5), the transformer encoder takes as input the two sets of encoded contexts and encoded target positions, which in practice are concatenated. Since the output is of the same dimension, we have corresponding processed latents for each context point and each target position, but only the output corresponding to the target position is used later on. For (7), we only pass the context measurements to the transformer encoder. We tried using concatenation notation for (5), but it was heavier, so we stuck to a more general mathematical formulation.

For your remark concerning the real-world need, we wanted to state that this work was developed with an industrial partner with a production application for the wind nowcasting task in the context of air traffic control in mind. This is also the reason why we focused on an architecture that is robust and reliable. And this is why we preferred offering clear justifications for the architecture instead of overcomplicating it. In this context, the CNP baseline was outperformed by a previously considered baseline. Adapting GENs for this task required more design choices, for example, how to define the underlying graph given the considered space, how to choose the links between nodes, and a careful hyperparameter exploration regarding the learning rate for the position of the graph. While we can't cite a specific example where this method underperforms compared to MSA, our model consistently offered better performance while being easier to implement.

Concerning the paper you mention:

[1] "Learning Temporal Evolution of Spatial Dependence with Generalized Spatiotemporal Gaussian Process Models" seems to use a specific Gaussian process to model spatiotemporal models. While Gaussian processes have strong theoretical properties, they can be hard to implement, require very specific knowledge from the final user, and are hard to scale to a large number of inputs. We considered Conditional Neural Processes (CNP) as a baseline, which were introduced to solve these problems.

[2] "Neural Spatio-Temporal Point Processes" appears to model discrete events and their effects that can happen irregularly in time and space and are more related to density estimation of time series data than conditioned prediction.

[3] "A Spatio-Temporal Statistical Model to Analyze COVID-19 Spread in the USA" seems to be similar to [1] in that they rely on a separable GP. We agree that having additional Neural ODE and GP baselines, or evaluating our method against them on their benchmark would be nice to have, but we thought that our benchmark was comprehensive enough to prove our point.

Considering the general application, we think that our method can be useful anytime there is a need to manage a fleet of moving vessels, for example in the case of airspace, but other applications could be drone coordination or maritime fleet management. We agree that, in essence, other practitioners could probably have the idea of using transformers for that task by themselves, but even in that case, our work would provide a way of adapting the general architecture for that specific task, a proof that it works, and some justifications about why it is a good idea to use transformers in that case, along with comparisons with existing baselines.

We thank you again for your clear and valuable opinion and your time reviewing our paper.

Dear reviewer jjj1,

Thank you for your time reviewing our paper. We will try to address your concerns regarding related works, the weaknesses you highlighted, and your questions.

First, on the related works: The five papers you are citing focus on mesh-based learned physics simulators. We believe that the setup studied in our paper is not the same, and these methods do not apply here; that's why we didn't consider them as baselines.

[1] Eagle: Large-Scale Learning of Turbulent Fluid Dynamics with Mesh Transformers The data inputs are meshes that are generated by numerical solvers (Ansys Fluent), which means that even if the graph is irregular, the models always have values at the nodes of the mesh, which is consistent across all timesteps of all different episodes.

We are trying to extrapolate sets of measurements that can be measured anywhere, where the number of measurements and the positions of measurements can vary from set to set and are not the same between context and targets. That is why using a mesh is not straightforward; actually, if we wanted to do so, we would need a spatial interpolation scheme for converting the measurements to the mesh, and then a second one to go from the mesh to the targets. That's actually what GEN (the considered baseline) does. If we wanted to adapt EAGLE to our case, we would need to add interpolation after the encoder and after the decoder, and so the final method would ultimately be a variant of GEN with some additional clustering, graph pooling, and attention. This final model would suffer from the same problems that GEN has. Note as well that we considered some message-passing scheme with attention in our Appendix 8, and it didn't help.

[2] Fourier Neural Operator with Learned Deformations for PDEs on General Geometries Introduce geo-FNO, which learns a mapping from an irregular mesh to a regular one on which FNO (based on FFT) can be applied, and finally maps from the regular lattice back to the irregular mesh. While this approach is interesting, it still needs to be applied to dense meshes. In our case, the space is sparsely populated with measurements: in a given set of data, we only have measurements in some parts of the space, not everywhere but irregularly. We tried at some point to adapt FNO to our setup by using nonuniform discrete Fourier transforms (nuDFTs), which seemed to be the simplest way of adapting FNO to our case. Sadly, the results were not efficient nor really stable, so we decided not to continue in that direction.

[3] MAgNet: Mesh Agnostic Neural PDE Solver This work uses the same encode-interpolate-forecast structure as Graph Element Networks and only differs because they use data from a parent mesh (so they don’t need to interpolate from the measurement to the mesh) and that it is applied to time series. Again, if we wanted to adapt their work as a baseline, we would probably end up with something very similar to Graph Element Network.

[4] Multiscale Meshgraphnets Is an extension of Meshgraphnet which falls back to a variant of GEN if we tried to adapt it to our sparse measurements setup. Actually, we mention Meshgraphnets in the related works, second paragraph. But we agree we could make that point clearer. We propose adding the following sentence: "[...] irregular but fixed mesh. While these approaches are achieving impressive performances in their domains, using a mesh-based approach has some drawbacks when working with sparse measurements as we need a way of mapping the measurement to the mesh and most of its nodes would have no related measurements."

[5] Reinforcement Learning for Adaptive Mesh Refinement This seems to use reinforcement learning to define the proper mesh for finite element methods. Again, having a fixed irregular mesh would be nice, but it does not help as, in our case, a set of data is sparse and does not "cover" the entire space, so most of the nodes of such a mesh would be empty most of the time.

So, to summarize, in our case, we cannot rely on meshes because the data is sparse and varies from set to set. As a practical example, where our work is used in practice, consider an airspace. We can have access only to measurements where planes are, and most of the space at a given time is empty. The only way of using a graph neural network, since the measurements can come from anywhere, would be to interpolate them onto the graph, then process them, and then extrapolate to where we have the queries. And this is exactly what GEN does. The other baseline we consider, CNP, uses a different approach but can also take a set of sparse measurements as input and interpolate from them as well, which these mesh-based architectures cannot do.