Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcasting

We would like to thank the reviewer’s feedback and suggestion. Here is our response to each of the reviewer bJuK's concern.

---

The authors sensibly use MSE loss as a baseline when evaluating their model...

Among the papers the reviewer suggested, BMSE [1] performs linear scaling at different pixel ranges, Multisigmoid loss (SSL) [2] preprocesses the images with linear transformations and nonlinear sigmoid function before applying MSE, [3] tested SSIM and MS-SSIM and recommended MSE+SSIM to be the loss function.

Due to limited time and resources, we present the results by adopting these losses with ConvLSTM trained on Stochastic Moving MNIST. For SSL, we follow the paper and pick $i \in (\frac{20}{70}, \frac{30}{70})$ and $c = 20$. The visualization is reported in the appended PDF.

Stochastic Moving MNIST:

HKO-7:

We would like to direct the reviewer to view Figures 2 and 3 in the global rebuttal PDF. We conclude the following summary from the above tables and appended figures, together with some of our previous experience:

1. Weighted MSE (such as BMSE) only “tilts” the focus. BMSE severely over-predicts in exchange for an improvement in CSI, sacrificing all other metrics such as MAE, SSIM, LPIPS, FVD, FSS and RHD.
2. SSL improves the model performance in general, but still cannot generate clear output under stochastic motion.
3. Losses integrating SSIM (and also L1) “dissolves” the prediction to zero over time under uncertainty. Such effect is especially significant for weaker models.

---

You mention that you introduce FCL because the FAL doesn't fully constrain the spatial positions in the image. A seemingly simpler solution would be to use the FAL in combination with MSE. Have you tried this approach and if so, do you see advantages with using FCL instead?

Our research indeed started with FAL in combination with MSE in the early stage. We replaced MSE with FCL later due to a few concerns.

Theoretical concern:

In Appendix D, we showed that FAL can be broken down into three terms: $L_2(X, \hat{X})$ (that is, MSE in the image space), $\Sigma 2X\hat{X}$, and $-\Sigma 2F\hat{F}$. The difference $\Sigma 2X\hat{X} - \Sigma 2F\hat{F}$ corresponds to the translation invariant factor and the L2 term corresponds to the correctness of values. When the image is translated (Figure 4 in paper), the two terms almost cancel out, resulting in a perfect invariance to translation.

If we form the new loss as a linear combination of MSE and FAL, the weighting of the L2 term increases, breaking the balance between the two terms. With this intuition, we believe the resulting loss will not generate better results if we mix MSE with FAL.

Practical concern:

Despite the theoretical concern, we did test FAL + MSE. Besides, we also tested FAL + SSIM loss since SSIM is also considered a substitute for MSE.

The models are trained on SEVIR:

Empirically, FAL+MSE and FAL+SSIM work but FACL outperforms them by some margin. We will append this information to the paper if the reviewer finds it helpful.

---

Also, you are using a loss where you randomize between FAL and FCL ... how is this not functionally equivalent to using a linear combination with the weight of FAL increasing over time?

A linear combination of FAL and FCL with increasing weighting on FAL is functionally equivalent with the current FACL, but the hyper-parameter tuning is different. We attempted to formulate $FACL = w(t) \text{FAL}(X, \hat{X}) + \text{FCL}(X, \hat{X})$ instead of Equation (6). Choosing a good $w(t)$ includes its curve type and maximum value, which can be challenging. Since FFT is orthonormalized (normalized by the image size $\sqrt{HW}$), we set the maximum value of $w(t)$ to $HW$ to generalize FACL to different datasets and image sizes. For HKO-7, $HW=230400$, which may cause numerical instability when multiplying to the FAL term. To avoid the hyper-parameters going too large, we find switching to a linear decay of selection probability between [0, 1] results in a more stable and neat solution. Since the adjustable range of $\alpha$ is small, it is also easier to control the tradeoff between accuracy and realisticity.

---

limitation

We think the suggested limitation is built on the premise that the model itself cannot generate multiple forecasts. For example, if a ConvLSTM trained with MSE cannot have ensemble outputs, then the same model trained with FACL cannot do it either. However, if another model trained with MSE can generate diverse outputs, replacing MSE with FACL here can also yield multiple sharper outputs.

Here we present the updated table for MeteoNet with new CSI thresholds (12, 18, 24, 32) based on the reviewer's feedback. Models trained with FACL still largely outperform those trained with MSE. With the increase of the CSI pooling radius, the performance of the diffusion models also drastically increases.

We would like to thank the reviewer’s feedback and suggestions. Here is our response to each of the reviewer 7mGa's concerns.

---

The evaluation is incomplete and unconvincing. The csi-m of EarthFormer is much lower than the results in [1] (0.3982/0.44).

We adopted the official Earthfomer model and SEVIR dataloader. However, the training script is re-implemented to ensure a fair test (by only replacing MSE loss with FACL while keeping other factors constant). This causes us to change multiple hyper-parameters, potentially changing the model performance. For example, due to the scheduling for FACL random selection, we should not apply early-stop to the models. Due to resource limitations, the batch size is set to 32 instead of 64, etc. These tiny factors may aggregate and change the overall performance. For example, in DiffCast, their 5-in-20-out setting in SEVIR causes Earthformer to have a CSI-m of 0.2513, showing that these factors can result in a huge difference in the ultimate performance.

Despite the difference, we do not think it invalidates our evaluation results. Our evaluation does not intend to convince the readers that an arbitrary model with FACL peaks any SOTA model in their optimal state. Instead, with a set of fair tests, we show that a model with FACL is better than the same model with MSE in precipitation nowcasting by its forecasts being more realistic (better LPIPS, FVD) and more skillful (better CSI, FSS, RHD). Such observation is consistently reflected by multiple tables and figures in the paper, both quantitatively and qualitatively. We hope the reviewer can at least agree with such a conclusion.

---

Besides, it could include more methods, such as NowcastNet[1] and DiffCast[2].

DiffCast natively does not support 13-in-12-out sequences. Moreover, since 384x384 is too computationally heavy for inference, DiffCast downsamples the SEVIR dataset to 128x128. This causes unfair evaluation especially for LPIPS and pooled CSI which are sensitive to the image size. We adopted the official checkpoint of DiffCast and upsample its forecasts to 384x384 on SEVIR:

To sum up, DiffCast only has slightly better LPIPS and FVD, while other VP models with FACL in general have better pixel-wise accuracy and skill scores. (The performance of the other models can be found in Table 2 in the paper; the visualizations can be found in Figure 4 in the global rebuttal PDF.)

Again, we would like to reiterate that our core argument is that VP models trained with FACL achieve more realistic and skillful forecasts than those with MSE. The inclusion of generative models serves as a reference to the readers on how well generative models perform under the same setting. We believe further including more references has limited help in delivering our core argument.

---

In figure9, ..., the FACL methods still lack predictions for extreme value...

To address the reviewer’s concern, let us present the individual scores with high thresholds: (160, 181, 219) for different pooling sizes 1, 4, and 16 on SEVIR.

From the table, FACL still exhibits better performance in most thresholds, except under a large pooling size. Although generative models attempt to generate more extreme values, the predicted points are distant from the actual location, forming false positives and hence lowering the CSI score. On the other hand, models trained with FACL, despite not predicting extreme values a lot, hit when they do. After considering everything in the intersection-over-union fashion, FACL can usually outperform the reference generative models in predicting extreme values unless with exceptionally large allowance.

---

The intuition of FACL has no strong connection with precipitation nowcasting.

The intuition of FACL has the following connection with precipitation nowcasting:

(1) FACL is proposed to solve blurriness caused by (random) motion. Previous video prediction models keeps improving on deterministic data by looking into pixel-wise metrics. FACL utilizes the translation invariant property of FAL by treating precipitation events as stochastic data.

(2) FACL is proposed to work on signal-based data. We assume there is no cohesiveness between channels such as RGB raw image and medium-range atmospheric forecasting, most preferably having only 1 channel with its range bounded. Among the data types, radar reflectivity and its derivation products such as VIL are good candidates with such properties.

---

The coefficients of FAL and FCL needs to be selected manually...

We refer the reviewer to the extra experiments for reviewer w5PB, which shows $\alpha$ is consistently stable in the suggested range (0.1 - 0.4).

---

Minor: Models adopt the “prediction and refine” may help improve the related work section.

Thank you for the suggestion. We will include a few more recent models such as CasCast and DGDM to extend the introduction to the “prediction and refine” type of work.

---

CSI thresholds for MeteoNet

Thank you for the feedback. The overall idea of choosing the CSI threshold was to select values that span the low, medium, and high range, so we conveniently followed the set used by SEVIR. We will update the CSI thresholds for MeteoNet following the DiffCast paper, by setting them to (12, 18, 24, 32) over 70 dbZ. A preview is reported in the comment.

We would like to thank the reviewer’s feedback and suggestion. Here is our response to each of the reviewer w5PB's concern.

---

Instead of the stochastic modification to Moving-MNIST, [1] already introduced a chaotic yet deterministic N-Body MNIST to mimic the complexity of Earth system interactions, would have been interesting to see the performance on this dataset.

We are pleased to also present the result of N-Body-MNIST. Due to limited time and resources, we trained ConvLSTM models with MSE and FACL on N-Body-MNIST for 100 epochs, with the results shown below:

For visualization, see Figure 1 in the global rebuttal PDF

Side notes regarding N-Body-MNIST:

As the reviewer suggested, N-Body-MNIST is deterministic in a closed system. However, we argue that the current research gap (blurry pattern for VP model) is the randomness causing VP models to predict poor quality forecasts. Such randomness cannot be simply solved by “improving the DNN models” as attempted in previous works. Strong models like Earthformer bring huge improvement to N-body-MNIST due to their increase of representation capability. However, upon non-deterministic motion (like Stochastic Moving-MNIST, and actual precipitation events), these strong models’ benefit is limited and they still suffer from blurry predictions. With such intuition, we point the culprit at the MSE loss, which mixes uncertainty into pixel value. By suggesting Stochastic Moving-MNIST as a non-deterministic system, we showed that a tiny stochastic factor (~1 pixel shift per frame) is sufficient to cause severe blurriness for the VP models trained with MSE regardless of their representation capability (as visualized in Figure 8 in the paper). FACL, on the other hand, enables the model to learn the expected trajectory and forms a clear output.

---

FVD and LPIPS calculated on the basis of models trained on natural images, difficult to see why it would extend to scientific data

Due to the blurriness issue, despite DNN models’ remarkable accuracy, meteorologists find that the model forecasts do not “look like” actual observations. The intuition of using FVD and LPIPS is to show how close the forecasts “look like” actual ones, as a quantified approximation to qualitative studies by humans. For example, the DiffCast [1] paper also reported LPIPS to represent perceptual similarity.

We agree with the reviewer that FVD and LPIPS only do not extend well to scientific data. This is also one of our motivations for proposing RHD, to provide a systematic way of measuring the forecast with a closer approximation to human perception. However, we still include the perceptual metrics to provide readers with a comprehensive study in all three aspects: accuracy (MAE, SSIM), visual realisticity (LPIPS, FVD), and skillfulness (CSI, FSS).

[1] D. Yu, X. Li, Y. Ye, B. Zhang, C. Luo, K. Dai, R. Wang, and X. Chen, “DiffCast: A unified framework via residual diffusion for precipitation nowcasting,” in CVPR, 2024.

---

Does the alpha parameter swing wildly between datasets? Any good strategy to figure this out?

We also conducted experiments on $\alpha$ using other datasets such as SEVIR.

Similar to the observation in Table 4, with the increase of $\alpha$ from 0 to 0.3, pixel-wise performance gradually drops and perceptual metrics gradually improve. At around 0.4-0.5, where the model may not fully converge in FCL, the performance starts to decay. Therefore, $\alpha = 0.1$ or $0.2$ as a good default value still holds.

The rule of thumb here is only exposing strong FAL to models that have been fully converged in FCL. (e.g. $\alpha < 0.5$, or have a long training time overall). Once the models converge in FCL, the effect of the proportion of FAL (0.1 - 0.4) is not very significant but a small tradeoff between pixel-wise accuracy and sharpness.

---

What are the benefits of RHD over a simple wasserstein distance between predicted and true distributions or their quantile error?

One key component of RHD is to divide the forecast map into smaller patches. With this operation, the measured intensity distribution is limited to a local region, rather than the whole map. As for the part of KL divergence, we believe it plays a similar role as Wasserstein distance. In other words, a similar measurement can be achieved if we keep the patching component and replace the KL-divergence with the EM distance.