Thank you for your feedback.

The MLSE loss function is specifically designed to address long-term forecasting problems by utilizing the concept of an 'error amplifier'. It is not a universal loss function for arbitrary timeseries tasks. We value your suggestions and are planning to extend our research to cover these interesting perspectives.

Answer to question

• We report the Root Mean Square Error (RMSE) in terms of two weather states, the T850 and the Z500, which are the crucial measurements in weather simulation tasks.

• We aim to test the effectiveness of MLSE across different models, various dataset scales, and disparate dataset resolutions.

• The $M_{t+N-1}^{N-1}$ is the propagation matrix produced by the Jacobian operator on multi-dimension Mean Value Theorem:.You can simply regard it as the $\nabla f_\theta({X}_{\delta})$. It is a matrix, if the input $X$ is a vector.

Thank you for your feedback.

due to the $|| {\epsilon}_{t+N}^{I} ||$ is the first order error and should be optimized in the pretrain phase for a pretrain-finetune diagram.

and the $||\epsilon_{t+N}^{N} ||$ is yet optmized, shoud much large than $|| {\epsilon}_{t+N}^{I} ||$

Questions:

For how many orders does the Dalpha strategy optimize for ? Does it pick one order ?

Figure 5 should the notation of Dalpha for first order and first + second order. Compute the Dalpha requires one-more forward operation (achieve t+N+1) compare to tradition long-term error (who only forward to t+N) and use the extra prediction compute the error. Thus, Dalpha-1 actually use the second order prediction (notice, not the error). And the Dalpha-2 use the third order prediction.

How is MASE an improved alternative over Dalpha? They seem two different things altogether. Dalpha optimizes the relative errors while MASE optimizes each error separately

Lets take the first order as example, the full Dalpha requires optimize the e1 and e2'/e1. It requires spontaneously minimum the e1 and maximum e1. This make training hard. In MASE we just remove the denominator.

I don't understand how the conclusion towards using the logarithm is made, in the paper it's just mentioned that "it should be", but that's not sufficient.

In our paper, we approached the optimization problem by analyzing an amplifier due to the denominator. Therefore, we switched to optimizing its logarithmic form. Let's delve deeper into the derivation for two timestamps optimization:

Notice minimize $\alpha^{II}$ equals to minimize $\frac{\mathcal{E}^{II}}{\mathcal{E}^{I}}$ which can be converted to log format as minimized $\log(\mathcal{E}^{II}) - log(\mathcal{E}^{I})$. Remember we still need minimize $\mathcal{E}^{I}$ which is same as minimize $\log(\mathcal{E}^{I})$. These two part are independent implying that we need to assign two coefficients to combine them like:

They are hyper-parameter and we just set the $a = 1, b = 2$ in this paper. This results in:

It's crucial to note there is no explicit theoretical formulation mandating the use of the logarithm. The MASE can also be the feasibly accommodation for using amplifier information. However, it need an additional forward computation for an extra peek into the future. And we want a zero-cost operation. Therefore we name this technology as a trick and leave its validation in a comprehensive experiments.

The equation $||\epsilon_{t+N}^{N} - \epsilon_{t+N}^{I} ||$ represents the difference between the error of a multistep forward prediction and the first step error. In the pretraining phase, we optimize the first step error $e_1=f(x)-y$ over a long period of epochs, resulting in a small error ($e_1$).

In contrast, during the pretraining phase, we place no constraints on the multistep forward error like $e_N=f(...f(f(x)))) - y$, so at the beginning of the fine-tuning phase, it is larger ($e_N >> e_1$) because it has been under-optimized. You can consider it as a summation of large vector $\epsilon_{t+N}^{N}$ plus a perturbation vector $\epsilon_{t+N}^{I}$. And resulting $||\epsilon_{t+N}^{N} - \epsilon_{t+N}^{I} || \approx ||\epsilon_{t+N}^{N}|| - || \epsilon_{t+N}^{I} ||$.

After the fine-tuning phase, this approximation remains valid for large $N$. For smaller $N$ (like $N=2$), the ratio $||\epsilon_{t+N}^{II}||/||\epsilon_{t+N}^{I}||$ is approximately $1.4$, indicating that the multistep forward error is larger, but not dramatically so.

Indeed, we can also consider the approximation from the perspective of bounds. The term $||\epsilon_{t+N}^{N} - \epsilon_{t+N}^{I} ||$ can be bounded as:

This implies that the difference between the multistep forward error and the first step error lies between the multistep forward error minus the first step error and their sum. This offers another perspective to understand the formulation and can serve as the basis for developing a lower/upper bound approximation. We can then aim to optimize the model performance with respect to these bounds.

Thank you for your feedback.

• We would like to emphasize that, to the best of our knowledge, this is the first paper that attempts to analyze the problem of error accumulation and proposes an effective and cost-free method to mitigate it.

• The experimental setup in this paper sets one step, $n=1$, to be equivalent to 6 real-time hours. Therefore, $n=12$ spans over a period of 3 days and $n=20$ over 5 days.
• It is to be expected that the performance advantage of the proposed method decreases for T+3, T+4, and so forth. This occurs due to the rapid decay of time correlation between T and T+4, making the addition of more timestamp information considerably inefficient. Owing to resource constraints, we may not be able to make a comprehensive comparison for larger $n$ values.

Please see follow is the Z500 score for the WeatherBench32x64-55k dataset and FourCastNet model. The finetune phase start from same pretrain weight and run in same hyperparameter like lr and optimizer and 100 epochs.

You can see

• The MLSE always outperform the MSE case.
• The effect for more future information (T->T+n) is decay
  • Notice, we directly add higher order term for one finetune phase in above experiment. We set this just for comparison. In practice, it is better do multiple finetune phase like first train for T->T+1, then T-> T+2 then T->T+3, and so on.

Thanks for extra experiment and clarification. The experimental results also shows the good performance given large forecasting horizon. Overall, after reading the other reviews, I decide to increase my score but tend not to accept the paper.

Thank you for your invaluable input. However, there seems to be a slight misunderstanding regarding the methodology employed in our paper.

The reviewer seems to be under the impression that we have used only the highest order error term for model optimization. This is not the case. In actuality, we have utilized all error terms during the training process. The term "fine-tuning" in our context refers to optimizing the entire trajectory, rather than a single step.

As such, the concern that fine-tuning a neural network may lead to the original concept being forgotten, or that it would cause the model to focus excessively on predicting a particular time step, is not applicable in this instance. Our model is designed to perceive and correct for 1st order error consistently, ensuring that it does not face an extreme 'forgetting' challenge.

Additionally, we would like to clarify that our experiments were conducted with utmost care to ensure fairness. The correct prediction sequence was always used for error computation and comparison. We strive for transparency in our research and stand by the fairness of our experimental design. We believe that these clarifications address your concerns, and we appreciate your engagement and insightful feedback.

One by one response:

Response to Weaknesses.1 : We would like to clarify a misunderstanding: The loss function used in our study is the Mean Log Squared Error (MLSE)$\sum ln(x_i^2)$, not the Mean Squared Logarithmic Error (MSLE) $\sum(ln(1+x_i)-ln(1+\bar{x}))^2$. These are fundamentally different concepts with different applications and interpretations. Our choice of MLSE was deliberate and based on the specific objectives and demands of our research. To the best of our knowledge, there has been no explicit discussion of the influence of MLSE in similar contexts. Therefore, we believe our exploration of this area offers a unique contribution to the field.

Response to Weaknesses.2 : We appreciate your interest in the detailed experimental setup. However, we believe there might be some confusion regarding the error calculation process we deployed. In table 2, our methodology involves calculating the error as an average across all time step predictions, such as 0 -> 96, 0->196, etc. This approach aligns with the paradigm used in the previous work like TimesNet, ensuring a fair basis for comparison between our ATT results and E2E results. A more clean clarify below:

• Train
  • the E2E model train on setting that use seq[0:96] to predicted seq[96:96+720]
  • the ATT model trained on setting that used seq[0:96] to predict seq[96:192] and autoregressive for seq[192:192+96].
• Testing:
  • the E2E use seq[0:96] to predicted seq[96:96+720] in one step
  • the ATT use seq[0:96] to predicted seq[96:96+720] in 720//96 + 1 times The error is compute via take average for each timestamp prediction for label[96:96+720]

Response to Weaknesses.3.a: The key challenge of long-term autoregressive optimization lies in the substantial resources required for storing optimizer-intermediates. For a more in-depth technical understanding, we refer you to studies such as Pangu, Fengwu, and GraphCast. To illustrate this challenge, consider the process of iterating the model $f$ multiple ($N$) times to constrain the long-term autoregressive error. When using an autodifferential engine to compute the gradient, it necessitates the storage of $N$ times the optimizer state. In a scenario where we use the ViT architecture (usually at least 400M parameters) and consider one image input as a (70, 1440, 720) image (a normal case for Weather data), it would require approximately 40G*N of GPU memory per update even under half precision. This presents a significant challenge when training a skillful long-term prediction model. Our research offers a unique solution to this challenge. We found that by merely changing the form of the loss from Mean Squared Error (MSE) to Mean Log Squared Error (MLSE), we can enhance performance using a smaller $N$. This approach significantly reduces the resources required for training while maintaining, or even improving, performance.

Response to Weaknesses.3.b: Kindly notice the fine-tuning procedure optimizes both the $\mathcal{E}^I$ and $\mathcal{E}^{II}$ error terms. This is in contrast to the pretraining phase, which only optimizes $\mathcal{E}^I$. Therefore, the fine-tuning phase does not cause the model to forget the original concept learned but instead enriches it with additional information pertaining to $\mathcal{E}^{II}$.

Response to Weaknesses.4: To address your query, we have provided extensive information on the experiment setting, including the hyperparameters and the 'mean and variance' in the appendix. Specifically, the hyperparameters are outlined on pages 13 and 14, and a snapshot view of the 'mean and variance' can be found on pages 9, 10, and 11.