CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting

We apologize for the ambiguity.

W1: What does non-optimizable term $\epsilon$ mean?

A1:

• Please refer to Eq(4), since test label $x_{t+m:t+m+n}$ is unavailable during dataset condensation, we transform it to the prediction of an expert model $M_f$ on test data $x_{t:t+m}$ for further analysis. Since there is always an error between the prediction of the model and the ground truth, we use $\epsilon$ to denote it.
• $\epsilon$ is optimized during the training process of expert model $M_f$. However, since we can only have access to the trained expert model $M_f$ during dataset condensation, $\epsilon$ is no longer optimizable.

---

W2: Why derive a bound of the gradient term?

A2:

• Indeed, the original form of gradient term in Eq(5) is not directly optimizable. Test data $x_{t:t+m}$ exists in gradient term, which is unavailable during dataset condensation.
• To solve this problem, please refer to Eq(6), we use Cauchy-Schwarz Inequality to decouple gradient term. Then we get the optimizable part $||\theta_f-\theta_s||^2$(the gradient of linear model is the parameter), and the non-optimizable part $||x_{t:t+m}-s_{t':t'+m}||^2$.
• Thus, it is necessary to derive the upper bound of gradient term because the original one is not optimizable.

---

W3: How to justify the empirical finding?

A3: In the cases you mentioned, the model is theory-driven, which means the prediction of the model relies highly on the hypotheses and theories. However, in this case, the models are data-driven, and no additional hypotheses are made. The prediction of the model relies only on the training data and the model structure. Thus, in this case, if the models share the same architecture and are trained with the same dataset, empirically they output similar outputs.

---

W4: How does our method compare to smoothing or model distillation?

A4:

• Firstly, model distillation and dataset condensation are different tasks. Model distillation updates student model parameter and produces distilled model. Dataset condensation updates the synthetic data and produces distilled dataset.
• Secondly, model distillation aims at aligning the output of $M_s$ and $M_f$. The training of $M_s$ is directly supervised by $M_f$, where $L$ is the loss function, e.g. MSE. $M_s=\arg\min_{M} L(M(s),M_f(s))$ However, in the case of our value term optimized by Eq(12), $M_s$ is trained alone using synthetic data $s$ and $M_f$ only has indirect access to the training of $M_s$. $s=\arg\min_s L(M_s(s),M_f(s))$ $**s.t.**\hspace{1em} M_s=\arg\min_M L_{train}(M, s)$ Thus model distillation is not comparable to our method.
• Meanwhile, we conduct experiments using LPF to perform target smoothing on the synthetic data and compare the results with CondTSF. Average MAE of 5 models is reported below. It shows that CondTSF is significantly more effective than target smoothing.

---

W5: What if the value term or the gradient term is removed? What are the impacts of these two terms?

A5: $L_{param}$ is the same form of Eq(8) in Sec.4.2, which is used to optimize gradient term. Please refer to l.143-145, our analysis shows that previous dataset condensation methods based on parameter matching, e.g. DC[1], MTT[2], etc. are optimizing the gradient term. Our method CondTSF is optimizing the value term.

• If value term is removed: The method falls back to the previous dataset condensation methods based on parameter matching(DC, MTT, etc.), and the results can be found in Table.1,3,5,7,9. Moreover, please refer to Fig.8, in the beginning 160 epochs, only gradient term is optimized, in the last 40 epochs, value term is also optimized. Optimizing value term significantly enhances the performance.
• If gradient term is removed: Please refer to l.222-224 and App.C, experiments on backbone methods that are not based on parameter matching, e.g. DM[3], IDM[4], etc. are also carried out. These backbone methods do not optimize the gradient term. Thus when combining these backbone methods with CondTSF, only value term is optimized, and the results are in Table.8. We conduct experiments that only optimize value term without any backbone methods, average MAE of 5 models is reported. The results are well-aligned with our analysis that optimizing gradient term or value term alone harms the performance.

[1] Dataset Condensation with Gradient Matching

[2] Dataset Distillation by Matching Training Trajectories

[3] Dataset Condensation with Distribution Matching

[4] Improved Distribution Matching for Dataset Condensation

---

W6: Is this method applicable to all regression tasks?

A6: Our proposed CondTSF is applicable to generic regression. Meanwhile, we have designs that are related to time series forecasting. In time series forecasting, when generating training data, the data is usually sampled overlap from the dataset. For instance, the data points can be sampled as $x_{0:24}, x_{4:28}$, etc. Please refer to Eq(12), inspired by the overlap property, we utilize an additive method to gradually update the synthetic data instead of directly using $M_f(s_{t':t'+m})$ to overwrite $s_{t'+m:t'+m+n}$ to avoid vibrations.

---

Minor: $k$ in l.147-152 used before introduced.

A: We are sorry for the ambiguity. We will correct that in next version.

W1: The demand for dataset condensation for time series forecasting is not urgent. What are the real applications of dataset condensation for time series foundation models?

A1:

• Firstly, dataset condensation is also important in other aspects besides reducing computational cost. For instance, data condensation is proven to benefit downstream tasks such as continual learning[1], privacy[2], etc. These tasks are vital for some aspects of the time series, such as finances.
• Secondly, in the era of large models, Neural Architecture Search(NAS) of large time series foundation models is still computationally intensive[3]. Utilizing condensed data to identify a smaller parameter set and conducting full training on this set can improve efficiency.
• Moreover, our proposed method could serve as a prototype and shed light on the related research area.

[1] An Efficient Dataset Condensation Plugin and Its Application to Continual Learning

[2] Privacy for free: How does dataset condensation help privacy

[3] Dataset Condensation for Time Series Classification via Dual Domain Matching

---

W2: How the time series foundation model performs remains unclear, and most are not good at zero-shot learning.

A2:

• Indeed, nowadays foundation models do not behave well in time series forecasting[1].
• However, more explorations of foundation models are carried out[2]. Therefore, our dataset condensation method can be used to lower the computational cost and accelerate the exploration of foundation models on time series tasks[3][4][5].
• Meanwhile, our proposed method points out an interesting direction in time series forecasting tasks. More efforts can be devoted to this field and our method can serve as a prototype.

[1] Are Language Models Actually Useful for Time Series Forecasting?

[2] A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Model

[3] Time-llm: Time series forecasting by reprogramming large language models

[4] Timegpt-1

[5] A decoder-only foundation model for time-series forecasting

We apologize for the ambiguity.

W1: What are the sample space of $x, s$? Are they univariate time series?

A1: Yes, $f, x, s$ are univariate time series. Please refer to Sec.3, l.93-95. A time series dataset can be viewed as a long vector. We cut the dataset into two vectors and obtain train set $f$ and test set $x$. Both $x$ and $f$ are vectors. Then we sample a short continuous part of train set $f$ as the initial synthetic dataset $s$. $s$ is still a vector.

---

W2: What is $t'$ in Eq(5)? Is it summed over?

A2:

• We use $s_{t':t'+m}$ to denote a vector(part of $s$) with length $m$ starting from position $t'$. $t'$ is arbitrarily chosen.
• Please refer to Eq(16) and l.444-446, $s_{t':t'+m}$ is an arbitrary vector used to perform Taylor expansion to get the value of $M(x_{t:t+m})$. The idea throughout our paper is that test set $x$ is unavailable during dataset condensation, thus we need to substitute $x$ with accessible synthetic dataset $s$ so the optimization objective becomes optimizable.
• The sum in Eq(5) is only performed on $t$, not $t'$. Please refer to Eq(3) and App.A1, $\sum_t$ is used in test error to sum over the test set $x$. For each $x_{t:t+m}$, we use an arbitrary $s_{t':t'+m}$ to perform Taylor Expansion, thus $t'$ is not summed over.

---

W3: How are Eq(7) and(8) optimized? Lack of information on base methods and the novelty is not clear.

A3:

• Eq(7) and(8) are optimized the same way as MTT[1], which is a well-acknowledged dataset condensation method. According to our analysis, MTT is optimizing the gradient term. There are multiple dataset condensation methods that use MTT as the backbone, such as TESLA[2], FTD[3], DATM[4], etc. We will add introduction of MTT in appendix in the next version.
• MTT and all other backbone methods are designed for classification. They perform poorly on time series forecasting. The novelty of our method is that CondTSF enhanced the performance of all previous methods on time series forecasting.

[1] Dataset Distillation by Matching Training Trajectories

[2] Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory

[3] Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation

[4] Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching

---

W4.1: The hypothesis of linear model trivialized the work.

A4:

• Please refer to App.A1 and Eq(16), using nonlinear models will add 2nd and higher order terms to the Taylor Expansion. We ignore higher order terms because their effect on the output is minor. Meanwhile, although linear models are simple in structure, they perform well on time series forecasting and outperform complicated transformer models[1].
• We add experiments on using nonlinear models as expert models, please refer to A7.

[1] Are Transformers Effective for Time Series Forecasting?

---

W4.2: The observation in Eq(9) is a simplification.

A5: The observation in Eq(9) is not a simplification. Please refer to l.157-159, The test model parameter is unknown during dataset condensation, thus we need to substitute them with accessible parameters. Thus Eq(9) is necessary, otherwise the value term is not optimizable.

---

Q1: Why is the result of one-shot dataset for models MLP, LSTM, CNN presented in the main text?

A6: Cross-architecture performance is an important metric reported in all dataset condensation papers. MLP, CNN and LSTM perform well on time series forecasting[1] and are widely used in the evaluation of dataset condensation[2]. Thus we present the result in the main text.

[1] Are Transformers Effective for Time Series Forecasting?

[2] Dataset Condensation for Time Series Classification via Dual Domain Matching

---

Q2: Add a nonlinear expert?

A7: Please refer to App.A1 and Eq(16). Given nonlinear models, our method ignores the 2nd and higher order terms in Taylor Expansion. We conduct experiments to show that ignorance is acceptable. We use nonlinear models CNN, 3-layer MLP(ReLU activated) as expert models. Average MAE of 5 models is reported below. The results show that CondTSF performs well with nonlinear models.

• CNN

• 3-layer MLP(ReLU activated)

---

Q3: What's the novelty beyond time series?

A8:

• All previous dataset condensation methods based on classification perform poorly on regression. The core novelty of our method is that it is the first dataset condensation method focusing on regression and significantly enhances the performance of all backbone methods.
• In time series forecasting, when generating training data, the data is usually sampled overlap from the dataset. For instance, the data points can be sampled as $x_{0:24}, x_{4:28}$, etc. Please refer to Eq(12), inspired by the overlap property, we utilize an additive method to gradually update the synthetic data instead of directly using $M_f(s_{t':t'+m})$ to overwrite $s_{t'+m:t'+m+n}$ to avoid vibrations.
• Additionally, other content related to time series can be explored in future works. We presented a simple yet effective method, which can serve as a prototype.

Q1: Is the claim "models initialized with different samples of parameter values predict similarly after trained on the full dataset given arbitrary input" in line 162 or Eq(9) supported by any evidence?

A1: We apologize for the ambiguity. Please refer to Figure 3 and line 166-168, we support this empirical observation with experiments. The large orange points in the figure are input data, and the yellow and blue points are predictions of the models initiated with different parameters. We used MDS algorithm to reduce the dimension of the original prediction and data. MDS algorithm preserves the distance between points, which means points closer to each other in high dimensions are also close in low dimensions. It can be observed that the predictions of the models are similar despite their initial parameters are different.

---

Q2: Why the label error on $s$ will increase when the parameters of $M_s$ are matched.

A2: We apologize for the ambiguity.

• Firstly, The label error is used to evaluate the synthetic dataset $s$ instead of the expert model $M_f$. Please refer to Thm.2 and Eq(11), label error is an upper bound of the value term. Therefore label error ensures the similarity between prediction values of $M_f$ and $M_s$.
• Secondly, the parameter loss is an upper bound related to the gradient term. It only ensures $M_s$ and $M_f$ have similar gradients on the input data, yet it does not ensure their value of predictions are similar.
• To sum up, our analysis in Sec.4.1 and Thm.1 showed that value term and gradient term are decoupled in the optimization. Parameter loss relates to the upper bound of gradient term while label error is the upper bound of value term. Therefore, even when parameter loss is small, the label error can still be large.