FEDNET: FREQUENCY ENHANCED DECOMPOSED NETWORK FOR OUT-OF-DISTRIBUTION TIME SERIES CLASSIFICATION

model complexity analysis(Q3) We have provide model commplexity in appendix D.8 MODEL COMPLEXITY.

Given a time series data $L \times C$, where $L$ donates its input length and $C$ represents the number of channels, FFT complexity is $O(L\log L)$. The time-deterministic complexity is $O(2 \times N \times L \times C \times C_{out} \times K)$ comes from the encoder and decoder, where $N$ is the number of hidden layers, $K$ is the kernel size of the 1D-Conv and $C_{out}$ is the output channels. The time-stochastic part uses patchify to reduce tokens in attention layer, with complexity $O(M^2)$ where $M = L/S \ll L$, $S$ is patch stride.

1. We have performed some lightweight MLP Encoder processing time-deterministic,it is feasible.

2. from time-stochastic study, we found unfold patches for time-stochastic featrues achieves the fast speed for self-attention and reduce $x_{patch}$ condition number. P donates patch length while S represents stride.

3. we also provide different time-stochastic encoders like MLP and depthwise separable convolution on EEG target domain-2.

limitation and time stochastic study(W1, Q1, W3): We complement the time-stochastic feature study with a three-level(feature, data, and domain) analysis, discovered some of the problems that the model would face. The shortcomings of our approach mainly stem from two aspects; on the one hand, the results usually become poor when the domain invariant features are insufficient or corrupted by deletions. On the other hand, the over-reliance on the spectral distribution of the source domain leads to bias in the invariant features that may be learned. A detailed analysis can be found in the response of Reviewer b2Zv time-stochastic study (W4).

writing error(W2): Thank you for your very careful review! we have fixed it.

comparison on general classification task(Q2) Thanks for mentioning this interesting issue! We find that our method can be trained and predicted directly in generalized temporal classification and can get close to the sota level. We chose the methods PatchTST, Dlinear,iTransformer,FEDformer,TimesNet for comparison control, epochs = 100, and the rest of the parameters were generically not turned on. We found that this VAE form of modeling also achieves good results on regular datasets.

By the way, we have added a lot of theoretical and experimental content in this new version, including the role of the two components, the effect of facing missing timing data, theoretical generalization upper bounds and frequency domain isolation levels, etc. We hope that these contents will be of interest to you as well :)

These upper bound definitions are also all based on the Covariate Shift Assumption.

Domain Adaptation Error Bound

Theorem 1 Domain adaptation error bound [1]

Let $d$ be the Vapnik–Chervonenkis (VC) dimension [2] of $\mathcal{H}$, and $D^s$ and $D^t$ be unlabeled samples of size $n$ from the two domains. Then for any $h \in \mathcal{H}$ and $\delta \in (0, 1)$, the following inequality holds with probability at least $1 - \delta$:

where $\hat{d}_{\mathcal{H}\Delta\mathcal{H}}(D^s, D^t)$ is the estimate of

$d_{\mathcal{H}\Delta\mathcal{H}}(P_X^s, P_X^t)$ on the two sets of finite data samples. $\epsilon^s(h)$ denotes ideal target risk and $\lambda_H$ denotes ideal domain divergence, they are both unpredictable.

In fact, $\hat{d}_{\mathcal{H}\Delta\mathcal{H}}(D^s, D^t)$ is the same measture $\mathcal{A}$-distance often used for estimate domain divergence, so that the smaller the distance is, the risk bound will be smaller. We also use it in our experiments in Invirant Feature Study.

Theorem 2 PAC-Bayesian Perspective on Domain Adaptation [3]

Let $\mathcal{H}$ be a hypothesis space, let $\mathcal{S}$ and $\mathcal{T}$ respectively be the source and the target domains on $\mathcal{X} \times \mathcal{Y}$. Let $q > 0$ be a constant. We have, for all $\rho$ on $\mathcal{H}$,

where $d_{\mathcal{T}}(\rho)$ is denote to ideal target risk (unpredictable), $e_{\mathcal{S}}(\rho)$ is empirical source risk, $\beta_q(\mathcal{T} \| \mathcal{S})$ denote the domain divergence, and $\eta$ estimate the joint error when $SUPP(\mathcal{T})$ is not entirely included in $SUPP(\mathcal{S})$.

[1] Ben-David S, Blitzer J, Crammer K, et al. A theory of learning from different domains[J]. Machine learning, 2010, 79: 151-175.

[2] Vapnik V, Levin E, Le Cun Y. Measuring the VC-dimension of a learning machine[J]. Neural computation, 1994, 6(5): 851-876.

[3] Germain P, Habrard A, Laviolette F, et al. A new PAC-Bayesian perspective on domain adaptation[C]//International conference on machine learning. PMLR, 2016: 859-868.

Thank you very much for your support of the theoretical part of our work, we sincerely hope that these replies can let you regain confidence in our work. If you still have any questions, you do not hesitate to tell us, if it is still on the experimental aspects of the existence of some doubts, we will do our best to add all the results of the experiments in the next period of time, and we still have time to modify the pdf tomorrow, if you If you think these theoretical contents need to be added into the pdf, we will add them in time, your advice is really important to us and wish you a happy Thanksgiving!

Domain generalization Error Bound itself are all conclusions of the extension of the domain adaptive upper bound in the case of multiple domains.

Theorem 3 Domain generalization error bound [1]

Let $\gamma := \min_{\pi \in \Delta_M} d_{\mathcal{H}\Delta\mathcal{H}}\left(P_X^t, \sum_{i=1}^M \pi_i P_X^i\right)$ with minimizer $\pi^*$ be the distance of $P_X^t$ from the convex hull $\Lambda$, and $P_X^* := \sum_{i=1}^M \pi_i^* P_X^i$ be the best approximator within $\Lambda$. Let $\rho := \sup_{P_X', P_X'' \in \Lambda} d_{\mathcal{H}\Delta\mathcal{H}}(P_X', P_X'')$ be the diameter of $\Lambda$.

Then it holds that

where $\lambda_\mathcal{H}(P_X^t, P_X^*)$ is the ideal joint risk across the target domain and the domain with the best approximator distribution $P_X^*$.

Theorem 4 PAC-Bayesian Domain generalization error bound [2]

Let $\mathcal{H}$ be a hypothesis space built from a set of source domains, denoted as $D = \{D_i\}_{i=1}^{N_d}$.

Suppose $q > 0$ is a constant, for any unseen domain $D_{U}$ from the convex hull $\Lambda_D$,

where $d_D(h)$ and $e_D(h)$ are ideal and expected risk of a domain $D$ respectively, $\epsilon = \beta_{q}(D_U \| \sum_{i=1}^{N_d} \pi_i D_i)$ is an ideal distance since we can't have access to $D_U$, while $\eta_{T/S}$ denotes the distribution of $(x, y) \in \text{SUPP}(Test) \backslash \text{SUPP}(Source)$, it is usually a small value. Suppose $(x, y)$ between the unseen domain for testing and source doamins have been fully covered by $\Lambda_S$, then $\eta_{T/S} = 0$ and there exists a finite upper bound $\rho = \sup_{i,j \in [N_d]} \beta_q(D_i \| D_j), \forall q > 0$ for any convex combinatorial domains $D_i$ and $D_j$.

We introduced the PAC-Bayesian error bound to justify the conclusion because here $\beta_q(D_i \| D_j)$ can be estimated using the marginal probabilities P(X) in any two domains, such as $D_i$ and $D_j$, or $D_{k}^i$ and $D_{k}^j$, while the original method can only be be tested by feature and cannot be linked to the frequency domain or any data prior. I hope this gives you a better understanding and contribution of Proposition 5.2 Frequency Perspective Risk Bound on Unseen Time series Domain.

[1] Albuquerque I, Monteiro J, Falk T H, et al. Adversarial target-invariant representation learning for domain generalization[J]. CoRR, 2019.

[2] Germain P, Habrard A, Laviolette F, et al. A new PAC-Bayesian perspective on domain adaptation[C]//International conference on machine learning. PMLR, 2016: 859-868.

We apologize for the confusion caused by the introduction of too many concepts in the new commit, in fact, the concepts we mentioned are the basic concepts within domain generalization.

To further clear your confusion, we will give you a detail introduction about domain generalization.

1. Domain [1]

Let $\mathcal{X}$ denote a nonempty input space and $\mathcal{Y}$ an output space. A domain is composed of data that are sampled from a distribution. We denote it as $D = [(x_i, y_i)] \sim P_{XY}$ contains $n$ samples, where $x \in \mathcal{X} \subseteq \mathbb{R}^d$, $y \in \mathcal{Y} \subseteq \mathbb{R}$ denotes the label, and $P_{XY}$ denotes the joint distribution of the input sample and output label. $X$ and $Y$ denote the corresponding random variables.

2. Covariate Shift Assumption [2]

Given the training andthe test data from two distributions $P_{XY1}(x, y)$, $P_{XY2}(x, y)$, covariate shift is referred to the case that the marginal probability distributions are different, and the conditional distributions are the same, i.e.,$P_{XY1}(x) \neq P_{XY2}(x)$, and $P_{XY1}(y|x)= P_{XY2}(y|x)$.

3. Domain Adaptation (DA) [3]

Before we get into domain generalization, we'd like to introduce you to a simplified version of it, which is one of the earliest types of OOD problems you'll have heard of. DA aims to maximize the performance on a given target domain using existing training source domain(s).

DA problems will usually only have 2 domains and hold on the Covariate Shift Assumption i.e. $P_{source}(X) \neq P_{target}(X), P_{source}(y|x)= P_{target}(y|x)$. Most importantly, it allows you to use the input data $X_{test}$ from the test set to align.

4. Domain Generalization (DG) [1]

In domain generalization task, we are given $M$ training (source) domains $D_{train} = \{ D^i \mid i = 1, \cdots, M \}$, where $D^i = [(x_j^i, y_j^i)]$ denotes the $i$-th domain contains $j=[1 ... n_i]$ samples. The joint distributions between each pair of domains are different: $P_{XY}^i \neq P_{XY}^j, \quad 1 \leq i \neq j \leq M.$ (usually hold on the Covariate Shift Assumption $P_{X}^i \neq P_{X}^j, P_{Y|X}^i = P_{Y|X}^j$ ) The goal of domain generalization is to learn a robust and generalizable predictive function $h : \mathcal{X} \to \mathcal{Y}$ from the $M$ training domains to achieve a minimum prediction error on an unseen test domain $D_{test}$

where $\mathbb{E}$ is the expectation and $\ell(·, ·)$ is the loss function. In addition, we can not access to the test data $D_{test}$ which is different from DA.

5. Temporal Covariate Shift [4]

Given a time series data $D$ with $n$ labeled segments. Suppose it can be split into $K$ periods or intervals,
$D = [D_1, \cdots, D_K]$, where $D_k = [(x_i, y_i)]$, where $i=n_{k}+1, \cdots n_{k+1}$ , and $n_1 = 0$ and $n_{k+1} = n$. Temporal Covariate Shift (TCS) is referred to the case that all the segments in the same period $i$ follow the same data distribution $P_{D_i}(x, y)$, while for different time periods $1 \leq i \neq j \leq K$, $P_{D_i}(x) \neq P_{D_j}(x) \quad \text{and} \quad P_{D_i}(y \mid x) = P_{D_j}(y \mid x)$.

This is in fact the same as our definition of Temporal Distribution Shift, which is also followed in the latest paper Diversify[5], but none of them gives such a detail definition as ours, either in the time series or in the frequency domain.

And we would like to emphasize that in both Adarnn [4] and diversify [5] it is in fact impractical to address the problem of shifting time distributions by using $K$ as a hyperparameter for modeling the shifts to be possible categories, while in our definition we define $K$ as unpredictable to explore more essentially the impact of such shifts on the time series.

The formulas have been slightly adjusted due to some issues with markdown syntax display, and the full formulas can be found directly from the original references.

[1] J. Wang et al., "Generalizing to Unseen Domains: A Survey on Domain Generalization," in IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 8, pp. 8052-8072, 1 Aug. 2023, doi: 10.1109/TKDE.2022.3178128.

[2] Shimodaira H. Improving predictive inference under covariate shift by weighting the log-likelihood function[J]. Journal of statistical planning and inference, 2000, 90(2): 227-244.

[3] Wang M, Deng W. Deep visual domain adaptation: A survey[J]. Neurocomputing, 2018, 312: 135-153.

[4] Du Y, Wang J, Feng W, et al. Adarnn: Adaptive learning and forecasting of time series[C]//Proceedings of the 30th ACM international conference on information & knowledge management. 2021: 402-411.

[5] Lu W, Wang J, Sun X, et al. Out-of-distribution representation learning for time series classification[J]. ICLR 2023.

Thank you for your response!

1. For General time series classification("EthanolConcentration", "Handwriting", "Heartbeat") and UCIHAR,SHAR,Opportinity, we follows the origin paper settings[1], they choose validation and testing the same and get the best results, you can check the code in Time-Series-Library.

2. For the rest 6 dataset, we keep 6:2:2 spilting settings and we save the best validation results model for testings follows the origin paper settings in diversify[2], and our code uses the same dataset and dataloader of their implementation added into Time-Series-Library Framework.

3. For domain dividing, we follows the origin code implementations without change anything. All the datasets we use are open source, only PAMAP is reproduced by us as they do not have an open-source version may have some different, we provided all the links in our code README.

4. For some important hyperparameters, such as temperature we set 0.07 or 0.2, and for $\alpha$ we test with $[0.2, 0.4, 0.6, 0.8]$, we mainly tested 0.2 and 0.4, the latter 0.6 and 0.8 due to time constraints did not test to cover all datasets, and then some individual dataset domain we tried 0.5 and 0.1, for $\lambda_{det}$ and $\lambda_{sto}$ we set to default 1.0, and we have tested it with $[0.1, 1.0, 10.0]$ it doesn't change much. The other model's parameters we just followed the default code settings in the benchmark, we didn't change them.

We have give all the detail settings in the appendix D IMPLEMENTATION DETAILS and all the settings and evaluation are follow the origin papers and code, we don't change it.

From the results you can see that some datasets in point1 may face your concern and their performance gap between different methods is smaller than the rest dataset, especially on the performance of contrastive learning.

Has your confusion been resolved? If you have any questions or suggestions, we will continue to add the experiment. We still have one week to discuss :)

[1] Wu H, Hu T, Liu Y, et al. Timesnet: Temporal 2d-variation modeling for general time series analysis[J]. arXiv preprint arXiv:2210.02186, 2022.

[2] Lu W, Wang J, Sun X, et al. Out-of-distribution representation learning for time series classification[J]. arXiv preprint arXiv:2209.07027, 2022.

Thanks for your clarification.

It seems like the Definition 2 is not mathematically rigorous, and not practical, do you have any related work built on this definition? maybe it is good to clarify more on how you made this definition.

our previous setup on UCIHAR, SHAR, and Opportunity was mainly to ensure a consistent way of comparison with the original paper. To further address your confusion regarding some of the experimental results, we have currently performed a 6:2:2 data split on the UCIHAR, SHAR, and Opportunity datasets, and adapted them into the same framework as Diversify. We will provide you with these experimental results here, hoping to address your concerns about these datasets. We will provide you with feedback as soon as we have the complete experimental results.

Dear Fq1k, To address your concerns in experiments, we have also split the UCIHAR, SHAR, Opportunity dataset into 6:2:2 for evaluation and have already updated our code and scripts for them named UCIHAR2, SHAR2, OPP2. The conclusion is same as before. Our model achieved always shows the best performance across the domains and ERM methods like FreTS and Patchtst performance are usually worse than ood based methods unlike before. Here we list the experiment parameters we tested for all the methods.

• $learning_rate: 1e-2, 1e-3, 1e-4$
• $seed: 2023, 2024, 2025$
• $\alpha: 0.2, 0.4, 0.6$
• model parameters we didn't change.

Accuracy

F1-score

For Opportunity, we only test it with seed=2023 lr=1e-3 and $\alpha$=0.2 since the dataset is too large, it need very long training time, but it also shows competitive results. we will continue to working on it if we have time.

Acuuracy

F1-score

Precision

Recall

If the theory and experiment have addressed your concern, Hopefully, it will give you new confidence in our work and reconsider the contribution of our paper.

Thank you for your response. I would like to clarify that our method is completely different from FEDformer and TFDNet and we also mentioned it in our related work. Here we will show you the details of the difference from motivation, method, code implementation.

Motivation

1.FEDformer: tried to use FFT/Wavelet in accelerating multi-head attention to modeling temporal dependence, it shows that we use a small set of frequency components that could low-rank approximate KQV attention matrix.

2.TFDNet: didn't use FFT for decomposition, it used moving average pooling to decompose the feature in seasons and trends. After decomposing the two features, they use STFT (an FFT method with multi-scale sliding windows), which will contain both time and frequency information spitting the features into 2d feature maps so that it could model seasonal features and trend features more carefully and locally.

3.FEDNet: our method directly uses FFT for precomputing and getting top-k high amplitude frequency component positions in advance across different domains for decomposition, it is ensured with Wold’s theorem and our work verified the robustness and stability in domain generalization.

Method

1.FEDformer uses the continuing first 32/64 position i.e. [0,1,2..until..32...64] of frequency components (they are all low ranges and didn't consider the amplitude of components, the difference shows in appendix Figure 12(a)(b)) or random index frequency component for approximate KQV self-attention in every attention layer, its complexity will be with FFT $O(N \times d \log d)$, where $N$ is the number of attention layers and $d$ is the dimension after getting low-rank composition.

2.TFDNet uses moving average for decomposition and the STFT they used in FFN encoders its complexity will be $O(N \times M \times d \log d)$, where $N$ is the number of FFN layers and $d$ is the dimension, $M$ is the number of sliding windows.

3.FEDNet take the high average amplitude into consideration and get the top-k energy components as stable frequency components across the domains and we only calculate it once before we feed them into the VAE or Patch encoders. Our complexity is completely $O(d \log d)$, $d$ is origin time series length $L$. We also use wavelets for decomposition to check the importance of the orthogonality frequency decomposition function, we test both orthogonal (db2, coif1) and non-orthogonal (rbio1.3) function in C.5 INTRINSIC RELATIONSHIP STUDY OF ORTHOGONAL FREQUENCY DECOMPOSITION, whose goal is different from FEDformer and FEDformer only tests the Legendre orthogonal function.

Code implementations

1. FEDformer

def get_frequency_modes(seq_len, modes=64, mode_select_method='random'):
    modes = min(modes, seq_len//2)
    if mode_select_method == 'random':
        index = list(range(0, seq_len // 2))
        np.random.shuffle(index)
        index = index[:modes]
    else:
        index = list(range(0, modes))
    index.sort()
    return index

class FourierBlock(nn.Module):
    def __init__(self, in_channels, out_channels, seq_len, modes=0, mode_select_method='random'):
        super(FourierBlock, self).__init__()
        self.index = get_frequency_modes(seq_len, modes=modes, mode_select_method=mode_select_method)
        self.scale = (1 / (in_channels * out_channels))
        self.weights1 = nn.Parameter(
            self.scale * torch.rand(8, in_channels // 8, out_channels // 8, len(self.index), dtype=torch.cfloat))
    def compl_mul1d(self, input, weights):
        return torch.einsum("bhi,hio->bho", input, weights)

    def forward(self, q, k, v, mask):
        B, L, H, E = q.shape
        x = q.permute(0, 2, 3, 1)
        x_ft = torch.fft.rfft(x, dim=-1)
        out_ft = torch.zeros(B, H, E, L // 2 + 1, device=x.device, dtype=torch.cfloat)
        for wi, i in enumerate(self.index):
            out_ft[:, :, :, wi] = self.compl_mul1d(x_ft[:, :, :, i], self.weights1[:, :, :, wi])
        x = torch.fft.irfft(out_ft, n=x.size(-1))
        return (x, None)

2. TFDNet

class moving_avg(nn.Module):
    def __init__(self, kernel_size, stride):
        super(moving_avg, self).__init__()
        self.kernel_size = kernel_size
        self.avg = nn.AvgPool1d(kernel_size = kernel_size, stride = stride, padding = 0)

    def forward(self, x):
        front = x[:, 0:1, :].repeat(1, (self.kernel_size - 1) // 2, 1)
        end = x[:, -1:, :].repeat(1, (self.kernel_size - 1) // 2, 1)
        x = torch.cat([front, x, end], dim = 1)
        x = self.avg(x.permute(0, 2, 1))
        x = x.permute(0, 2, 1)
        return x


class series_decomp(nn.Module):
    def __init__(self, kernel_size):
        super(series_decomp, self).__init__()
        self.moving_avg = moving_avg(kernel_size, stride = 1)

    def forward(self, x):
        moving_mean = self.moving_avg(x)
        res = x - moving_mean
        return res, moving_mean

3. FEDNet

class Exp_Classification(Exp_Basic):
    def __init__(self, args):
        super(Exp_Classification, self).__init__(args)

    def _get_mask_spectrum(self, freq_type):
        """
        get shared frequency spectrums
        """
        train_data, train_loader = self._get_data(flag='TRAIN')
        amps = 0.0
        for data in train_loader:
            lookback_window = data[0]
            B, L, C = lookback_window.shape
            # mask
            # lookback_window = self.precise_random_mask_batch(lookback_window, self.args.mask_rate)
            frequency_feature = None
            if freq_type == "fft":
                frequency_feature = torch.fft.rfft(lookback_window, dim=1)
            elif freq_type in ['db2', 'sym2', 'coif1', 'bior1.3', 'rbio1.3']:
                wavelet = pywt.Wavelet(freq_type)
                # print("ortho=", wavelet.orthogonal)
                lookback_window = lookback_window.permute(0,2,1)
                device = lookback_window.device
                X = lookback_window.numpy()
                cA, cD = pywt.dwt(X, wavelet)
                frequency_feature = np.concatenate((cA, cD), axis=2).transpose((0,2,1)) # B D C
                frequency_feature = torch.from_numpy(frequency_feature).to(device)

            assert frequency_feature != None
            amps += abs(frequency_feature).mean(dim=0).mean(dim=1)
        mask_spectrum = amps.topk(int(amps.shape[0]*self.args.alpha)).indices
        print("mask_spectrum:", mask_spectrum)
        return mask_spectrum # as the spectrums of time-invariant component

class FrequencyFilter(nn.Module):
    """
    Fourier Filter: to time-variant and time-invariant term
    """
    def __init__(self, mask_spectrum, freq_type):
        super(FrequencyFilter, self).__init__()
        self.mask_spectrum = mask_spectrum
        self.freq_type = freq_type
        
    def forward(self, x):
        B,L,C = x.shape
        if self.freq_type == "fft":
            xf = torch.fft.rfft(x, dim=1)
            mask = torch.ones_like(xf)
            mask[:, self.mask_spectrum, :] = 0
            x_sto = torch.fft.irfft(xf*mask, dim=1)  
        elif self.freq_type in ['db2', 'sym2', 'coif1', 'bior1.3', 'rbio1.3']:
            wavelet = pywt.Wavelet(self.freq_type)
            device = x.device
            data = x.permute(0,2,1).cpu().numpy()
            cA, cD = pywt.dwt(data, wavelet)
            Dims = cA.shape[-1]
            frequency_feature = np.concatenate((cA, cD), axis=2)
            mask = np.ones_like(frequency_feature)
            mask[:, :, self.mask_spectrum] = 0
            frequency_feature = frequency_feature * mask
            cA_sto, cD_sto = frequency_feature[:, :, :Dims], frequency_feature[:, :, Dims:]
            x_sto = pywt.idwt(cA_sto, cD_sto, wavelet)
            x_sto = torch.from_numpy(x_sto).to(device).permute(0, 2, 1)
        x_det = x - x_sto
        return x_sto, x_det

challenge in time series ood(W1): We apologize for the confusion in understanding the article due to the lack of clarity in our picture, and we have revised the figure and the definition of temporal distribution shift in the article. Essentially, the temporal distribution shift depicted in our Figure 1 refers to the fact that the temporal distribution will change due to temporal non-stationarity at different time periods within the domain, leading to a shift in the marginal probabilities, and thus a shift in the joint probability distribution.

In this case, our artificial division of the domain is not ideal, and we need to solve the problem of how to achieve out-of-distribution generalization under the condition that the data within the source domain does not completely follow its domain distribution.

the role of frequency filter(W2,W4):

1. Thank you for your interest in the principles behind our frequency domain decomposition mechanism! Specifically, the advantage od frequency orthogonality is the key reason why we can solve the timing OOD problem. Unlike the time-dependence of different time points in the time domain, the frequency domain view is orthogonal between different frequency components, which allows the effects of these two offsets to be independent of each other, which inspires us to find stable frequency components that are less affected by the offset of the temporal distribution, and thus obtain pure domain-invariant features. At the same time, we retain the random component to prevent information loss as an auxiliary classification feature to increase the generalization space.

2. We derive the relationship between domain generalization theory and frequency domain components from Proposition 5.2 (Frequency Perspective Risk Bound on Unseen Time series Domain). The proof of the relationship between the domain generalization theory and the frequency domain components is given in the appendix of the paper. And we verify that using $x_{det}$ features instead of the original features further enhances the effectiveness of the IRM method. For more detailed descriptions, see Reviewer dy8u Frequency Composition(W4,W5,W6,W11) and Reviewer b2Zv motivation(W1)

frequency filter process(W3): We are very sorry for misleading you about the model figure due to our inappropriate expression, here we have modified the formulae and pictures in the text to be consistent with our actual code implementation version, the decomposition process will only be performed once fft to extract the frequencies of topk, and then set the amplitude of these frequencies to 0 (i.e., mask operation), and then we get $x_{sto}$ through the ifft, and then we can use $x - x_{sto}$ to get $x_{det}$. Then we can compute $x_{det}$ using $x - x_{sto}$, the original version is to show you that the process is equivalent.

limitations (W5): We analyzed some of the flawed situations of our method.

1. We visualize the spectrogram of FEDNet before and after masking, as in Figure 12 (c)(d) in the paper.It can be seen that, through masking, the diversity between domains is more pronounced, while many redundant frequencies are eliminated.But when there are some outliers in the frequencies, this definitely affects the training process of the model on domain invariant features.

2. We analyzed the OOD generalization for the classification of EEG signals with ultra-long sequences (L=3000), and found that the spectrograms produced by ultra-long sequences are more prone to unknown frequency distributions, as shown in Figure 8, and when we use domain 2 as the target domain, our model is unable to detect such unique frequency variations from the training domain, and it is possible that these frequencies also belong to invariant features, this example shows us the drawbacks that our method may face, that is, if we completely rely on the training domain to extract the invariant features learned from stable frequencies is not sufficient there may be bias, especially in the case of very long sequences will be lost a lot of frequency signals, which is why we need to retain and design the stochastic encoder module.

3. We add an interesting experiment of missing value with time series, which suggests a valuable insight for our approach. That is when we extract insufficient domain invariant features, it is crucial to supplement more time-stochastic information and improve the generalisation space. Meanwhile, this also suggests more interesting directions and improvement questions for the future, on the one hand, how to deal with biased or insufficient domain information, and on the other hand, how to do Time Series Imputation with OOD. For more analysis, see the Reviewer b2Zv time-stochastic study (W4) and see the C.3. TIME-STOCHASTIC FEATURE STUDY in the appendix.

Thank you very much for your timely reply, we understand your need for experimental performance, as the training of multiple domains is relatively time consuming, the newly added dataset is relatively large, in order to supplement the new experiments and theories we have not specifically adjusted the various parameters of the model, our next step is to fine tune the model's hidden_size, alpha hyperparameters, num_layers and so on.

We promise to continue to increase our experimental efforts to address your concern, since we still have one week. We will try our best to improve the performance to address your concern, as soon as we have new experimental results we will give you immediate feedback here

Thanks again for your advice to improve our work.

Thank you for your response. I have reviewed the revised paper, and a new concern has arisen. From additional experiments, it appears that the proposed solution does not have SOTA performance across all datasets. This raises doubts about whether the proposed solution is indispensable from an experimental perspective. I will maintain my score.

By the way, we have added a lot of theoretical and experimental content in this new version, including the role of the two components, theoretical generalization upper bounds and frequency domain isolation levels, etc. I hope these interesting experimental phenomena have given you more interest, and luckily, we still have a week of discussion and wishing you a Merry Christmas in advance!

motivation(W1): We are very sorry that our description of the temporal distribution shift in figure 1 may have caused you a misunderstanding. I'd like to first clarify that Temporal distribution shift is essentially a property of the non-stationary time series over time, and is not caused by a sliding window. More specifically, the temporal distribution shift is a change in the marginal probabilities over time due to the non-stationarity of the time series, and we have revised the description of temporal distribution shift in Figure 1 and the definition of temporal distribution shift in the paper. We have modified the description of temporal distribution shift in figure 1 and reworked the definition of temporal distribution shift in the paper.

In order to give you a better understanding of the background and motivation of our paper, I'll answer your questions in turn from the multiple perspectives you mentioned.

1. The role of sliding windows in mainstream time series classification tasks. A sliding window is also necessary for temporal classification tasks. In fact the original samples contain many invalid time segments[1], these segments can also affect feature learning and distribution if they are not eliminated, and the samples are not always keep the same class states at different times[2], you can see this in detail in the dataset's introduction and in the code of the open-source dataset we reproduced. Increasing the length will not alleviate this problem, but make it more internally complex and the time complexity of inference will increase because it contains more noise. It is common to separate the data into different time segments after cleaning them using a suitable length, which does not eliminate temporal non-stationarity, but it is a technique similar to data augmentation and makes it easier to capture localised patterns, e.g. some segments features of the same period may follow the same distribution.

2. The benefit for modeling distribution shift in frequency.

   • Frequency orthogonality to eliminate time dependence. The most fundamental reason for using the frequency domain is that changes in the distribution of a time series in the time domain are difficult to model with higher-order moment estimation, and the dependence between times is not conducive to our analysing the sources of distributional shifts, whereas all frequency components in the frequency domain space satisfy orthogonality and do not interfere with each other. Therefore we can assume that both distributions affect each frequency independently, and we can derive expressions for the frequency domain marginal distributions as shown in Corollary 1 (Frequency Marginal Probability Distribution of Time Series Data) in the appendix of our paper. This facilitates us to analyse the relationship between different distributional shifts and individual frequencies. We verified the relationship between the distribution shift and the frequency domain components, and found that we can use the orthogonality of the frequency domain to separate the two components and weaken the effect of the temporal distribution shift to purify and obtain more robust domain invariant features.
   • Reducing redundant frequency components is beneficial for lowering the maximum distance between domains, thus lowering the generalisation upper bound. We derive Proposition 5.2 (Frequency Perspective Risk Bound on Unseen Time series Domain) by combining the theory of PAC-Bayesian domain adaptation[1] with the form of the marginal probability distribution in the time series frequency domain.And we have theoretically and experimentally verified the optimal frequency $\alpha$ ratio ranges 0.2-0.4, the details refer to Reviewer dy8u empirical domain divergence study with $\alpha$ isolated level(W10).

diverse dataset(W3)

1. The ERM method also performs well, on the one hand, because the signal variations in the human activity dataset are easy to capture, on the other hand, the goodness of the generalization space usually depends on the gap between the training domains and the test domains, if a space consisting of multiple training domains is diverse enough, then the direct gaps between it and the test set are relatively small, at this time it is normal to get a good prediction result with ERM, as we The UCIHAR used is less heterogeneous, so it works well on multiple domains, as mentioned in the diversify[1] paper. In addition, we can also focus on the fact that the ERM method generally has the largest fluctuation of results under different domain tests, which is formally the problem we want to solve in OOD.

2.To address your concern, we added more diverse datasets for the experiments, we continue to reproduce the DSADS, PAMAP two datasets with more heterogeneity used by diversify, as well as the EMG (electromyogram) signal data, and the EEG(Electroencephalogram) signal dataset with the ultra-long sequence of L=3000 as the baseline, and supplemented all the experimental results. It can be seen that on these datasets the ERM method performs overall worse than many ood methods, especially on the EMG and DSADS datasets.

3. By the way, empirically we also agree with this point you mentioned, as you can see that many OOD related works[2] have not been compared with strong baseline models, and it would be valuable for the development of the whole field for us to make this point in our paper.

Regarding some issues with the performance of the added experiments, the training is more time-consuming due to the somewhat large size of these multi-domain datasets. We are doing our best to solve it, because the previous 2 weeks added more content and did not deliberately adjust the relevant parameters, fortunately, we currently have a week, we are continuing to add.

[1] Lu W, Wang J, Sun X, et al. Out-of-distribution representation learning for time series classification[J]. ICLR,2023.

[2] Gagnon-Audet J C, Ahuja K, Darvishi-Bayazi M J, et al. WOODS: Benchmarks for out-of-distribution generalization in time series[J]. arXiv preprint arXiv:2203.09978, 2022.

loss study(W2)

1. cross vs contrastive loss. Thank you for pointing out the imperfections of the experiments, we added performance ablation experiments and domain invariant feature $\mathcal{A}$-distance evaluation experiments for all datasets for both constrained losses.

We added the results of cross label loss and contrastive loss for all the datasets used and updated the results of the ablation experiments, and we can see that the use of contrastive learning effectively improves the performance of the model.The UCIHAR, UniMiB-SHAR, and Opportunity datasets do not have a validation dataset set up in the implementation of the original paper. We have directly followed the original in order to maintain the accuracy of the reproduction, and there may be some strong overfitting problems. The other datasets all use a 20% validation set, and we can very clearly find that the performance is more robust using contrastive learning.

In addition, we found that using contrastive learning loss can further reduce the $\mathcal{A}$-distance. This suggests that, compared to standard cross-label loss, contrastive learning can produce more robust invariant features. This is likely because contrastive learning inherently possesses properties similar to DRO (distributionally robust optimization) [1]. A more detailed explanation can be found in the Invirant Feature Study in the paper and the responses in Reviewer dy8u time-deterministic with variational inference(W8).

2. hyper-parameters $\lambda_{det}$ and $\lambda_{sto}$ study. We didn't deliberately adjust these parameters, we used the default 1.0 for all datasets, to further address your concern, we tried $\lambda_{det} = [0.1, 1.0, 10.0]$ and $\lambda_{sto} = [0.1, 1.0, 10.0]$ for some easy adjustments. We performed performance prediction for 9 sets of hyperparameter combinations using UCIHAR's domain-0 as the target domain. We found that the performance of our approach is robust and does not require very sensitive tuning of them.

[1] Wu J, Chen J, Wu J, et al. Understanding contrastive learning via distributionally robust optimization. NIPS 2023.

time-stochastic study (W4) Thank you for mentioning time-stochastic features! First I need to point out that we used d_model=512 and e_layers=2 in order to align with the default parameters of PatchTST's patch embedding for fair comparisons, and we didn't intentionally adjust them. What's more, we have analyzed the time-stochastic moudle at three levels (feature, data, domain) and found some interesting phenomena, we designed this module most of all to avoid the loss of information, and secondly as an auxiliary feature to increase the generalization space of the domain invariant features, our here design idea is similar to the idea mentioned in Figure 1 of the INSURE[1] .

1. Patch condition number (feature level). We analyzed the original patch matrix after dividing the Time-stochastic feature into patches.We used the condition number $\text{cond}(x_{patch}) =\sigma_{max} / {\sigma_{min}}$, where $\sigma_{max}$ and $\sigma_{min}$ represent the maximum and minimum singular values of $x_{patch}$ to evaluate this part of the features. We find that using non-overlapping patch matrices effectively reduces the condition number of patch matrices, which makes it easier to train his input patch embedding linear layer stably and can effectively reduce the number of patches and accelerate training. P donate the patch len in the table bellow:

2. Time series missing value study (data level). We tried an interesting experiment is to missing the original time series to different levels while keep $\alpha= 0.2/0.4$ unchanged, we want to destroy as many time points as possible, so that the invariant features and the main frequencies change drastically, thus making the invariant features problematic, which may be a more realistic kind of scenario, this also provides some valuable insight! We find that the use of $x_{sto}$ brings more gain when we have more severe data missing, i.e., the worse the invariant features are, which illustrates the necessity of the time-stochastic moudle design, when we face insufficient invariant features, and we need to avoid the loss of information to enhance the generalization ability.

[1] X. Yu, H. -H. Tseng, S. Yoo, H. Ling and Y. Lin, INSURE: An Information Theory iNspired diSentanglement and pURification modEl for Domain Generalization, in IEEE Transactions on Image Processing.

3. Considering temporal distribution shift is gainful for Long time series. To clear up your confusion, we specifically supplemented a case study with a long EEG sequence dataset(L=3000) to show you that very long length sequences considering this offset will work better than traditional domain generalization methods. Due to the very long training time of this dataset, here we have only run through the results for EEG domain 2 for test.

4. Our fundamental motivation and question. Although it may appear that we are addressing issues specific to time series, our goal is to argue a more fundamental problem: in real-world scenarios, the domains we artificially divide are not perfect, and the data within a domain does not strictly follow an ideal domain distribution. The key question is how to achieve out-of-distribution generalization under such conditions. One approach, similar to Adarnn[4] and Diversify[5], is to use ensemble weighting or adversarial training to redefine pseudo domain division for better feature learning of time segments. However, this method essentially equates to data augmentation, and the multi-stage training significantly increases the model's training time complexity in our experiments. On the contrast, our motivation follows a completely different idea: how to leverage existing data itself priors to purify and extract ideal domain-invarint features when facing temporal shift int the domains since the periods could be infinity in realistic, limiting the maximum number of pseudo-domains is not always desirable.

[1] Reiss A, Stricker D. Introducing a new benchmarked dataset for activity monitoring[C]//2012 16th international symposium on wearable computers. IEEE, 2012.

[2] Supratak A, Dong H, Wu C, et al. DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG[J]. IEEE transactions on neural systems and rehabilitation engineering, 2017.

[3] Germain P, Habrard A, Laviolette F, et al. A new PAC-Bayesian perspective on domain adaptation. ICML 2015.

[4] Du Y, Wang J, Feng W, et al. Adarnn: Adaptive learning and forecasting of time series[C]//Proceedings of the 30th ACM international conference on information & knowledge management. 2021: 402-411.

[5] Lu W, Wang J, Sun X, et al. Out-of-distribution representation learning for time series classification[J]. ICLR,2023.

Thank you for your reply!

Theory and Related Work with Frequemcy

I would like to clarify to you that frequency domain information has been proved to be robust in many tasks, we have mentioned them in our related work, and I would like to add more details to you here.

1. For non-staninary time series forcasting, many methods try to use both frequency and time to extractor global and frequency features, these works have mentioned that frequency features are more stable and have global perspective. Some methods try to demonstrate that removing redundant frequencies by using the orthogonality of frequencies can reduce the variance[1][2]. (Statistic level)

2. For time series pre-training task, some works try to compare the learning domain training in both time and frequency domains, and it is proved that the frequency domain has a more global perspective, and obtains more stable features, which can be migrated to cross-dataset, cross-person, one-to-all task[3].(Representation level)

3. For time series domain adaptation, some works have mentioned that the frequency domain information is more stable than the time information distribution, and proposed a specific form of the frequency domain joint marginal distribution magnitude and phase[4], which is followed in our paper Lemma 4 (Distribution of Fourier Component),Corollary 1 (Frequency Marginal Probability Distribution of Time Series Data). (Distribution level)

4. For Domain Generalization, unlike the first three approaches that use frequency domain information at the level of representation learning, a common strategy of using the frequency domain is to utilize frequency domain information for data augmentation, and the effect of the magnitude and phase of the frequency domain components of a fine-grained study on data generation[5][6] is demonstrated in these approaches. (Data Generation level)

In contrast, our approach attempts to introduce frequency domain information to the representation learning level for Domain generalization, demonstrating the importance of extracting domain-invariant features from frequency domain information, and proving the proof of the relationship between domain divergence and frequency domain, you can see the details Proposition 5.2 (Frequency Perspective Risk Bound on Unseen Time series Domain) and Empirical Domain Divergence Study, In addition, Through these experiments we were able to verify that our experimental results were in agreement with the theory and determined that the optimal frequency domain to use is usually in the range of 0.2-0.4. (Fundamental Domain Generalization Error Bound level)

If you feel that these explanations can eliminate your confusion about frequency, we also hope that you will reconsider the contribution of our theory and methods or if you have better suggestions, we will continue to add relevant experiments and text to modify our submitted version, we still have the opportunity to modify the pdf today, and also have close to a week to improve our work.

Thank you again for your support of our work!

[1] Ye W, Deng S, Zou Q, et al. Frequency Adaptive Normalization For Non-stationary Time Series Forecasting[J]. arXiv preprint NIPS 2024.

[2] Liu Y, Li C, Wang J, et al. Koopa: Learning non-stationary time series dynamics with koopman predictors[J]. Advances in Neural Information Processing Systems NIPS 2023.

[3] Zhang X, Zhao Z, Tsiligkaridis T, et al. Self-supervised contrastive pre-training for time series via time-frequency consistency[J]. NIPS 2022.

[4] He H, Queen O, Koker T, et al. Domain adaptation for time series under feature and label shifts[C]//International Conference on Machine Learning. ICML 2023.

[5] Demirel B U, Holz C. Finding order in chaos: A novel data augmentation method for time series in contrastive learning[J]. NIPS 2023.

[6] Xu Q, Zhang R, Zhang Y, et al. A fourier-based framework for domain generalization[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. CVPR 2021.

writing suggestions(W1, W2): Thank you very much for recognizing our work! Your understanding is right to our idea, we have revised the introduction of frequency domain decomposition, the explanation of propotion. We modifiedthe definition of temporal distribution shift to be more clear. Essentially, the root of this shift is the non-stationarity of the series, and we explicitly gave the concept of periods, and we considered that data belonging to the same period inside the domain will follow the same data distribution, while different periods will be different from each other lead to changes in the joint probability distribution due to changes on their marginal probability distribution. We also add the theory related to domain generalization in the appendix A.4 THE RISK FOR DOMAIN GENERALIZATION OF TIME SERIES FROM FREQUENCY VIEW, as well as more experimental insights of $x_{det}$ and $x_{sto}$ components, we hope you are interested in!

shift on time-stochastic(Q1): Generally, the time-stochastic part is indeed affected by the domain shift. With the orthogonality of frequencies, both shifts affect each frequency component independently, but our more important goal is still to purify the invariant features that are not affected by time-shift, and in our experiments, we can also see that the random component module plays more of an auxiliary role and prevents the loss of information.

Anyway, there are some works [1] also proved that the generalization space of domain-invariant features can be increased to some extent by using domain-specific features that are beneficial for classification. Meanwhile, for a more detailed study of the role of time-stochastic component, you could see the TIME-STOCHASTIC FEATURE STUDY in the appendix of the paper or the answer to Reviewer b2Zv time-stochastic study (W4).

[1] X. Yu, H. -H. Tseng, S. Yoo, H. Ling and Y. Lin, INSURE: An Information Theory iNspired diSentanglement and pURification modEl for Domain Generalization, in IEEE Transactions on Image Processing.

time-stochastic study(W12, W9) The main purpose of the design of time-stochastic is to prevent the loss of information, and another aspect is to serve as auxiliary classification information to prevent the invariant features from over-dependence on the source domain, and increase the space of domain generalisation,, meanwhile, we found that when the time sequential data are missing to interfere with the invariant features, the supplemental time-stochastic information can significantly improve the gain, we We designed three different levels of experiments (faeture, data, domain) to illustrate the role of time-stochastic, detailed information can be found in the appendix C.3 TIME-STOCHASTIC FEATURE STUDY of the paper as well as the response of Reviewer b2Zv time-stochastic study(W4).

We have revised Proposition 5.1 in the paper and combined it with the experimental findings to clarify the roles of the two components, and we have revised the proof section to incorporate the new definitions, which hopefully clears up any confusion you may have.

time-deterministic with variational inference(W8): Thank you very much for mentioning this insightful issue! We analyzed the invariant features in different methods using the classical domain generalization metric $\mathcal{A}$-distance[1]. It can be approximated as $d = 2(1 - 2\sigma_{\mathcal{A}})$, where $\sigma_{\mathcal{A}}$ is the risk of a binary classifier distinguishing features between source and target domains. $d \in [0, 2]$ and the smaller the indicator, the more invariant features are. We measured model invariance features for multiple domains of the EMG dataset. This experiment has been added to the paper.

We found that our proposed variational inference approach can learn stronger invariant features compared to IRM. Theoretically, classical IRM often faces parameter overload failures when addressing domain generalization problems. When the number of model parameters is relatively large, the regularization of IRM tends to cause the model to overfit[2][3]. Some studies have also demonstrated that utilizing variational inference with decoupling enhancements is effective for domain generalization [4].

In addition, we found that using contrastive learning loss can further reduce the $\mathcal{A}$-distance. This suggests that, compared to standard cross-label loss, contrastive learning can produce more robust invariant features. This is likely because contrastive learning inherently possesses properties similar to DRO (distributionally robust optimization) [5].

We also supplemented the analysis by comparing the $\mathcal{A}$-distance before and after applying deterministic IRM gains using different IRM methods. The results showed that incorporating deterministic components can yield purer invariant features.

[1] Ben-David S, Blitzer J, Crammer K, et al. Analysis of representations for domain adaptation[J]. NIPS 2006.

[2] Yong Lin, Hanze Dong, Hao Wang, Tong Zhang, Bayesian Invariant Risk Minimization, CVPR 2022.

[3] Xiao Zhou, Yong Lin, Weizhong Zhang, Tong Zhang, Sparse Invariant Risk Minimization, ICML 2022.

[4] Towards Principled Disentanglement for Domain Generalization, CVPR 2022.

[5] Wu J, Chen J, Wu J, et al. Understanding contrastive learning via distributionally robust optimization. NIPS 2023.

Writing problem(W1,W6,W7,W13,W14): Thank you very much for your very useful comments on our thesis writing! This is our first time submit to ICLR, and we ask for your forgiveness for the problems caused by the citation of the first draft. We have reworked all the issues you mentioned and changed to suitable citation:

• We have revised the description of IRM in the introduction, specifically on line 47.
• We elaborated on the key ideas of the three types of domain generalization methods, in lines 112-118.
• We updated all citation formats throughout the text for your convenience in reading the details.
• We resolved the issue of the repeated $\lambda$ hyperparameter, in line 230 and line 343, Eq. 16.

Definition of Temporal Distribution Shift(W2):We sincerely apologize for any misunderstanding caused by our writing. We have revised the definitions and the challenge in figure 1 to be more clear, the mentioned temporal distribution shift is consistent with your understanding. Essentially, it stems from the non-stationarity of time series in the real world. In the new definition, we explicitly introduce the concept of period to illustrate that time-series data across different time periods exhibit marginal probability distribution changes. This results in segments within the same domain not being independently and identically distributed. Therefore, the domains we define in practice are not entirely ideal.

This is also a key reason why out-of-distribution (OOD) generalization in time series differs from that in static image datasets. We believe this phenomenon is not limited to time series. Fundamentally, we aim to argue the broader issue of how to improve a model's generalization ability when the data within a domain does not fully follow a single distribution. This perspective aligns more closely with real-world scenarios, as the domains we define manually often deviate from perfection.

Definition of Frequency Distribution Shift(W3): The relationship between time series data and frequency components can often be rigorously expressed through Fourier transform as an integral summation $\mathcal{F}\{f(t)\} = F(\omega) = \int_{-\infty}^{\infty} f(t) e^{-i \omega t},dt$. In fact, the intent here is to review this time series distribution problem from frequency perspective. We provide the basic formulation of the Fourier frequency domain distribution Distribution of Fourier Component and Frequency Marginal Probability Distribution of Time Series Data in Appendix A.2 PRELIMINARY. Since the concept of time series distribution cannot be simply defined, if we consider the most fundamental non-stationarity property, such as changes in mean $\mu_{t}$ and variance $\sigma_{t}$.These changes in $x_{t} = \mu_{t} + \sigma_{t} \times z$, where $z$ could be an stationary componets could reflect to variations in the amplitude and phase of $n$ independent frequencies. Exploring the magnitude of these changes impact on the frequency domain components is precisely the core focus of our paper's analysis and modeling efforts.

Frequency Composition(W4,W5,W6,W11) : Sorry for the misunderstanding of our decomposition method due to my effectively limited English writing skills. Here we use Wold's theorem, which is in fact based on the basic assumption of the temporal shift problem, i.e., the temporal data belonging to the same time period within the domain will follow the same distribution, it fits the local weak-sense stationarity. The main benefit of weak-sense stationarity is that any time series can be put into the context Hilbert Space. It means any time series can be decomposed by a set of orthogonal increments in the space. Bochner's theorem ensures that there exists a group of Fourier-type orthogonal complex exponential function $\{e^{-2 \pi i \xi t}\}$ componets to generate $x_t$ in Hilbert Space. We have modified the text to make it more precise. Due to the orthogonality of frequencies and the commom shared space within different time series periods, it inspired us to explore whether the deterministic and stochastic components are dominated by specific frequencies, even in different periods and domains. This idea was verified in our experiments on the coefficient of variation for frequency domain decomposition. These two components are dominated by two different sets of frequencies, so that we could study and address the two different distribution seperately to get a purified invariant features. We provide an intuitive example for you: $x_t = \alpha sin(w_0 t) + \beta sin(w_t t) + \epsilon_t$ where $\alpha, \beta, w_0$ do not change with time, $\epsilon_t$ is white noise and $w_t$ varies with peroids. $\alpha sin(w_0t)$ can be regard as time-deterministic components while $\beta sin(w_t t)$ can be seen as time-stochastic components. There are already some related works[1][2] that attempts to use this decomposition idea to solve non-stationary problems.