Enriching Time Series Representation: Integrating a Noise-Resilient Sampling Strategy with an Efficient Encoder Architecture

Thank you for dedicating your time to reviewing our paper and providing detailed, informative comments. Besides the responses, we have uploaded our revised manuscript and appendix, with all the revisions and additional details made based on your constructive feedback. Below are our responses to your specific questions:

1. Regarding forecasting baselines

Thank you for sharing the interesting studies, and we have completed the reading of these remarkably interesting works. We cite these references in our revised manuscript in the literature reviews section. We are working to add these methods into our experiments. However, due to the short rebuttal time and extensive experiments, we can not guarantee that such comparisons appear within the time period. Nevertheless, we will do our best to add the references to the experiments in the next version of our manuscript.

2. The diversity of forecasting datasets seems limited, focusing only on the electricity domain.

About the use of 4 datasets for forecasting task, there are several reasons for us to do so:

• They are widely used to assess performance of time series frameworks when predicting long future sequences.
• We incorporate these datasets in line with earlier research [1,2], and we follow the specific configurations outlined in those studies. This procedure is taken to minimize potential biases arising from varying experimental conditions, ensuring a fair comparison of our method with existing research.
• Three of four datasets come from the same repository (ETT datasets), and one from UCI repository. These 2 repositories have the distinguished target variables: For ETT datasets, the target variable is Oil Temperature (OT), while the variable of Electricity dataset is electrical consumption in kWh. This diversity enhances the overall Forecasting section, rendering it more comprehensive and universally applicable.

3. How to quantify the performance improvement with respect to the noise levels in the original datasets? For example, do higher noise levels in the original datasets correspond to larger performance improvement after adopting the proposed noise-resilient approach?

We thank the Reviewer for this insightful comment. Following this suggestion, we plan to further design and conduct an experiment investigating the relationship between noise levels and models’ performance. We are working to add experimental result in the next version of our manuscript. However, due to the short rebuttal time and extensive experiments, we can not guarantee that such comparisons appear within the time period.

4. Clarification on the choice of Discrete Wavelet Transform (DWT)

Our initial preference for the Discrete Wavelet Transform (DWT)-based low-pass filter is grounded in several considerations outlined below:

• DWT, as opposed to another commonly used method in frequency analysis, FFT, offers a distinct advantage that aligns with our objectives in the low-pass filter. While FFT provides a decomposition without temporal information, DWT deconstructs series into multiple frequency sub-bands with reduced time resolutions. Opting for DWT grants us more flexibility in determining which frequency bands to diminish at specific time ranges, allowing targeted reduction without completely eliminating entire frequency spectra from the series of interest.
• This filter operates as an independent module and requires no training parameters, making it decoupled from other modules in the CoInception framework. This modular design opens space for potential replacements that may be better suited to specific datasets or tasks.

It's crucial to underscore that our primary contribution lies not in the low-pass filter itself but in the objective of generating noise-resilient representations. Consequently, our current selection of the DWT low-pass filter may not be universally optimal for all datasets or tasks. For particular datasets, we believe spending extra effort to carefully choose the suitable low-pass filter could enhance the CoInception pipeline’s performance even further.

We appreciate your time and effort spent on reviewing our manuscript. We are glad to receive your acknowledgment of our encoder design and our efforts in conducting experiments. Besides the responses, we have uploaded our revised manuscript and appendix, with all the revisions and additional details made based on your constructive feedback. We address and clarify these concerns point-by-point below:

1. Regarding high frequency noise

Thank you for this insightful comment. We are sorry that our sentences gave a false impression that we assume high frequency in noise. We agree with you about the two references and we remove them since they potentially lead to a misunderstanding of our intended idea. We hope to discuss this in detail and clarify our work better with you.

While we focus on addressing high-frequency noise, we do not necessarily assume that noise is inherently high frequency. However, we have motivations to consider this setting, which is the observation that high-frequency noisy components often correspond to short-term variations or bursts that may not significantly contribute to the overall structure of the time series. The contextual information derived from these short-term fluctuations can be misleading and may not accurately represent the underlying series of interest. Therefore, our objective is to make the CoInception framework insensitive to these high-frequency noise. By doing so, we aim to ensure that the output representations are more robust to variations and disturbances in the input. In addition, this approach aligns well with various studies [1,2,3,4] emphasizing the utilization of disentangled components of raw series, such as seasonality or trends, which exhibit persistence in a long-term context.

For the above reasons, we argue that the study of this setting is useful and beneficial in practice, and this fact is proven by our extensive experiments. In the revision, we clearly explain why we are specifically addressing high-frequency noise and update our references accordingly.

2. Regarding our writing

Thank you for this straight and constructive feedback. As mentioned, we have refined our writing in this revision (taking into account all three examples you mentioned), carefully with our choices of words to avoid any potential misunderstandings and concerns. If you see any other examples, please let us know and we will work with you promptly to make sure our manuscript is clear and concise.

3. What are the characteristics of datasets that you expect your proposed method to work better on? For example, do they tend to have noise in distinct frequencies or require longer sequence dependence?

We thank you for an insightful question on the further analysis of our methods. While our primary focus is on making the CoInception framework robust to high-frequency noise, it's important to note that we don't presume superior performance on specific datasets. This intentional lack of expectation ensures the versatility of our design, allowing it to be applicable across a broad spectrum of datasets and tasks. Specifically, when defining the 'high-frequency' spectrum in our framework, we don't restrict it to a fixed range of frequency bands. Instead, it is calculated based on the characteristics of the datasets to which the CoInception framework is applied. This adaptability feature allows the CoInception network to flexibly adjust to various datasets and tasks. It's worth mentioning another potential concern, constructively mentioned by Reviewer CkvB: the risk of over-smoothing effect in CoInception when high-frequency components are informative for representing the original series. We efficiently address it with our Triplet loss regularization term by prioritizing alignment between two raw overlapped series. Further details on this matter can be found in our original response to the first question posed by Reviewer CkvB.

With these responses, we hope that we have addressed all of your current concerns. Should you need any other clarification, please leave a comment and we are happy to further discuss with you. Given your appreciation for the CoInception encoder design and our comprehensive experiments, we sincerely hope that once all your concerns are resolved, you may reconsider the paper, potentially increasing your score and supporting our approach. Thank you for your time and consideration.

Thank you for your time and the positive evaluations of our paper. We also express gratitude for your constructive comments and suggestions. Besides the responses, we have uploaded our revised manuscript and appendix, with all the revisions and additional details made based on your constructive feedback. We hope that our responses below sufficiently address all of your questions:

1. An additional ablation study on the encoder architecture

We thank you for these great constructive comments. We find this experiment can be directly addressed, given that it is related to our framework and worth investigating. We plan to update these results together with our revised manuscript in the next few days.

2. The noise reduction is demonstrated in Figure 2, but it is hard to tell what to expect from the figure.

Thank you for pointing out this. We have made our motivation clearer with Figure 2, including quantitative correlation calculation measured by cosine similarity to show the proximity between the representations of original time series and the perturbed ones.

3. The noise reduction effect is mainly attributes to which of the three proposed compoenents?

The initiation of the noise reduction effect occurs during the sampling stage for training data samples, where segments of both raw and disturbed series are selected. Subsequently, the actual realization of this effect is guided by our alignment loss and stabilized by the Triplet Loss regularization term through the training process.

Nevertheless, while the CoInception Encoder does not directly support the noise reduction effect, its contribution is also crucial in the whole pipeline. It balances efficiency and effectiveness, contributing significantly to the enhanced performance of CoInception, as demonstrated in Table 4 of the ablation experiment. Moreover, owing to the independent nature of these two modules, future studies might be benefited from different aspects of the CoInception pipeline, such as the reutilization of the CoInception architecture, without necessitating modifications or assumptions about other modules.

Once again, thank you for your positive evaluations of our paper. Please leave us with other responses if you have any further concerns. When all your concerns are resolved, we sincerely hope that you could consider increasing your score and support us.

Thank you for dedicating your time and effort to reviewing our paper. We appreciate your understanding of the work amount we invested in this study. At the same time, we are sorry that you found our claims being vague in this version. Here, we aim to address and clarify these concerns in our subsequent responses to specific questions and points you made. Besides the responses, we have uploaded our revised manuscript and appendix, with all the revisions and additional details made based on your constructive feedback. If further clarification is required, please feel free to leave a comment, and we would be delighted to engage in a discussion with you.

Regarding our position in time series modeling literature, we kindly direct you to our common response to all Reviewers, where we offer a concise summary of all the contributions presented in our work. Furthermore, we assure you that in the revised version, we will exercise our utmost care to mitigate any potential ambiguity or misunderstanding arising from the choice of words.

1. Regarding how to deal with noises in time series data

While we agree with you that the intentional use of Neural Networks is partially attributed to their ability to cope with noise, we want to emphasize that this effort may not be sufficient to effectively diminish the impact of noise, especially when our goal is to produce robust and resilient time series representations. We visually highlight this point through Figure 2 in the manuscript. Considering the visually negligible impact of noise on the original series in this figure, we expect the fundamental characteristics in the learned representation to remain intact. However, we acknowledge the inconsistent representations generated by an existing State of The Art framework when presented with raw and noise-added series. This experiment is one of our motivations for explicitly considering noise resilience as a crucial characteristic of learned representations in the design of the CoInception framework.

2. What "Frameworks" are the authors talking about?

In these specific sentences, our intention is to underscore that the presence of noise in raw series should not impact the representations learned for the true underlying series under investigation. The term "frameworks" encompasses generic models aimed at producing robust time series representations for various downstream tasks. To enhance clarity, we revise the sentences as follows, and will accordingly update them in the revised manuscript: "we devise a sampling strategy guided by the principle that the presence of noise in the series should not hinder the functionality of our framework. Ideally, it should generate consistent representations whether provided with noise-free or raw series, highlighting noise-resiliency characteristics"

3. Regarding the basic aspects of the model but are presented as technical contributions

We apologize if you have the impression that these basic aspects are the technical contributions of our paper. However, we think this is just a misunderstanding and we have revised our writing to make sure there would be no confusion. As you rightly noted, this Low Pass Filter is well-established and extensively studied over its long history. Instead, we highlight its advantageous applications in the context of time series representation learning, aligning well with our initial goal: ensuring noise resilience characteristics. Furthermore, our current implementation of the LPF in the sampling stage effectively isolates it from other modules within the CoInception framework. Thus, the LPF serves as a tool assisting in achieving the ultimate goal of learning noise resiliency, and it can be replaced by any filters or modules that function to reduce potential noise in the given raw series, while other CoInception modules remain intact.

Finally, we hope our answers address all of your questions. Given your acknowledgment of the effort invested in this study, we hope that once all your concerns are resolved, you may reconsider the paper, potentially increasing your score and supporting our approach. Thank you for your time and consideration.

We appreciate your precious comments and thank you for your recognition of our work on multiple aspects. We hope that our responses below answer all of your questions. Besides the responses, we have uploaded our revised manuscript and appendix, with all the revisions and additional details made based on your constructive feedback. Should you need any other clarification, please leave a comment and we are happy to discuss with you.

1. Regarding the novelty of this paper.

We agree that there are some alignments of our proposal with the previous work - TS2Vec, and understand your concern. However, we want to make further clarifications that those alignments are not overlapping with our contributions.

• First, our novel noise-resilience sampling strategy is driven by its distinguished motivations. Concurrently, we ensure context-invariant characteristics through a modified version of the technique introduced with TS2Vec. The rationale behind this modification is well-founded, addressing potential collapses in certain scenarios, with detailed explanations provided in our Appendix A.1.
• Second, it is important to note that the stacked Dilated Convolutional network itself is not a new concept and has been employed in prior works (e.g. [1]). We leverage and build upon this existing framework, as it aligns with our stated objective of achieving both robustness and efficiency.

2. Detailed analysis of the limitations and potential drawbacks of the proposed approach

Thank you for this suggestion. We hope to discuss this with you. We do rely on some assumptions as well as make some considerations upon designing the CoInception framework, which might be its limitations and potentially lead to drawbacks in some scenarios, listed below:

• Regarding the DWT-based sampling strategy, our focus on high-frequency noise implicitly confines the type of noise being targeted in this study. By removing the high-frequency components, the filter helps to smooth out the signal and eliminate rapid fluctuations caused by noise, while better revealing the underlying trends or slow-varying patterns in the time series. However, this strategy does not guarantee to create an ideal noise-free signal of the series.
• Regarding the encoder architecture, our current design, while making a positive step toward balancing efficiency and effectiveness, only partially attains this objective. Despite the automation of scaling dilation factors with the Inception idea, the challenge of optimizing the number of layers remains unanswered. This issue is inherent in the efficiency-effectiveness trade-off, indicating that we have not fully realized our goal. Presently constrained to a 3-layer encoder, we make no claim about the optimal number of layers to be used, recognizing the need for fine-tuning based on specific tasks or datasets.

Given these insights, we will certainly take these factors into account for future work. Due to space limitations, we plan to incorporate these reflections in a dedicated section titled "Discussion" within the Appendix of the modified manuscript.

3. Regarding comparison with other state-of-the-art methods

We thank you for your suggestion, and we cite more references in our revised manuscript in the literature reviews section. We are working to add these methods into our experiments. However, due to the short rebuttal time and extensive experiments, we can not guarantee that such comparisons appear within the time period. Nevertheless, we will do our best to add the empirical comparisons (including TimesNet) in the next version of our manuscript.

4. Regarding high-frequency noise and trade-offs in addressing this issue

We account for this scenario in the design of the CoInception framework and address it implicitly through the introduction of a Triplet loss regularization term, detailed in Section 2.3 of the revised manuscript. Specifically, if the vital information of the original series is predominantly in the high-frequency spectrum, solely aligning raw samples' representations with disturbed ones could result in substantial information loss. The Triplet loss regularization term mitigates this effect by giving more priority to the alignment between two raw overlapped samples, thus alleviating the loss of essential information. Empirical results presented in Table 4 highlight the detrimental impact of ablating this Triplet term.

Thank you so much for your time and your evaluations of our paper. We are glad that you think our empirical studies are comprehensive and provide thorough understanding. Besides the responses, we have uploaded our revised manuscript and appendix, with all the revisions and additional details made based on your constructive feedback. We hope to discuss about the originality and novelty of our work which we try to clarify in the following responses to specific questions and concerns:

1. Regarding comparison with other state-of-the-art methods

We express our gratitude to you for sharing the three interesting studies, and we have completed the reading of these remarkably interesting works. We cite these references in our revised manuscript in the literature reviews section. We are working to add these methods into our experiments. However, due to the short rebuttal time and extensive experiments, we can not guarantee that such comparisons appear within the time period. Nevertheless, we will do our best to add the references to the experiments in the next version of our manuscript.

2. Regarding our CoInception backbone model and originality

Our sampling strategy is intentionally decoupled from the encoder architecture, and there are several benefits of making them modular.

• For the sampling strategy, as you mentioned, we use an existing method - DWT low pass filter, hence it is not our main focus in this work. With the independent nature of this module, a different noise-canceling methodology, which is more suitable for particular tasks or datasets, can be installed in a plug-and-play manner, without affecting the function of the Encoder.
• For the Encoder architecture, with this modular design, it carries no assumption about the data as well as the task being dealt. This nature, together with its design in accordance with our intention of balancing out between effectiveness and efficiency, explain for its robustness in both three tasks and various datasets discussed in our Experiment section. In addition, owing to the modular nature, any future studies can refer to our work and be benefited from our Encoder architecture without tied assumptions.

That being said, inside each of these modules, there are strict connections between sub-components.

• Regarding the sampling strategy, it should not be viewed in isolation within our framework. Instead, it should be integrated with our objectives, as represented by the loss functions. In addition to temporal-wise and instance-wise loss functions, which align samples from raw and disturbed series, we introduce an additional term acting as a regularization factor. This serves to stabilize the training process and mitigate the over-smoothing effect. The combination of the sampling stage and loss functions guides the CoInception framework in learning noise-resilient representations.

• Concerning the Inception and Dilated Convolution architecture, their interaction ensures enhanced efficiency and effectiveness. The stacked Dilated Convolutional architecture has a notable weakness related to the selection of dilation factors. When these factors are too small, the parameter-efficient gain from dilation diminishes, and if they are too large, the framework overly focuses on a broad range of contextual information, potentially neglecting local details due to skipped spatial locations and limitations on the number of kernel filters used. To address this challenge, we incorporate the Inception idea into our Encoder. The Inception design naturally serves as a solution to automate the incorporation of various dilation factors in a single layer. By introducing different dilation factors of varying sizes into a single layer, our aim is to capture both local and global context without going too deep vertically into the number of network layers, while also reaching a large receptive field.

In conclusion, we hope that our responses have sufficiently addressed all of your questions. We appreciate your favorable evaluations of our paper. Once all your concerns have been resolved, we sincerely hope for your consideration in increasing your score and offering your support. Thank you again for your time and consideration.