Prompting-based Efficient Temporal Domain Generalization

We thank the reviewer for your positive, insightful and valuable comments and suggestions which are very crucial for improving the quality of our manuscript.

The paper's problem description promises a domain generalization solution, but the actual solution seems to focus only on concept drift within the same dataset, falling short of initial expectations.

This paper focuses on temporal domain generalization, addressing data distribution shifts over time. We followed the experimental setup of existing state-of-the-art temporal domain generalization methods [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019)]. And we cited the state-of-the-art method results in Table 1. We are not the first to propose temporal DG, and we followed the standard practice.

---

Clarification on the classification of different windows in a dataset as 'domains' when they represent data drift over time, suggesting a need for true domain generalizability and adaptability evaluation, as per existing literature on concept drift.

We followed the experimental setup of existing state-of-the-art temporal domain generalization methods [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019)], and we split the dataset following the SOTA temporal DG methods. We are not the first to propose temporal DG, we didn't propose a new way to split the dataset. We followed the standard practice.

---

Clarification on elements of Algorithm 1, including uniformity in data points per domain, the generation process of temporal prompts, differences between Lines 12 and 13, and the specific nature of Y in the context of the housing price prediction example.

The number of data points in each domain can vary. We do not enforce a uniform number of data points across all domains, as this depends on the dataset and the nature of the temporal segments being considered.

In Line 12 of Algorithm 1, we are referring to the input tokens (PS1, PS2, ..., PS(t−1)) for the prompt generation module. These tokens represent the domain-specific prompts from previous time periods. In contrast, Line 13 describes the generation of PT(t), which is the current temporal prompt. This prompt is produced by the module using the concatenation of previous domain-specific prompts as input. The key distinction here is between the input prompts from previous domains (Line 12) and the output prompt for the current domain (Line 13).

Y represents the label for the input data in the same domain. To illustrate with the housing price prediction example: if the features of a house in 2021 (X) correspond to a value of 200K, then Y (the label) would also be 200K. In other words, both the input (X) and output (Y) are derived from the same domain, ensuring that the model learns to predict within the context of each specific temporal domain.

---

Effectiveness of our Method in abrupt drift scenarios.

Our model is optimized for gradual temporal drifts. If abrupt drifts are present in the training data, it's likely our model can adapt to similar abrupt changes during testing. However, without exposure to abrupt drifts in training, the model may not effectively handle sudden changes in the test data.

We thank the reviewer for your positive, insightful and valuable comments and suggestions which are very crucial for improving the quality of our manuscript.

What's the mathematical formulation of the prompt?

Our work is inspired by the success of prompt-tuning in NLP tasks [Lester et al. (2021); Vu et al. (2021); Gu et al. (2021); Li & Liang (2021); Asai et al. (2022); Wang et al. (2023)]. As for the mathematical formulation of prompt, [Oymak et al. (2023)] developed a statistical foundation for gradient-driven prompt-tuning, examined its optimization and generalization behaviors, and investigated how it facilitates attending to context-relevant information. Additionally, they demonstrated that one-layer softmax-prompt-tuning is provably more effective than other methods, such as one-layer self-attention.

[Oymak et al. (2023)]: Oymak, S., Rawat, A.S., Soltanolkotabi, M. & Thrampoulidis, C.. On the Role of Attention in Prompt-tuning. Proceedings of the 40th International Conference on Machine Learning (ICML 2023).

---

Doubts raised about the need for using prompts, considering the claim that other methods can't generate time-sensitive prompts for temporal dynamics, despite the existence of various alternatives to learn temporal dynamics.

While alternative methods exist, our approach is well-supported and aligned with current research trends in the field. We have extended prompting from NLP to a lot more applications, which is truly valuable. Most previous prompting methods only demonstrated their efficacy in NLP tasks or NLP-related neural networks like GPT or CLIP. We proposed the first prompting-based temporal DG method, which is a generic model and is applied to various tasks. This shows great potential in using prompt learning for various applications other than NLP.

---

Typos and increasing readability.

Thank you for your feedback. We will address the noted issues by correcting typos, clarifying undefined notations, and revising our citation format to improve readability and presentation.

---

The logic behind training the backbone network on an aggregated dataset, especially for datasets with manually altered distributions across domains, which could complicate learning the decision boundary.

Through aggregating datasets, our network is designed to learn general/common features across the entire dataset, while domain-specific prompts target specific feature sets.

---

The ablation study can be further designed to remove the backbone model to verify if the backbone model is truly useful.

Sorry, We didn't understand the question. How would prompting work without a backbone model? We followed the standard practice of prompting [Lester et al. (2021); Vu et al. (2021); Gu et al. (2021)], and a pre-trained backbone model is required.

---

Clarification on whether both and are learnable parameters in the context of pre-pending inputs with prompts for each domain.

Prepending input with a prompt is a standard practice of prompting learning [Lester et al. (2021); Vu et al. (2021); Gu et al. (2021)]. In our model, the inputs are standard, non-learnable data from the dataset, similar to other classic models. Only the prompts are learnable parameters.

---

Missing baselines in Table 3.

Other baselines are not applicable on Time series datasets. We will clarify that in Camera Ready.

---

Confusion about the claim stating that: "Our paper presents a novel prompting-based approach to temporal domain generalization that does not require access to the target domain data", as having no access to the target domain data is a default rule of domain generalization.

Yes, we agree that no access to the target domain data is a default rule of domain generalization. We just want to emphasize it and highlight it, since it is the key difference between domain adaptation and domain generalization.

We thank the reviewer for your positive, insightful and valuable comments and suggestions which are very crucial for improving the quality of our manuscript.

Overclaim 1. How is the method "parameter-efficient and time-efficient" when it requires training a transformer with more parameters than existing methods like DRAIN?

We appreciate the opportunity to address the concerns regarding any perceived overclaims in our paper. We would like to assure you that our assertions are carefully calibrated to reflect the findings of our research. As shown in Table 3, our approach, despite involving the training of a transformer (Temporal Prompt Generator), is actually more parameter-efficient. In fact, there are structural differences in how each method processes weights. DRAIN uses hypernetworks to generate weights for the backbone networks, and the size of hypernetworks is significantly higher than the backbone network. Especially when the domain task is a time series application, DRAIN requires generating weights for transformers, which requires an extremely high number of parameters. While for our method, whatever the domain task is, the goal is to generate a short prompting vector, which is much more parameter-efficient. Another aspect to note, for each domain DRAIN generates a backbone network, namely if there are $n$ domains DRAIN requires $n$ backbone networks. While for our method, the backbone network is shared, only a short vector (at most with length 128) is required for each domain. In general, time and parameter efficiency are the most well-known characteristics of prompt-tuning methods.

---

Overclaim 2. How is the method time-efficient when there's no comparative analysis of training time and both pre-training and fine-tuning a transformer for temporal soft-prompt generation are known to be inefficient?

The assertion that our model lacks time-efficiency is addressed in Table 3, which presents a detailed comparison of training times between our method and existing approaches. This table clearly demonstrates that our model is not only efficient in terms of parameter usage, as previously discussed, but also in training time.

---

Unfounded. The authors claim that "Only a few methods studied temporal DG problem .... which are inefficient and complex to be applied to large datasets and large models," which is unfounded, and without any quantitative analysis for demonstrating this assumption.

The claim about the inefficiency and complexity of existing methods for temporal dynamic graph (DG) problems is backed by the data presented in Table 3. This table specifically illustrates how the number of parameters in the DRAIN method increases exponentially with time series datasets. Given that DRAIN relies on MLPs for encoding and decoding weights, this exponential increase in parameters makes it impractical for use with larger models. Moreover, DRAIN requires generating weights for the backbone networks using hypernetworks, and generating weights for larger models is impractical.

---

The method shows only minor, insignificant performance improvements and underperforms compared to DRAIN on the basic 2-moon dataset, casting doubt on its effectiveness.

We evaluated our method on 4 synthetic datasets and 6 real datasets. Our model demonstrates a significant improvement in performance compared to DRAIN, which is the current SOTA model, with an average margin of 20% across various benchmarks, as detailed in Tables 1, 2, and 3 on 6 real datasets and 3 synthetic dataset, and achieved comparable result on only 1 synthetic dataset. This general superiority is indicative of the model's robustness and effectiveness in various applications including classification, regression, and time series forecasting. Let alone our method is more time-efficient and parameter-efficient.

---

Applying proposed framework to various transformer-based networks.

We appreciate the suggestion to evaluate our framework across multiple transformer-based networks. Indeed, our model is designed to be versatile and adaptable to a variety of models dealing with temporal data. While the current paper focuses on a single transformer-based network for a thorough and focused analysis, we recognize that using our model on other baselines can also be an interesting and Exploring this in future work is definitely something we find compelling and valuable.

---

We thank the reviewer for your positive, insightful and valuable comments and suggestions which are very crucial for improving the quality of our manuscript.

Clarification on stopping criterion for pre-training the backbone.

The pre-training process is halted when the validation loss fails to improve for a predefined number of epochs (3)

---

Clarification on how learnable domain-specific prompts are connected to the backbone/output, including details on concatenation, masking during pre-training, and potential use of a linear layer.

We follow the standard practice of prompt learning to learn the prompts[Lester et al. (2021); Vu et al. (2021); Gu et al. (2021)], namely domain-specific prompts are extra tokens, and we are not increasing the dimension of tokens. The concatenation is about increasing the length of the input sequence, and transformers can take variable-length inputs. To answer your question, yes prompts are learnable parameters and they are part of the inputs inputting to the backbone network. No, we don't need to know the size of the prompt for pre-training.

---

Clarification on whether a new temporal prompt generator is trained for each prompt or if a single one is reused.

For our temporal prompt generator, we adopt a dynamic tuning approach rather than training a new generator for each temporal prompt. Specifically, the process involves loading the weights from the previous step and then fine-tuning them based on the domain using previous domains information.

---

clarification on Figure one and and the sequential training versus simultaneous output from the temporal prompt generator.

As explained in Section 3.3, it is sequential training. The temporal prompt generator is sequentially trained to generate prompts for the upcoming new domain, and previously learned prompts are fixed.

---

Concerns about the appropriateness of artificially splitting time series data into different source domains, with specific reference to the arbitrary period selection in the crypto dataset, lacking justification for the chosen duration.

We followed state-of-the-art papers [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019) ] to split the dataset, and we split the Crypto dataset in the same way. We proposed one reasonable way of splitting the dataset, and we did not aim to propose the best way to split the dataset. Brute-force all kinds of ways splitting the dataset is a NP-hard problem, which is impossible and is also trivial for a paper.

---

Method's performance largely lies in the confidence intervals of competing methods.

Yes, but our methods achieve the performances with a lot fewer parameters and much less training time. Normally machine learning models need to trade performance for efficiency, while our model shows great performance with better efficiency.

---

There is much more potential in this work than reducing it to performance metrics.

We agree, so we expanded our works to more tasks and datasets. Most previous prompting methods only demonstrated their efficacy in NLP tasks or NLP-related neural networks like GPT or CLIP. We proposed the first prompting-based temporal DG method, which is a generic model and is applied to various tasks. This shows great potential in using prompt learning for various applications other than NLP.

---

Request for experimental results on datasets naturally from different domains, or if not available, whether your method can identify different domains, possibly by comparing learned domain-specific prompts through cosine similarity.

For experimental results in Table 1, we followed state-of-the-art temporal DG methods [Nasery et al. (2021), Bai et al. (2023), Wang et al. (2020a), Ortiz-Jimenez et al.( 2019)] and adopted commonly used datasets. Crypto dataset is a time series dataset that naturally comes from different temporal domains. Domain-specific and/or temporal prompts may contain some information to determine the existence of different domains, while generic prompt cannot since it contains shared information across different domains. We will compare our domain-specific prompts in the camera-ready version.

---

Inquiry about the performance of gradient-boosted trees on regression tasks and standard forecasting methods like ARIMA/Gaussian Processes on the datasets.

We have compared our methods with the current state-of-the-art methods. It would be great if the reviewer could kindly point us to existing state-of-the-art papers that apply these methods.

---

Standard deviations in Tables 3, 4, and 5.

We will report that in Camera ready, they are similar to other tables.

---

Uncertainty regarding the suitability of the term "prompt" in the context of time series, considering its origin in NLP.

We believe it is the best-fitting wording. We followed the exact standard practice of prompting [Lester et al. (2021); Vu et al. (2021); Gu et al. (2021)].

---