Thank you for your feedback and positive points about our paper. It appears to be the primary concern raised by the reviewer that there is a substantial body of work in this area and related to this problem. As we explained below, this is not accurate to the best of our knowledge and we hope the reviewer considers increasing their score after setting our response.

I admit I am no expert on dealing with data drift, however I am sure methods other than the proposed one and the baselines (which are all quite naive) exist. While it is good to see that on the given data the developed method outperforms the baselines, it seems to me like a rather low bar and not very surprising.

We respectfully disagree with the reviewer. Despite reviewer claims, our baselines are indeed very strong and suitable to this problem. We want to underscore that this problem is not only highly challenging and difficult to address but also represents a novel and crucial issue, with only a limited number of effective works in this particular area ( which we already included them). We are among the first to formulate this setting and propose a simple but yet effective way for it.

Nevertheless, if the reviewer can direct us to any methods applicable to this problem setting, we are happy to discuss and check whether or not they are relevant to our work. To the best of our knowledge, there is no existing method for our problem that we have not discussed.

I know that a huge area of work is addressing data drift issues with calibration techniques. I think in order to really show the merits of the method, the authors need to consider other state of the art methods of dealing with data drift.

To the best of our knowledge, there is not such a large body of literature using calibration techniques that specifically investigate settings and problems identical to ours. More importantly, while there is a small chance that some less relevant works may not be explicitly covered in this paper, we have ensured to discuss all important, main, and critical works and include them as baselines. Finally, we are not aware of any other state-of-the-art methods known to us that go beyond the ones presented in this paper.

We urge the reviewer to provide pointers to related works that discuss and study the very same problem as this paper. We are happy to discuss and include them in our paper if they deem to be relevant. Without this, it is difficult to address what the reviewer is concerned about.

Finally, we would like to bring the reviewer’s attention to the experiments in Figure 4 of the appendix where we adapted the "Meta-Weight-net" method of [9] to our setting and ran experiments comparing it to our method. As shown in Figure 4 of the appendix, our method outperforms Meta-Weight-net significantly, which provides additional evidence of the efficacy of our proposed method.

[9] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, Deyu Meng Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting,. NeurIPS 2019.

[10] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. ICML 2019.

[11] Kirkpatrick, James, et. al. Overcoming catastrophic forgetting in neural networks. PNAS, pp. 201611835, March 2017. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1611835114.

Shared by the reviewer:

[1] Wang, Q., Fink, O., Van Gool, L., & Dai, D. (2022). Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

[2] Hoyer, L., Dai, D., & Van Gool, L. (2022). Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

[3] Bobu, A., Tzeng, E., Hoffman, J., & Darrell, T. (2018). Adapting to continuously shifting domains. ICLR workshop track

[4] Zenke, F., Poole, B., & Ganguli, S. (2017, July). Continual learning through synaptic intelligence. In International conference on machine learning

I have found many more works in this area & I am sure you will be much better at identifying which one is most closely related and best suitable to apply to your scenario. However, I am convinced there is prior work that is easily adpated to your case, and that you should compare your method to them in order to demonstrate the merits of and the need for your approach.

As previously mentioned, this problem setting is relatively fresh and we are among first who formulate it where we propose a simple yet highly effective method to address it. Importantly, we have written an elaborate paper and have discussed the relevant work fairly and exhaustively. We have used strong baseline methods to evaluate our work clearly, e.g., using the Everything baseline, or using Fine-tuning and we consider both RL and surprised learning. The papers that you have suggested are on continual learning and not directly related to this problem. It is not at all fair to us to say that “you are convinced there is prior work that is easily adapted to our problem”. We would appreciate a pointer to any related pointer if it is missed. However, none of the above papers is relevant as we explained.

Thank you for your positive, detailed and valuable feedback. We hope you'll consider increasing your score after reading our responses as we believe we have addressed all your comments. Please let us know if you have more comments.

The baselines considered for the supervised learning experiments are reasonable, but the RL experiments don't consider these baselines. As written, it's unclear if the time-vary propensity score is any better than simpler methods.

Thanks for the question. The distinct nature of RL and supervised learning methods necessitates modifications to the underlying RL algorithm to accommodate the same baselines as supervised learning. In particular, SAC, an off-policy algorithm, utilizes a replay buffer to store data during exploration and samples uniformly during training. Implementing a baseline like fine-tune in SAC necessitates an additional replay buffer that only retains recent data, introducing further alterations and new hyperparameters to the SAC algorithm. In our RL experiment, we design experiments such that it requires minimal changes to the original RL method and it doesn't change how the underlying RL algorithm works as our goal in this paper is to show rather applicability and efficacy of our method in RL, rather than proposing a new RL algorithm.

Also, ROBEL DClaw is missing the SAC + Context baseline.

No, it is not missing. We already included this in the Figure 11 of appendix. We also have another additional experiment for ROBEL DCLAW where tasks are sampled differently to show the effectiveness of our method in another setting. Figure 10 shows the result of this experiment. This experiment is another piece of evidence that shows the effectiveness of our method.

We also like to note that when we combine the SAC or SAC + Context with our time varying propensity score, we see that our method consistently demonstrates lower variation across runs, indicating its better stability ( see Figure 10 and 11 for examples). This is particularly crucial for RL methods susceptible to instability due to distributional shifts between the policy used for data collection and the policy undergoing optimization (a.k.a. off-policy methods in RL). Our results effectively demonstrate the ability of our method to lessen the detrimental effects of off-policyness of off-policy methods like SAC.

It would be useful to give a brief summary of the ablation findings (appendix C) in the main paper. For instance, mention that the proposed method performs just as well as Everything when there is no shift,

That is an excellent suggestion and it is an easy fix. Due to page limits and the extensive number of experiments conducted in this paper, only a portion of them could be presented in the main body and the additional experiments are relegated to the appendix. We will ensure that the final version of the paper text will be updated to direct readers to these additional experiments and we also provide more information about these experiments in the main text. We will update the paper based on this suggestion.

The shaded regions for the RL curves should be darker; as of now, they are very difficult to see.

That is a very easy fix and we've now updated the figures.

Could the authors explain the CIFAR shifts described by Eq. 21? I'm hoping for a plain English description of the shift.

This formula describes the label shift used in our experiments. At each time t, the task consists of images from two particular classes, say class 6 (10% samples) and class 7 (90% samples). In this case, at the step (time t+1) causes a new class to be added to the task, i.e., it consists of class 7 (90%) and class 8 ($10$). At time t+2, the task consists of class 7 (80% samples) and class 8 (20% samples). At any time t, the test data consists of images sampled from the task at time t+1. It is important to note that there is a clear distribution [label] shift between the train and test distributions due to this construction and therefore it is an extremely direct test of our proposed approach. This shift is also gradual, in some cases the label distribution changes and in some changes a new class is introduced, but the new class only accounts for 10% samples at the instant of introduction. In simple words, the data distribution consists of two classes and the support of this distribution shifts over all the 10 CIFAR-10 classes, as it gradually undergoes label shifts.

Also, there is a typo in that formula where H should be T. This has been fixed.

Thank you for your detailed feedback. While we are disheartened by the low score you assigned to our paper, despite recognizing its novelty, we remain hopeful that, upon reading our responses, you will reconsider and increase your score as your primary comments revolve around minor clarification questions rather than fundamental issues. Furthermore, it seems that your lower score is linked to minor presentation issues. This is surprising, as all other reviewers are content with the quality, describing it as well-written and easy to follow. Please let us know if you have more questions.

The first weakness comes from how this work is situated in the literature, specifically the effort to distance this work from related work in continual learning. Specifically, on page 3 in Remark 2. I would argue that your setting is exactly encapsulated by continual learning frameworks. For a framework https://arxiv.org/pdf/2012.13490.pdf is a good resource. Specifically, definition 1 is a good starting point. While these definitions take a reinforcement learning perspective, they have counterparts in supervised settings as well, and can be readily adjusted to the supervised setting.

We think there is a simple misunderstanding here. We don’t try to distance our work from continual learning literature but rather we study a different problem and there is a very clear distinction between what we do and continual learning. In particular, we focus on the problem of “how to select which past samples are similar to the current distribution of data”. We use this to predict only the immediate next task, namely $p_\{T+1\}$. Continual learning focuses on how to update a model to a new task, and it seeks to predict accurately on all past tasks. The two are totally different problems, but both are important and valuable problems. Further, we have 100s of tasks in our experiments, current continual learning methods are ill-suited to address such a large number of evolving tasks.

I’m not sure I’m understanding the construction of your dataset D, and subsequently p(t). To me, each datum is received sequentially and therefore t_i is unique for each entry. This means p(t) is just a uniform distribution for the number of timesteps (of course this is not entirely accurate as t is assumed to be from the reals and not integers, but in effect this is what I understand). Given how you use this distribution, I’m assuming this is not the case. But it is not clear what t is here, how does t relate to how the data was generated? How is your dataset constructed? This confusion continues throughout the paper including all of section 2, and when defining objectives in section 3.

Let us repeat a bit more elaborately the construction in Section 2.1. Suppose the dataset is $D =\\{x_i, t_i\\}^N_i$. The time $t$ is a modeling choice, it can be an integer or a real number. The time $t_i$ simply refers to the time at which a datum was recorded, say the Unix time. Let us suppose it is a real number, like we have done in the paper. If each datum is received at a unique time then all the $t_i$s are different. Like we described in the paper, the distribution $p(t) = \frac{1}{N} \sum_{i=1}^N \delta_{t_i}(t)$, i.e., the empirical distribution of the times recorded in the dataset. It is important to note that some times may not have any samples; this is not an issue when we take expectations/integrals over time $t$ because the distribution $p(t)$ will have zero probability over such times. If multiple samples were recorded at some instant, there would simply be two Dirac deltas in this summation. All the theory remains the same.

The three lines before remark 4 are very confusing, and don’t clarify how is estimated. Do you mean t’ instead of s’?

Thanks, yes, s’ should be t’. This has been fixed. Please see the response to the following question.

This comes up again after remark 4 under equation 10. It is not clear how the dataset is being generated to estimate g_\theta, and the subtle difference from equation 8&9 is completely unclear (yes different distributions of the data are used, but how). It is very possible Algorithm 2 will clarify this, but I think a clarification on what the time in the original dataset (see 1) will deal with a lot of the confusion here.

Let us explain algorithm 2 to clarify how data is generated.

To train our propensity score, we need to create a triplet $(x_j,t',t)$ for each of the samples in the dataset. In particular, if $t_j$ denotes the true time from which $x_j$ is observed, then we want our training data to have $t' = t^j$ ( and $t$ equals to a random time) in half of the training examples with label 1 and $t'$ equals random time in the other half (with $t =t_j$) with label -1. Once this dataset is generated, the rest is to train a binary classifier using equation 19.

Please let us now if this is still unclear, happy to provide more explanation.

Thank you for the detailed response! I did not mean to dishearten you from the low score, but didn’t think the paper as presented was ready for publication and wanted to give a clear signal. It is surprising that my review is quite different from others, which might mean I was misreading some fundamental part of the paper which threw me off.

I’m always looking for clarity in a paper so that I can exactly replicate the results without many assumptions, and the many questions left by your paper (as indicated in my review) left me with little confidence to be able to do this. I know some of these comments come off as me being pedantic, but it is to try and make a paper clearer not as a means to dishearten or discourage. Thank you for your clarifications, as they do help.

I still find myself grasping to understand how g_theta is being estimated. I think the key piece of information that I was missing was that t and t’ are randomly sampled from some distribution? Where do these new values come from (because they are clearly not just t_j)? If this is clarified it would make the rest of my questions self-answerable.

Some other comments:

No access to t+1, always testing against t+1 These comments are coming from a place of confusion with your method. I know you are proposing to estimate distributions into the future. I was trying to clarify whether I was misunderstanding your motivation, not understanding the method, or missing something else. Thank you for the clarification.

Relative Accuracy:

1. I appreciate this definition being added to the paper. While it is standard in the literature, these details still merit appearing in the text (even if relegated to the appendix). This comes from a desire to reduce the barrier of entry of the field.
2. I mentioned this metric being misleading not in an effort to suggest you were lying, just as a desire to understand the absolute performance of our models. While we can make a comparison across methods using the relative measure, this only gives us an ordering of algorithms. If all the algorithms perform poorly, then the relative performance quickly becomes a meaningless metric.

Baselines: I suggest the oracle baseline not as a true competitor but as a way to understand “what is the best our methods could ever hope to do”. This again is me wanting to understand more about the absolute performance.

Shaded Regions: This can vary quite a bit from paper to paper. Sometimes shaded regions are standard error, sometimes they are 95% confidence intervals calculated w/ bootstrap. I did not see this specified in this particular paper, and again comes from me asking the question “could I recreate these results”.

Misunderstanding of the hyperparameters Thank you for the clarifications! I had read these tables as changes only to your method, and not all the methods,

After reading my original review and looking at the paper again, there were obvious misunderstandings I had but I do believe there is more this paper could have done for clarity. I agree with the authors that my original score was a bit tough. I think if the authors could clarify where t and t’ come from, and address my additional comments I will be happy with the state of the paper.

Thank you for revising and increasing your score and acknowledging the contribution and value of our paper. We are grateful for the reviewer's scholarly interaction with us which we used to further improve our paper.

Sure, we'll update the final version with clarification about t and t'. Please let us know if you have more questions. Thank you.

Evaluating on test data seems very counterintuitive as typical settings would not have this privileged information except during training. Can you provide how this specific choice of evaluation can be overcome? I find it quite limiting to do so.

There seems to be a simple misunderstanding here. We are not sure why the reviewer finds the test setting to be counterintuitive and why they think our method has access to privileged information during the test time which is otherwise only available during training time. Could you please provide more details on these matters so that we can address their concerns effectively?

In the meantime, let us clarify further the test and training time settings. For the purposes of calculating the propensity score, we create the datasets $\\{(x,+1), (x’,-1)\\}$ using the training samples. This estimator is run on all the training samples from $p_T$ and $p_t$ for $t < T$ to calculate the propensity of each of the $p_t$ with respect to $p_T$. No test samples are ever used during training time (as is appropriate). A model is trained using the weighted version of the dataset (which now has samples from t = 0 to t = T) and is used to make inference on samples from $p_{T+1}$. No samples from $p_{T+1}$ are used to train the propensity estimator or the model.

Moreover, we emphasize that there is no information about when, where, or what shifts happen in the past, present, or future during both training and test times and our method doesn’t know them as prior knowledge. More importantly, this is why it is remarkable that our method can identify continuously changing data distributions without knowing when or how the distribution changed.

Is it possible to get the best of both worlds instead i.e. get small risk on the latest task/future task while maintaining low risk across ALL tasks that the agent has seen, as otherwise there is a recency bias to only caring about the immediate next data point.

That is certainly possible and that is a good suggestion. In this work, we mainly focus on the setting where the problem is not formulated as a meta-learning objective and there is no access to the test-data to do adaptation which is closer to real-world setting. Our results confirm the effectiveness and applicability of our proposed method. However, combining our method with meta-learning based techniques can be a future research direction, but that is beyond the scope of this paper. In general, one should expect we cannot both have a small risk on the latest task and small risk on all past tasks, a number of papers have argued this, e.g., [10]. There can be situations when the tasks have solutions that are very similar to each other, and in such cases one will indeed have small risk on the latest task as well as past tasks (but there is also no effective distribution shift in such problems).

Finally, we would like to bring the reviewer’s attention to the experiments in Figure 4 of the appendix where we adapted the "Meta-Weight-net" method of [9] to our setting and ran experiments comparing it to our method. As shown in Figure 4 of the appendix, our method outperforms Meta-Weight-net significantly, which provides additional evidence of the efficacy of our proposed method.

[9] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, Deyu Meng Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting,. NeurIPS 2019.

[10] Rahul Ramesh, Pratik Chaudhari, Model Zoo: A Growing "Brain" That Learns Continually,. ICLR 2022.

Please let us know if you have more questions.

Thank you for your positive, detailed, and valuable feedback. We hope you'll consider increasing your score after reading our responses as we believe we have addressed all your comments. Please let us know if you have more comments.

The assumption that data evolves gradually which might not be true in many real world situations and limits the scope of application of the work.

While we agree with the reviewer that there are other kinds of shifts that can change data, the gradual shift assumption is relevant for many real-world scenarios where data is collected from evolving distributions that change over an extended period. For instance, drift in environmental variables such as temperature caused by day/night or seasonal changes affects both the forecasting systems and the systems that collect the data [1]. There are also many examples in oceanography, e.g., currents and other relevant oceanographic variables such as salinity and temperature in a coastal area change slowly with time [2,3] and modifying the forecasts for these variables is challenging because of the drift in the data. Even for problems in healthcare, there are many situations where data drifts gradually, e.g., the different parts of the brain atrophy slowly as the brain ages for both healthy subjects and those with dementia; tracking these changes and distinguishing between them is very useful for staging the disease and deciding treatments [4,5]. Viruses can also mutate and drift over time, which can make them resistant to existing treatments [6,7,8]. We hope that this addresses your concern about the motivation of the problem: gradual drift data is indeed a problem with broad applications. There is enormous economical and intellectual value in addressing graduate data drift.

[1] Learning under Concept Drift: A Review. IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 12, pp. 2346-2363, 1 Dec. 2019, doi: 10.1109/TKDE.2018.2876857.

[2] Operational Forecast System, Center for Operational Oceanographic Products and Services. https://tidesandcurrents.noaa.gov/models.html

[3] Changing sea levels: effects of tides, weather and climate. David Pugh. Cambridge University Press.

[4] Saito Y, Kamagata K, et al., Temporal Progression Patterns of Brain Atrophy in Corticobasal Syndrome and Progressive Supranuclear Palsy Revealed by Subtype and Stage Inference (SuStaIn). Front Neurol. 2022.

[5] Rongguang Wang, et al., Adapting Machine Learning Diagnostic Models to New Populations Using a Small Amount of Data: Results from Clinical Neuroscience. 2023. https://arxiv.org/abs/2308.03175

[6] Ewen Callaway. The coronavirus is mutating — does it matter?, September 2020.

[7] William T Harvey, et al., COVID-19 Genomics UK (COG-UK) Consortium, et al. Sars-cov-2 variants, spike mutations and immune escape. Nature Reviews Microbiology, 19(7):409–424, 2021.

[8] C.J. Russell. Orthomyxoviruses: Structure of antigens. In Reference Module in Biomedical Sciences. Elsevier, 2016. ISBN 978-0-12-801238-3.

Consider for e.g. the sudden shifts in the data distribution. Would the proposed approach be extendable to such settings? What do we need to cover this setting as well?

This is a good question. Yes, our method can be potentially extended to such a setting as long as there is data in the past which are similar or close to the current sudden shift. If the shift is sudden, e.g., if there is no predictable pattern in how the data evolved suddenly and the new data is very different from all past data, our method and in fact no other method will work without explicitly using test data to adapt.

How is the Finetune baseline different from typical online-meta learning approaches? Finetune typically would continue learning a hypothesis on all past tasks and then adapts it further using data from the new task at hand that is in round T . for e.g. in [1] and [2].

That is a great question. The main difference between Finetune and online meta learning is that online meta learning methods are formulated as meta-learning problems and they use test data for the adaptation. For the fine-tune baseline, we don’t have access to the test data to do adaptation and we don't formulate our problem like a meta-learning objective.

Is it primarily that in your method the desiderata is essentially different i.e to obtain a small risk on pT +dt — as opposed to obtaining a small population risk on all tasks.

Exactly, that is correct. Afterall, that is the most realistic problem in situations where the data distribution changes; the model must work well for the new data, not on average over all past data.