Learning under Temporal Label Noise

Thanks for your time and detailed feedback! We've addressed most of the questions and comments below. If there is anything else, please let us know!

W1) My primary concern is with the problem formulation, or more specifically, the goal of the problem…in time-series problems, the environment is constantly changing, and the optimal model for different timestamps should also be changing.

This appears to be a misunderstanding. Time series indeed represent changing environments, but it’s overwhelmingly common to assume the relationship between inputs and outputs is fixed because the outputs can still depend on the inputs (e.g., ARIMA makes this assumption). Any ML model trained to predict a label based on a window of time series makes this assumption.

If this question references non-stationary time series, where the generative model changes over time, then this is out of our scope. Many domains feature stationary time series and most methods assume stationarity, which allows for clearer analysis and is often sufficient for practical use. Most state-of-the-art approaches in time-series assume stationarity, which allows for clearer theoretical analysis, and does not present a problem in practice (see e.g., recent papers from top conferences NeurIPS 2024, NeurIPS 2024, ICML 2024, ICLR 2024)

For further empirical evidence, we see from our own experiments on several real-world time-series datasets that we learn near-optimal models without considering non-stationarity.

W2) The paper appears to assume that the function Q is known to the learner, and focuses only on optimizing the parameter w.

We want to flag a misunderstanding here: we never assume $Q$ is known to the learner. The only assumption the learner needs is about the class of functions parameterizing $Q$ (e.g., DNNs). $Q$ is parameterized by $\omega$ , which is learned entirely from the noisy labels, without any prior assumptions about its specific functional form. The true underlying $Q$ is known only to us, the authors, as it allows us to measure how well our proposed methods can learn $Q$ - we report these results in Table 2 and 3 (see Approx. Error).

The entire purpose of Section 3 is to construct methods that can learn $Q$ from data. To ensure this point is clear, we have modified the text to better motivate this section and outline the assumptions we make (and, more importantly, do not make).

We hope this clarifies any confusion! Please let us know if this is not the case.

W3) The authors seem to only conduct experiments on low-dimensional datasets. I think that including experiments on larger-sized datasets would strengthen the results and make the findings more convincing.

We disagree - these datasets are of a reasonable size as each instance is roughly 100 time steps long. So for each dataset, we have NxT instances with each instance having D dimensions (e.g., Sleep dataset contains 96,400 unique instances, each with 7 features and a label). Additionally, these datasets have been widely adopted as benchmarks in time-series research and are not exclusive to this paper.

That said, if there are bigger time-series datasets (with sequences of labels at each time-step) you would have liked to see included, please let us know, and we will do our best to incorporate them in a revised version during the discussion period!

—

We hope to have clarified any misunderstandings and questions you had. However, if there are any other questions we can answer in the discussion period, please let us know and we are happy to provide more details.

Thank you again!

Thank you to the authors for their response.

Regarding Q1, I recommend that the authors include the references in the revised version. Additionally, it would be beneficial to include a discussion on applying the method to handle non-stationary environments.

For Q2, my concern remains unresolved. Specifically, the statement "The only assumption the learner needs is about the class of functions parameterizing Q" suggests that the learner must know the type of function. However, I think that identifying an appropriate class of functions for Q is the most challenging aspect of this problem.

Thanks for your time and detailed feedback! We've addressed most of the questions and comments below. If there is anything else, please let us know!

Q1) How is learning achieved when weights are initialized for each time step in discontinuous estimation / Can the estimator in the discontinuous case be replaced by a simpler model (with lesser number of parameters)?

The discontinuous case is actually the simplest possible model we can use. In this case, we are just learning the specific entries for the temporal noise function (a C \times C matrix) at each time-step $t$. For example, in the binary-setting each time-step will only have 2 parameters we are estimating (because each row of $Q_t$ are probabilities that sum to 1). We think it is quite reasonable to achieve learning of 2 real-valued parameters with the dataset sizes we use.

However, you bring up an interesting perspective, one that we think actually demonstrates the value of our approach. Choosing a specific model is largely a practitioner's decision. They can choose their own approach and still use it with the machinery we construct. For example, we use a DNN to model $Q$ in the Continuous case, but a practitioner could use a GP or set of ODEs to achieve the same goal based on their specific use-case.

Q2) It is interesting that both continuous and discontinuous estimations perform quite similarly on the real-world stress detection example. Does this mean that the proposed approach works best when there is large variance in noise rates?

This is an insightful observation. There are several possible explanations for this effect, one of which aligns with your suggestion. It is plausible that the performance of each method depends on the specific properties of the dataset and the practitioner’s design choices. In this case, the discontinuous method may perform well due to high variance in noise rates, where neighboring time-steps exhibit volatility. Another potential explanation is that different individuals in the dataset may follow slightly different temporal noise models, requiring each method to learn the most representative one across the entire dataset - therefore the method with the ability to capture the most variability (Discontinuous) will perform well.

This observation reinforces the importance of studying temporal label noise, a largely unexplored yet important area of research. We hope that discussions like this will inspire further study in this field.

Q3) Is there an assumption that all data instances have [an] equal number of time steps?

Not at all! Our problem setup only requires a model that takes the form $p(y_t \mid x_{1:t})$. In practice, the RNN architecture we use can handle sequences of varying lengths, even within the same dataset. However, the discontinuous approach does have a limitation: it requires all sequences to have an equal number of time steps, as a specific $Q_t$ is defined for each time step. In contrast, the continuous approach is more flexible, as it learns $Q(t)$ , a function of time, which can theoretically adapt to any time step $t>0$.

—

Thank you again for the feedback, and if there are any other questions we can answer in the discussion period, please let us know.

Thanks for your time and detailed feedback! We've addressed most of the questions and comments below. If there is anything else, please let us know!

W1) “(important) …the appendix of the paper is… dissociated to the main paper”

Thank you for suggesting these revisions! We have made all suggested changes to the Appendix. In summary, we have made the following changes:

• Streamlined the appendix by removing Appendix A (not referenced in the main article) and moving extra supplemental figures to the anonymous repository (e.g., Appendix G.2 has been removed). To clarify, Appendix A originally described a secondary, less-performant noise-tolerant sequential loss function.
• All figures in the Appendices now have detailed captions.
• Proposition 1 now stands out without Appendix A, further highlighting the proof of our primary theoretical result.
• The assumptions listed in the Appendices were equivalent to the ones in the main text - we have unified the description of the assumptions to ensure that this is clear to future readers.
• Moved definitions and quantities for proofs into a table, thank you for this suggestion!

W2) Regarding notation and problem formulation:

We have revised our paper as follows to address the concerns about notation and problem formulation:

• $q_t$ denotes a probability that is conditioned/dependent on time. We have clarified in Section 2 that this notation is to highlight this property.
• We have elaborated on the assumptions we use to make them more intuitive for the reader. In Preliminaries (Section 2), we clarify that Assumption 1 requires that the current observation is independent of the future observations (i.e., the present does not depend on the future) and Assumption 2 specifies a feature-independent noise regime - both of which are standard and intuitive.
• In Section 3.2, we have added a key citation on convex optimization which motivates our use of the Frobenius norm in our optimization objective.
• We have fixed the typos in line 260 and line 277, thank you for pointing these out!
• We have also clarified the dimensionality of the input to $h_\theta$ in Section 2.1. $h_\theta$ accepts as input $x_{1:t}$, where $t \leq T$. Thank you for pointing this out!
• Lastly, we have improved the description of the role of the Lagrangian multiplier in Section 3.2. It is the Lagrangian multiplier used to satisfy the equality-constrained optimization problem in Eq 2.

Q1) “If the noisy label is always accurate, can we use it for prediction?... I understand this is a slightly different setting, as the goal of the paper is to learn a predictor.”

This is an interesting idea, and is outside the scope of our problem, as you note. Our goal is to develop a time-series classifier that predicts clean labels at each time step based solely on the features. The main challenge in incorporating the noisy label into predictions is exactly as you mentioned: we cannot reliably determine when the noisy label is accurate and when it is not. If the learner had this information then that implies they have some prior knowledge of the noise model. We do not assume this to be the case in our paper, and instead learn this noise model from data.

This point is interesting, as there may be alternative strategies to learning under temporal label noise. This is a promising direction of future research and our work can inspire other such directions.

Q2) “[Can you clarify]... the difference … between Eq(3) and the model discussed in “discontinuous estimation”. It seems to me that t is an integer, and Qw(t) could be completely different than Qw(t+1) in principle"

$Q(t)$ absolutely could be completely different from $Q(t+1)$ – this is exactly why we designed the Discontinuous method, which assumes independence between neighboring timesteps and therefore can capture this local temporal variance.

So these models differ in how they model temporal noise. We introduced multiple methods in order to be as flexible for different practitioner needs. In short, the Discontinuous estimation learns parameters for a $Q_t$ at each time-step, treating each time-step independently. Continuous learns a unified function $Q$ with parameters $\omega$ (using a DNN) that represents a function of $t$ across all time-steps. They both use the same optimization strategy (Eq 3).

To make this distinction clearer, we have added more points in Section 3.4 - where we compare and contrast Discontinuous with Continuous. We hope this clarifies, but are happy to elaborate further if needed.

Q3) “What is the temporal relationship discussed in line 274?” The temporal relationship is baked into the Continuous method, which learns a single, time-dependent noise function. The temporal relationship arises because we are explicitly learning a single, time-dependent function (i.e., temporal) to represent all time points.

Thanks for your time and detailed feedback! We've addressed most of the questions and comments below. If there is anything else, please let us know!

W1) “In many settings, the performance of Plug-In method is inferior to that of static methods.”

‘Plug-in’ is only one of three temporal methods we propose. Overall, we find that ‘Continuous’ is by-far the best temporal method, outperforming all static methods (and outperforming ‘Plug-In’). Further, ‘Plug-In’ is a temporal extension of the static ‘Anchor’ method (not ‘VolMinNet’). So the fairest comparison is ‘Plug-In’ vs ‘Anchor’, where we actually see ‘Plug-In’ consistently outperforms static ‘Anchor’. As discussed in Section 3.4, each temporal method has advantages and disadvantages, depending on the data and task. It is up to a practitioner to balance performance with the various advantages and disadvantages of each choice.

We have clarified this point in the Results (Section 4.2) - thank you for pointing it out!

W2) “Only one of the five datasets in the manuscript contains both clean and noisy labels. In the remaining real-world scenarios.”

A lack of data is a key issue in this research area—our work is actually a step towards fixing this issue because we show that addressing label noise improves accuracy. And by including real-world data, our work can inspire other researchers to consider how temporal label noise may affect their problems, ultimately encouraging the curation of new datasets.

As the idea of temporal label noise is new, it will take time for such datasets to emerge, but our work is a crucial first step. For example, in the static label noise setting the standard dataset with both clean and noisy labels is Clothing1M. This dataset was only developed after the pace of research on static noisy labels picked up. Similarly, our work can spark progress in understanding and addressing temporal label noise.

We have highlighted this point in the Conclusion (Section 6).

W3) “The experimental scenarios are focused on healthcare”

Collecting time-stamped observations is indeed a common process across many fields. This paper was inspired by our own real-world work in healthcare: for instance, while collecting periodic health surveys alongside wearable device time-series data, we observed that participants often mislabeled their activities depending on what they were doing and when.

That being said, if there are specific time-series datasets (with sequences of labels at each time-step) you suggest, please let us know, and we will do our best to incorporate them in a revised version during the discussion period.

Q1) “What is the meaning of variable d … a multivariate time series with d variables?”

Yes, that is correct! We have updated the Preliminaries (Section 2) to clarify that our inputs are multivariate time series with d variables.

Q2) “When performing a train-test split on the datasets, were individuals also split?”

Splitting strategies depend on the dataset. For example, in the real-world stress detection demonstration, the training and testing splits did not share individuals, as there is one time series per individual. For other datasets, such as ‘moving’ and ‘senior’, we used the given train-test splits.

We have added this information to Dataset Details (Appendix E.1) so it is clear for future readers.