Learning with Temporal Label Noise

Thank you for your time and feedback! We address your questions below:

"`The paper does not discuss the computational complexity. How about the training time compared to other methods?"

Thanks for raising this point! The compared methods have the same sequential model architecture (GRU), number of gradient updates, and their computational graphs are quite similar. Therefore, differences in complexity will be negligible and arise from the noise-transition estimation function. TENOR is actually significantly more efficient in time and memory complexity than the nearest competing baseline VolMinTime. This efficiency stems from TENOR’s single network to estimate the noise-transition function for all time steps. This process can actually be done in a batched fashion for all time steps at once, with a single gradient update. Meanwhile, VolMinTime learns a separate set of weights at time step, each requiring an independent gradient update. In terms of memory complexity, as the length of the time-series increases, TENOR’s memory cost remains constant as it models for all time. Whereas, VolMinTime stores a separate set of weights at each time, therefore growing linearly with T.

”Is it possible to show that the datasets satisfy the assumption?"

Thank you for this question. The assumptions we make regarding modelling sequences are the most common assumptions made by baseline and SOTA models for this problem setup [1-3]. Plus, our findings are the same on synthetic and real-world data, suggesting the assumptions baked into the synthetic data remain valid.

”Fix the references”

Thanks for catching this, we have fixed the typos in the references.

References

[1] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in Neural Information Processing Systems, 13, 2000.

[2] Yo Joong Choe, Jaehyeok Shin, and Neil Spencer. Probabilistic interpretations of recurrent neural networks. Probabilistic Graphical Models, 2017.

[3] Rubanova, Yulia, Ricky TQ Chen, and David K. Duvenaud. "Latent ordinary differential equations for irregularly-sampled time series." Advances in neural information processing systems 32 (2019).

Thank you for your time and feedback! We respond to your questions below:

"What motivated you to introduce this new label noise setting? Does the temporal label noise exist in the real-world?"

Thank you for asking this question! We were actually motivated to work on this because of real-world applications that we work on in the healthcare domain. In particular, user-annotated human activity time series data, where participants notoriously mislabel their activities depending on when and what they are doing.

Stepping back, we identified that this problem arises in a substantial class of real-world applications where labels are collected over time, especially when people label their own data over time. Well-known examples include:

1. Human activity recognition: Human activity recognition models detect people’s activities over time, usually from wearable devices or smartphones. For example, smartphone apps may tailor notifications to someone’s behaviour. But training labels are provided by people annotating their own activities over time [1,2]. Such self-annotation is notoriously noisy. For example, people often inaccurately report when they fall asleep or wake up [3]. Such annotations suffer from temporal label noise because some times have more noise than others (e.g., around sleep).
2. Modelling mental health over time: Consider suicide-risk prediction tools that use wearable sensors to alert doctors of patient risks. These models are trained on self-reported indicators of suicide-risk [4]. But studies consistently show that users can fear repercussions and underrepresent their true suicide risk [5]. In this case, periods of time where individuals are suicidal are subject to increased label noise.
3. Temporal bias in grading: When grading student assignments, later assignments can receive worse grades, likely as graders tire [6]. This sequential bias implies label noise increased over time in the 30 million records studied in this work. We have included a new citation to this case in the Introduction.

"The authors assumption that ‘the sequence of noisy labels is independent of the features given the true labels’ can be strong, since the label noise in the real-world can be instance-dependent and class-dependent."

You are right that this may be a strong assumption. We wanted to be explicit about this assumption in the manuscript.

You are right that there are other variants of real-world label noise that can arise. Stepping back, we actually wanted to focus on the canonical setting in which we had class-conditional label noise that is feature-independent – this was important to pinpoint how methods and performance would change as a result of temporal effects.

Our method is built to handle class-dependent label noise. Handling instance-dependent is out of scope (and generally a challenging problem even in the non-temporal, static setting) [7-9].

We also consider the setting where noise is simultaneously class- and time-dependent (that is, a unique time-dependent noise process for each class). The temporal noise type “Mixed” reported in the paper represents class- and time-dependent noise. We showcase the superior performance of TENOR in this case (Table 1)! Additionally, we have also added experiments on synthetic and real-world multiclass sequential classification tasks, showing that TENOR has strong performance in these settings as well (Appendix E.7). We believe this further increases the utility and generalizability of our work.

"The comparison methods can be outdated. It is better to consider more SOTA methods as baseline. Furthermore, how you adaptive the baseline to fit your specific setting"

We tried to choose baselines that were used in prior work: VolMinNet (ICML 2021) [10] and Anchor (CVPR 2017) [11]. We’re happy to run experiments against other methods if you would like (just let us know). In general, we think that there is a marginal benefit in adding more baseline methods since they assume that label noise is static over time and will fail when the label noise is temporal (albeit at different rates). We show that such static methods are insufficient, even if they estimate the noise perfectly (Fig 2), so more-extensive comparisons are out of scope. As it stands, these points apply to other methods and future attempts to make existing methods time-dependent.

One point that we would like to make is that some of the “baseline methods” that we present in the experiments are actually methods that we developed ourselves (e.g.: VolMinTime and AnchorTime)! These are methods where we take the existing static method and modify it to a temporal setting as described in Section 4.4 . The results show that some of the issues resulting from temporal label noise can be mitigated by allowing for temporal variation. Our results show that this improves their performance, though our approach remains superior.

"It is advisable to adapt the real-world noisy label learning dataset Clothing1M to your framework to validate the efficacy of the proposed method"

Thanks for mentioning this. We considered applying this dataset since it contains real label noise. Unfortunately, Clothing1M is not naturally applicable to our setting since each instance only has a single label – i.e., there is no “sequence” of labels over time.

Given your suggestion for real-world noisy labels, we did run experiments with the “ExtraSensory” dataset [12] which also contains “real world noisy labels” for a time series prediction task. ExtraSensory includes human activity data from smartphones and smartwatches from 60 users over a timespan of 300K minutes.

In this case, we then train binary classifiers to predict when users are asleep or awake, which is a key benchmark task. In contrast to HAR and HAR70 datasets in our experiments, the labels in ExtraSensory are user-provided and subject to label noise (since users often misreport falling asleep and waking up, especially when they are sleepy or in a rush to leave in the morning).

Our results for this dataset show that TENOR successfully identifies an interpretable temporal noise function (Appendix E.2) and that TENOR predicts there exists higher label noise near sleep/awake transitions. We also see that TENOR is responsive to these temporal effects. In this case, we cannot validate if these are the correct predictions since we do not have any cleaned ground truth labels (like HAR, HAR70, or even Clothing1M), but this is the kind of behaviour we would expect to see from a good model in this setting. We included these results in Appendix E.2 and will work to integrate them into the Experiments if you think this is valuable.

"Is the proposed method only applicable to binary classification?"

Our proposed method works for multiclass classification problems! We have included results from new experiments on multiclass data for synthetic and real-world tasks (ranging from 2 to 4 classes). We find that our method still outperforms existing methods under a variety of temporal noise functions. Please refer to the following table, demonstrating performance on 3-class classification for the synthetic dataset. Complete results can be found in Appendix E.7.

Thank you for your time and feedback! We address your questions below:

"Named entity recognition (NER) with label noise is a highly related domain... authors should discuss the similarities and differences between those domains, and if possible, apply TENOR to NER to better showcase the performance superiority."

We want to flag this comment as a potential misunderstanding: these NER papers study neither temporal noise nor time series. Instead, they randomly unlabel named entities in classic NLP datasets. Further, they use short sentences containing 10-20 words. We study time series, which often span hundreds or thousands of timesteps. However, we agree that TENOR is likely applicable broadly and we hope our work inspires such adaptations.

"Authors should consider more mild assumptions that enable loss correction to be applied to temporal classification."

We would like to flag another potential misunderstanding: the focus of our work is loss correction for temporal classification. As we mention in Section 3, the conditional independence assumptions we make are common in such sequential classification problems [1-3]. Our findings also apply to our synthetic and real-world datasets, suggesting our assumptions are realistic.

Also as mentioned in Appendix C, our assumptions provide a strong foundation for future work. Existing work in statistical learning theory on highly-dependent sequences may be a promising direction for future research, particularly in the context of temporal label noise, which we define in this work.

"The proposed TENOR framework is significantly overlapped with the existing method, authors should devise new methods that are specifically designed for temporal classification (orientated to more specific technical challenges)."

TENOR is designed for our exact setting; it conclusively outperforms existing methods. While all works are inspired by others, TENOR is the first method that addresses our novel problem setting. Beyond TENOR, we formalize a real, unstudied problem, spotlighting a critical weakness of popular methods.

---

References

[1] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in Neural Information Processing Systems, 13, 2000.

[2] Yo Joong Choe, Jaehyeok Shin, and Neil Spencer. Probabilistic interpretations of recurrent neural networks. Probabilistic Graphical Models, 2017.

[3] Rubanova, Yulia, Ricky TQ Chen, and David K. Duvenaud. "Latent ordinary differential equations for irregularly-sampled time series." Advances in neural information processing systems 32 (2019).

References

[1] Fu, Xiao, et al. "Robust volume minimization-based matrix factorization for remote sensing and document clustering." IEEE Transactions on Signal Processing 64.23 (2016)

[2] Fu, Xiao, Kejun Huang, and Nicholas D. Sidiropoulos. "On identifiability of nonnegative matrix factorization." IEEE Signal Processing Letters 25.3 (2018)

[3] Li, Xuefeng, et al. "Provably end-to-end label-noise learning without anchor points." International conference on machine learning. PMLR, 2021.

[4] Bertsekas, Dimitri P. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.

[5] Patrini, Giorgio, et al. "Making deep neural networks robust to label noise: A loss correction approach." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[6] Kremer, Jan, Fei Sha, and Christian Igel. "Robust active label correction." International conference on artificial intelligence and statistics. PMLR, 2018.

Thank you for your time and feedback! We respond to your questions below. We hope that this addresses some of the missing details that you were looking for, but please let us know if this is not the case.

"It would be good to discuss why the optimization works and is not derailed by too much label noise”

Thanks for your question, we’re glad to have a chance to elaborate!

"Why the Optimization works”

A naive approach would be to treat each time-step independently and optimize a separate function or set of weights for each $Q_t$ - this is the VolMinTime baseline in our experiments. The clever part of our implementation is how we link each $Q_t$ using the same function (a neural network with a single set of weights). Now in order to see how we can identify $Q(t)$, we need to consider how the noisy posterior decomposes. Specifically, the noisy-label posterior at time $t$ can be expressed as a product of a noise-transition matrix and a clean-label posterior

$P(\tilde{y}\_t|x\_{1:t}) = Q\_t ^ {\top} P(y\_t|x\_{1:t})$

In our case, $Q_t$ is identifiable provided the posterior distribution is sufficiently scattered – i.e., it is not uniformly distributed in the unit-simplex. This follows from an important theoretical result in Non-negative Matrix Factorization (NMF) [1, 2] and motivates a different method to estimate the noise transition-matrix estimation matrix [3], which we use as a baseline in our experiments.

"Why isn’t TENOR derailed by too much noise"

If we assume that the posterior distribution is sufficiently scattered, then $Q_t$ is identifiable, and its columns form the minimum-volume simplex that encloses $P(\tilde{y}\_t|x\_{1:t})$. Regardless of the amount of noise, the noisy posterior is a linear combination of the columns of $Q_t$ and the clean posterior. In our experiments, we go up to 40% label noise and find our methods are still effective (see Fig 3 and Appendix E).

In order to efficiently estimate $Q_t$ at all timesteps, we set up the equality-constrained optimization problem in Eq 3 as an augmented Lagrangian objective [4], which is guaranteed to converge under some regularity conditions [4].

"Theorems 1 and 2 are about equivalences, but there are no forms of estimation/finite sample guarantees in the paper.”

Thanks for raising this point! We agree that these are infinite sample guarantees. We prioritized these since it seems to be the standard in previous work [5,6]. With that being said, we agree that finite sample guarantees are most important. We will work to see if anything can be said for the finite sample condition. Meanwhile we have updated the text to state this clearly and explained its limitations (Appendix C - Limitations and Future Work).

“It's unclear how the TENOR objective imposes regularity across time 't… Part of this confusion arises because of my lack of clarity in how $Q(t)$ is modeled via a neural network”

We briefly discuss how we model $Q_{\omega}(t)$ in Appendix D.2 and are happy to elaborate more here. We use a fully-connected DNN with weights $\omega$ that takes a real-valued time-step as input ($t \rightarrow Q_t$) and outputs a valid noise transition matrix as per (Def 1). To ensure that the output is a valid we take the output and:

1. reshape it into a square matrix $C^2 \times 1$ vector is converted into a (C x C) matrix)
2. apply a softmax operation row-wise to ensure entries are valid transition probabilities
3. add an identity matrix to ensure diagonal dominance
4. and finally we row-normalize to ensure the final matrix is row-stochastic.

Since these operations are differentiable, we can then fit the underlying function $Q(t)$ using standard training algorithms.

In this case, the loss function that we use represents the aggregate volume across all timesteps. The optimal weights of the network are therefore the weights that generate the minimum volume transition matrix at each time step. Because the network modelling $Q(t)$ shares one set of weights for all timesteps, we can then model any time-dependent function, due to the universal representation properties of neural networks.

Stepping back, one important point that we want to make is that TENOR can work with any model $Q(t)$ – i.e., beyond the ones that we learn from data using the method described above. For example, we could also use a Neural ODE to parameterize $dQ/dt$ and estimate $Q(t)$ for all t in 1…T by solving an initial value problem. We describe this in greater detail in Appendix C.

Dear reviewers,

Thank you again for all of your feedback. We'd like to kindly remind you that we've addressed your concerns in our responses below and we look forward to hearing back. We have worked hard to address your reviews by adding experiments, clarifying some misunderstandings, and reflecting your suggestions in an updated manuscript. As the window for discussion is closing, we want to make sure we have the chance to clarify any lingering concerns.

Thank you once again for your invaluable feedback.