Optimal Information Retention for Time-Series Explanations

Comment:

We extend our sincere appreciation to Reviewer Szo4 for providing valuable feedback and acknowledgment of our research.

• Theoretical Claims: This paper has three criteria: Semantic Information Retention, Minimum Redundant Information Retention, and Maximum Effective Information Retention. In the appendix, the authors provide proofs of the equivalence between these criteria and the implementation loss functions. The proof of Theorem 1 appears to be well-structured, but this reviewer is uncertain about the proof of Theorem 2, particularly the relationship between Equations 32 and 33.

  Reply: For Equations 32 to 33, it is described how to derive positive and negative samples and the corresponding loss function in contrastive learning. Specifically, we learn a mask matrix $M$ that can separate the effective features, and mask $X$ to get $X^m$. However, $X^m$ cannot be directly used as a positive sample, as this would introduce too many 0 values and cause OOD. To do this, we use an MLP to map $X^m$ to $\widehat{X}$. Conversely, the inverse mask filters out redundant features, which can be used as negative samples that are not relevant to the task. The positive sample $\widehat{X}$ and the negative sample $\widehat{X}^-$ thus constructed can be used for contrastive loss, namly, Equations 33.

• Other Strengths And Weaknesses:

  • In Figure 1, the description is not precise. Since the overlapping area represents mutual information, the green area should correspond to $H(X)$.

    Reply: Thanks for your advice. We will fix this description in an updated version.

  • The implementation of Criterion 2.3 is unclear. In line 167, the assumption states: "Assuming $\widehat{X}$ has no overlap with the portions of $H(X | Y )$". However, the statement, "For the second term, since X and $\widehat{X}$ share effective information in $X_m$ , they can naturally regarded as the anchor samples and the positive samples, respectively, in contrastive learning." contradicts this assumption.

    Reply: For $\widehat{X}$ can be considered as the product of $X^m$ learned by MLP, while $X^m$ is considered as the valid feature retained after occlusion. Ideally, $H(X^m)$ should correspond to $I(X; Y)$ consistent. When the information of $Y$ is excluded, the remaining redundant information of $X$ can be considered as irrelevant to $\widehat{X}$, so "no overlap with the portions of $H(X | Y)$". Furthermore, $\widehat{X}$and $X^m$ have information consistency with respect to the label $Y$ that $X$ is oriented to, which is also the motivation for constructing positive samples. So $\widehat{X}$ can be used for contrastive learning from $X^m$. Accordingly, $X$ can be regarded as the anchor of contrastive learning.

Comment:

We express our sincere gratitude to Reviewer w93i for providing comprehensive review, insightful perspectives, and thought-provoking questions.

• Claims And Evidence: However the claim that " We achieve ... on real-world datasets (Figure 3).

  Reply: Thank you for your advice. We will revise the presentation to faithfully reflect the experimental conclusions.

• Methods And Evaluation Criteria: However, the paper misses a discussion of the interpretability of the masks... A lack of discussion / experiments / visualizations on this aspect is a critical drawback of this paper.

  Reply: Thank you again for your constructive suggestions, which will be very helpful for a discussion on interpretation and expert cognition. We evaluate the meaning of mask-based explanations with expert explanations from quantitative and qualitative perspectives, respectively.

  1. For synthetic data, we followed the same experimental setup as TIMEX and TIMEX++ with pre-defined Ground-Truth explanations. Quantitative analysis of the consistency between salient features and Ground-Truth can accurately evaluate the interpretation effect. For visualization of synthetic data FreqShapes explanation, as shown in Figure R4 (https://anonymous.4open.science/r/orte/anonymous.pdf). Compared with the strongest baseline TIMEX++, our method maintains stronger consistency with the Ground-Truth. Likewise, we provided corresponding visualizations and discussions in the supplementary material of the manuscript.
  2. For annotated real-world data, ECG data has expert-predefined Ground-Truth to refer to. Table 2 of the manuscript provides a quantitative evaluation. In order to further explore explain the results and human domain expert cognitive consistency, we visualize our method and the interpretation of the baseline method, as shown in Figure R3 (https://anonymous.4open.science/r/orte/anonymous.pdf). The Figure R3 compares our method with IG, TIMEX++, and Ground Truth (QRS complex waves). In addition, we invited clinicians to analyze the interpretation results. The professional physician proposed that the Ground Truth in the figure may be mixed with the P wave on the left that does not belong to the QRS wave, and the more accurate Ground Truth should be more narrow, which further verified that our method may provide an insight for the interpretation of real data.
  3. For the real-world data without annotations, on the one hand, we followed TIMEX and TIMEX++, progressively occluding the saliency features at the bottom of the $p$-percentage. Ideally, the redundant features are assigned low saliency, and all the important features (i.e., completeness) are assigned high saliency. When the low saliency features are occluded, the prediction performance of the algorithm will be less affected. On the other hand, we followed the suggestion of reviewer QJfM and supplemented the imputation experiment to support the requirements of redundancy and completeness in the interpretation results, as detailed in the reply to the Claims And Evidence section of reviewer QJfM.

• Other Strengths And Weaknesses: the trade-offs of the mask continuity.

  Reply:

  1. For mask continuity, we follow the TIMEX experimental setup, we also investigated the influence of continuity, as shown in Figure R2 (c) (https://anonymous.4open.science/r/orte/anonymous.pdf). We tested AUPRC, AUP, and AUR on SeqComb-MV data. The results showed that continuity did not affect the algorithm significantly, so we chose a consistent setting.

  2. For the sparse mask, Figure R2 (d) (https://anonymous.4open.science/r/orte/anonymous.pdf) investigated the sensitivity. In the interval $[0.001, 0.05]$, the interpretation of the algorithm is relatively stable. However, there will be mode collapse when the sparsity is too large, which is consistent with our expectation that effective features will be missed when the sparsity is too large.

• Questions For Authors:

  • Q1. the human-interpretability of the mask explanations, and the impact of the "mask continuity" parameter?

    Reply:

    1. For the human-interpretability of the mask explanations, please refer to the reply of Methods And Evaluation Criteria.
    2. For the impact of the "mask continuity" parameter, please refer to the reply of Other Strengths And Weaknesses.

  • Q2. the incomplete literature survey, and how the proposed method compares to the literature on masking explanations?

    Reply: We will further clarify our literature survey, and TIMEX and TIMEX++ as our baselines, following masking explanations, both are based on mask learning saliency distributions. And the other baseline method Dynamask, is also based on mask learning. In the updated version we will add a more extensive discussion, including time series and other general methods as you mentioned in Essential References Not Discussed.

Comment:

We sincerely appreciate Reviewer voUp for offering valuable insights and recognizing of our work.

• Claims And Evidence:

  • C4. Practical utility: claim lacks detailed comparison of computational demands versus simpler approaches, which would be important for real-world applications. Introducing (even theoretical) estimation of time/memory complexity would be welcome.

    Reply: We evaluated the computational requirements of interpretability methods on two real-world datasets (PAM and Epilepsy), as detailed in Table R1 (https://anonymous.4open.science/r/orte/anonymous.pdf). The computing infrastructure consisted of Ubuntu 18.04.6 LTS and an NVIDIA GeForce RTX 2080. Table R1 compares our method (ORTE) with four baseline approaches (IG, Dynamask, TIMEX, and TIMEX++) in terms of parameters (M), FLOPs (G), and Inference Runtime (s). Compared with TIMEX and Timex++, our inference time is slightly longer due to the double-STE to implement the regulation of discrete maps, but all three are in the same rank, i.e., less than 1 second. IG and Dynamask require longer inference time, which is consistent with expectations since both methods operate recursively on a single sample, while our method requires only one forward propagation.

  • C5. Real-world evaluations: while the occlusion experiments on real-world datasets without ground truth (Figure 3) show ORTE maintains high prediction AUROC, this is only a proxy measure and doesn't definitively prove the explanations are correctly identifying the truly important features. Other work has introduced evaluations by consulting subject-matter experts and comparing their interpretation of variables of importance with those generated by the proposed framework.

    Reply: Thanks for your advice. We consulted clinical experts and analyzed real-world ECG data, as shown in Figure R3 (https://anonymous.4open.science/r/orte/anonymous.pdf). Figure R3 compares our method with IG, TIMEX++, and Ground Truth (QRS complex waves). Although IG accurately locates some points of the QRS, more information is missed compared with the Ground-Truth. While TIMEX++ highlights the salient features, it also mixes the redundant features, which makes it difficult to understand the prediction of the algorithm. Our method maintains less redundancy while highlighting more complete salient features, which verifies the optimal information retention principle proposed in this paper. Notably, clinical experts identified a critical annotation discrepancy: the reference Ground Truth inadvertently incorporated proximal P-wave components (non-QRS elements) at leftward leads. This observation suggests that the benchmarks may require narrower physiological bounds, which further supports that our method may provide new insight for the interpretation of real data.

• Other Strengths And Weaknesses:

  • W1. The real-world evaluations as previously mentioned.

    Reply: For the evaluation of real-world data, on the one hand, we followed your friendly advice and consulted clinical experts as a supplement to the explanation evaluation, as mentioned in C5 reply of Claims And Evidence. On the other hand, we followed the advice of reviewer QJfM and supplemented the imputation experiments, as detailed in the reply to the Claims And Evidence section of reviewer QJfM.

  • W2. No significant investigation into edge-cases like dealing with irregularly sampled or missing data.

    Reply: We investigated edge-case of missing data, as shown in Table R2 (https://anonymous.4open.science/r/orte/anonymous.pdf). We compare the proposed method ORTE with the strongest baseline TIMEX++ on SeqComb-MV data. We set three scenarios including random loss of 5% and 15% of data points and random loss one variable, respectively. The results show that our method performs well on AUPRC, AUP and AUR, which is consistent with the claims of the manuscript.

• Questions For Authors:

  • Q1. How does the computational complexity of ORTE compare to methods like IG or TIMEX++?

    Reply: Please refer to C4 reply of Claims And Evidence.

  • Q2. How sensitive is the method to the choice of hyperparameters, particularly those in the contrastive learning component?

    Reply: We investigated the hyperparameter choice of contrastive learning (i.e., $\alpha$), as shown in Figure R2(b) (https://anonymous.4open.science/r/orte/anonymous.pdf). When $0.1 \leqslant \alpha \leqslant 13$, AUPRC and AUP slowly rise and gradually plateau, while AUR gradually rises. The AUP will decrease significantly when $\alpha$ is larger, that is, the explanation fails. A value proximate to 10 is recommended for our experimental applications.

• Essential References Not Discussed:

  Reply: Thanks. We will supplement and analyze these references in the updated version.

Comment:

We sincerely appreciate Reviewer QJfM for the careful analysis of our work and valuable suggestions.

• Claims And Evidence: 1. Occlusion alone does not fully capture redundancy and completeness. & 2. Insertion experiments should complement occlusion.

  Reply: Thanks again for your advice. We believe the insertion experiments can serve as a beneficial supplement to our occlusion experiments. We completed the insertion experiments by gradually inserting the bottom $p$-percentage salient features, as shown in Figure R1 (https://anonymous.4open.science/r/orte/anonymous.pdf). Similar to the occlusion experiments, the imputation experiments compare our method (ORTE) with three baseline methods (Dynamask, TIMEX, and TIMEX++) and random insertion (Random) on three real-world datasets (PAM, Epilepsy, and Boiler). The results show that ORTE achieves a lower AUPROC when inserting the bottom 75% of salient features. As the insertion percentage increases, the predicted AUROC gradually improves. When the insertion ratio reaches 97.5%, ORTE attains the highest AUPROC. This indicates that the most interpretable or informative features are concentrated in the high-saliency features, further validating the manuscript's claims of low redundancy and high completeness. In addition, we notice that Random always maintains a high AUROC, which is because the random selection of features does not distinguish the salient or importance of features and contains informative time points or segments.

• Other Strengths And Weaknesses: Other than Limited Experimental Evaluation for Real-World Datasets and Gaussian Noise Imputation for Negative Samples, there are no concerns.

  Reply:

  1. As replied in Claims and Evidence, the imputation experiments, serving as a complementary evaluation metric on real-world data, is able to jointly support the low redundancy and high completeness claims of this paper together with the occlusion experiments.
  2. We investigated the impact of Gaussian noise intensity on negative sample constructions, as shown in Figure R2 (a) (https://anonymous.4open.science/r/orte/anonymous.pdf). We tested $\sigma = [0.1, 0.3, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0]$ eight groups of noise intensities on the multivariate synthetic dataset (SeqComb-MV). The results show that AUP is lower at $\sigma \leqslant 1.0$, while AUPRC and AUR are higher, which indicates that insufficient noise intensity may be not beneficial for the separation of interpretation patterns from redundant information. When $1.5 \leqslant \sigma \leqslant 3.0$, AUP and AUPRC are higher, which indicates that the interpretation patterns of positive samples are effectively discriminable and do not produce OOD patterns. A value proximate to 2.5 is recommended for our experimental applications. And the more analysis of gaussian noise imputation for negative samples on other datasets will be included in the updated version.

• Questions For Authors: Comment on Figure 3. ... how do the results relate to redundancy and completeness?

  Reply: For Figure 3, we stepwise occluded the bottom $p$-percentage salient features, following TIMEX and TIMEX ++. Ideally, the redundant features are assigned low saliency, and all the important features (i.e. completeness) are assigned high saliency. When the low saliency features are occluded, the prediction performance of the algorithm will be less affected. On the contrary, if the important features are identified as low saliency and are occluded, the prediction performance of the algorithm will be significantly reduced. On the one hand, this can reflect the requirements of redundancy and completeness in interpretation results, but also as pointed out in Claims and Evidence, due to the lack of ground-truth interpretation of real-world data, occlusion experiments still have the limitation of robustness. To this end, the imputation experiment can serve as a powerful supplement. Specifically, when the low saliency features are occluded or the high saliency features are inserted, the proposed algorithm can obtain higher prediction performance, which will further verify the claim of this paper.

• Essential References Not Discussed: To strengthen this connection, the authors could discuss ROAR ([1], NeurIPS 2019) ... Referencing ROAR would further validate the effectiveness of ORTE's evaluation strategy.

  Reply: Thanks for the friendly suggestion, and we will discuss the reference of ROAR in the updated version.

• Other Comments Or Suggestions: 1. Typo. Line 208 (2.3) & 2. Hyperlink issue.

  Reply:

  1. We addressed this issue in the updated version.
  2. We double-checked the hyperlinks to references and sections on multiple reading platforms.