Understanding the Limits of Deep Tabular Methods with Temporal Shift

Thank you for your valuable feedback! We will address your concerns in the following responses.

First, we would like to clarify several key differences between our work and TabReD and PLR.

1. Difference from TabReD: TabReD shows that real-world tabular datasets, inherently containing temporal shifts, require temporal splits for realistic test sets. Random splits may lead to misleading assessments, and TabReD demonstrates that temporal splits significantly alter model rankings. Our work builds upon TabReD by identifying validation set splitting as a crucial factor affecting model performance once the test set position is fixed. We find that, in this setup, randomly splitting the validation set yields better results than using the temporal split in TabReD. We further analyze the underlying reasons behind this observation and propose our refined temporal split. Therefore, our contribution focuses on validation set splitting, whereas TabReD primarily addresses test set splitting, making the two approaches complementary rather than overlapping.
2. Difference from PLR emb: Our temporal emb differs from PLR emb both in scope and design. PLR emb is a numerical feature emb method that samples periodicities from $N(0, \sigma)$ to capture cycles in numerical features. In contrast, our temporal emb is specifically designed to incorporate timestamp information into models, then treat it as a numerical input feature. It is a plug-and-play approach. Structurally, our method relies on prior cycles, making it more suitable for handling temporal patterns. Additionally, we explicitly address the challenges of multi-period coupling and trend representation, which are essential in temporal settings but are not considered in PLR emb. We also consider PLR emb a baseline when designing our temporal emb (figure 8 left).

We hope this clarifies our contributions. Our code is available at https://anonymous.4open.science/r/Tabular-Temporal-Shift-BCCA/, with additional results.

Random split perform identical.

The random split serves as a baseline. By analyzing its differences with temporal splits in TabReD, we identified the impact of training lag and validation bias, motivating our temporal split strategy.

While the random split performs well, it suffers from instability (Std increased by 154%). Our proposed temporal split not only maintains competitive performance but also significantly improves stability, as shown in Table B in repository.

How are non DL/autoregressive/ICL methods effected by the splits?

1. We have presented the impact of non-DL methods (including Linear, Boosting methods, and Random Forest) under different splits in Figure 2, Figure 5, and the Appendix (page 13). These methods consistently show improvements, with generally better performance in our new temporal split.
2. We tested the Mambular method. Due to the large size of the TabReD dataset and the inefficiency of autoregressive methods, we only provided results for six datasets. Under our split, its performance showed a significant improvement (+4.56%), as shown in Table E in repository.
3. We tested TabPFN v2. Since the general model does not require training, we adjusted its context samples: in the Original split, we randomly selected 10,000 context samples, while in Ours, we selected the 10,000 context samples closest to the test set. This also led to an improvement (+0.71%), as shown in Table E.

Since Fig 8 you analyze random splits, how does random split compare in Fig 3?

Figure 8 focuses on comparing our updated temporal split with the improvement brought by adding temporal embs. The random split is only presented in Figures 2 and 5.

In Figure 3, the bar chart on the right compares the performance of four specifically constructed splits to analyze the effects of reducing training lag, mitigating validation bias, and ensuring validation set equivalence. Since these splits contain different amounts of data, their results cannot be directly compared to those of the Original, Ours, or Random splits.

Where are the splits a), b), c) and d) coming from?

The splits a), b), c), and d) are entirely our contribution. Our work focuses on different aspects compared to TabReD.

Why is "Ours" not shown in the graphic on the right?

The (a,b,c,d) splits are used only for analysis of training lag, validation bias, and validation equivalence, which can be seen as an ablation study. These splits were created by discarding data due to dataset limitations. In contrast, both the original and our temporal splits use the entire dataset before $T_{train}$. As a result, they cannot be directly compared with the Original or Ours.

We hope this response addresses your concern. Please feel free to raise any further questions!

We are grateful for your constructive suggestions! We will address your concerns in the following responses.

The detailed design for these different methods is not discussed.

We apologize for this oversight. We will add an additional section in the preliminary in the revision to introduce the fundamentals of learning from tabular data.

They present the percentage change in the robustness score, which may not be intuitive.

Thank you for your suggestion! We have now additionally provided the change in standard deviation in Table B in the repository: https://anonymous.4open.science/r/Tabular-Temporal-Shift-BCCA/. In our comparison, while our method results in a slightly higher standard deviation than the original split (+16.7%), it achieves a significantly lower standard deviation compared to random splitting (+154%).

It is recommended to use a metric that is robust to outliers.

We agree. In the revision, we will replace the average percentage change calculation with a robust average, which excludes the maximum and minimum values when computing the percentage change across the eight datasets. We have updated Figure 2 by Figure A in repository, and it continues to support the same conclusion: random splitting significantly improves model performance compared to the original split and aligns more closely with existing benchmark results.

Include a performance comparison of the methods under different protocols.

We have included a comparison of model performance under different protocols, specifically after applying our splitting method and temporal embedding, shown in Figure A bottom. All comparisons use the robust average to compute the average percentage change relative to the original split for MLP, ensuring direct comparison. Additionally, we provide the average rank of the models for a more comprehensive evaluation, shown in Table A.

We hope this response addresses your concern. Please feel free to raise any further questions!

We appreciate your thoughtful comments! We will address your concerns in the following responses.

The authors do not seem to provide the code, so I remain conservative about the results reported in the paper.

Our code is now available at https://anonymous.4open.science/r/Tabular-Temporal-Shift-BCCA/. Enjoy the code!

The temporal emb approach is tested primarily on a fixed set of known cycles.

The focus of this paper is to analysis why models perform poorly in temporal shift scenarios. We identified the absence of temporal information in model representations and improved performance by introducing our temporal embedding, thereby completing our argument.

Handling unknown or variable cycles will be the focus of future work, and we will include this discussion in the limitations section. Regarding the approach to learning unknown or variable cycles:

• A common approach is to reweight training samples [1], but such methods require an accurate validation set (instead of a validation set similar to the test time), which is difficult to obtain in temporal shift tasks.
• Another method is to introduce a matching mechanism between the training and test set distributions (e.g., attention), but this requires having a test distribution at test time, meaning multiple test samples must be obtained, which is closer to test-time adaptation [2].

Based on the MMD visualization and experimental results comparison presented in the paper, we believe the existing fixed cycles already effectively cover most scenarios (lines 379-384). We have also provided experimental results with variable cycles, as addressed in the next question.

Have you tried letting the model learn different frequencies if the known cycles are not relevant?

Yes, we also experimented with tuning hyperparameters to find the cycles, results in Table C in the repository. When using fixed prior periods, ModernNCA achieved a 0.30% performance improvement, while setting adjustable cycles resulted in a -2.48% performance decline.

In temporal distribution shift scenarios, due to the absence of an entirely accurate validation set, we believe that prior knowledge of fixed cycles is more stable and interpretable than adjustable cycles.

It's also important to note that in many tasks, complete cycles are not available. For example, in the weather dataset, there is a yearly cycle, but the training set does not span a full year, which highlights the importance of prior knowledge.

Are there practical guidelines for selecting these cycles or discovering new cycles automatically in domain-specific tasks?

For domain-specific tasks, we still recommend using prior cycles informed by expert knowledge or setting them based on MMD visualization.

Is it possible to cause data leakage problems?

This splitting method does not introduce data leakage, as no information that would be unavailable at deployment is used during training. Therefore, the model's performance on the test set remains reliable.

We hope this response addresses your concern. Please feel free to raise any further questions!

---

[1] Mengye Ren et al. Learning to Reweight Examples for Robust Deep Learning. ICML 2018: 4331-4340

[2] Changhun Kim et al. AdapTable: Test-Time Adaptation for Tabular Data via Shift-Aware Uncertainty Calibrator and Label Distribution Handler. CoRR abs/2407.10784

Thank you for your insightful feedback! We will address your concerns in the following responses.

The authors should be carefull ...

Our experimental setup strictly follows TabReD. Moreover, since this is a temporal scenario where each test sample is evaluated individually, the varying time gaps between each sample and training set reflect real-world conditions.

Additionally, this does not impact our training protocol, which only provides guidance stated in lines 307–316. Extensive experiments have validated the robustness of our training protocol, even when the training lag cannot be reduced to zero ((c) vs. (d) in Figure 3).

The newly proposed temporal split is primarily intended to illustrate the effectiveness of this protocol, reinforcing our focus on analyzing training lag and validation bias.

Temporal emb is not fully studied. Hardly helps SOTA.

This issue is already identified in our paper (lines 401, 410). Our temporal emb converts timestamps into numerical inputs, which may be incompatible with PLR emb. Specifically, once timestamps are embedded, their representation reflects temporal similarity. Applying another periodic transformation via PLR could increase optimization difficulty.

Directly feed the temporal emb into the model backbone consistently improves: MLP-PLR +0.01% → +0.26%, TabM +0.07% → +0.15%, MNCA +0.30% → +0.38%. Please refer to Table D in the repository: https://anonymous.4open.science/r/Tabular-Temporal-Shift-BCCA/.

How does new strategies affect model comparisons?

Please refer to our response to Reviewer 2MA7.

Fig 3 does not consider the confound effects ...

Our current experiments already addressed this issue. Our analysis of the loss distribution (Fig 4 right) isolates the impact of validation and test set variations while fixing the training set.

In splits (a,c), with the training set fixed earlier in time and shifts in validation and test sets, we observe in lines 234-251 the impact of reducing training lag and validation bias.

The Ours variant is not present for the (a,b,c,d) splits of equivalent size in fig 3.

The (a,b,c,d) splits are used only for analysis of training lag, validation bias, and equivalence. These splits were created by discarding data due to dataset limitations. In contrast, both the original and our temporal splits use the entire dataset before $T_{train}$.

Figure 6 done just for MLP.

We will add visualizations for SOTA model representations.

The result on HD excluded from all comparisons.

HD dataset was excluded only from the comparison in Fig 2. This is because Figure 2 calculates the improvement of other methods relative to MLP, and MLP performs poorly on the HD dataset. As a result, the relative improvement is significantly larger (~80%) compared to the average improvement on other datasets (<6%). Including this dataset in the mean obscures the result.

However, all other comparisons do not rely on relative improvements over MLP, so the HD dataset is included in all other results. Suggested by Reviewer 2MA7, we will adopt robust average to compute the mean improvement in the revision to better handle such cases.

TabReD suggest that MLPs without numerical embs ...

Here we are not referring to numerical embs during training, but feature encoding during preprocessing, such as noisy-quantile encoding. This will be clarified in the revision.

This aspect is not mentioned in the TabReD paper but can be inferred from the code: https://github.com/yandex-research/tabred/blob/main/exp/mlp/homecredit-default/tuning.toml#L54, which assigns noisy-quantile and ordinal for the MLP method on the HD dataset. TabReD assigns different encodings to different method-dataset pairs.

We believe this introduces fairness inconsistencies. Special encodings (e.g., noisy-quantile) do not always improve performance. Therefore, we used only basic encoding (none for numerical and one-hot for categorical features) for a fair comparison, despite lowering MLP’s performance on the HD dataset. This explains the difference between our results and those in the TabReD paper.

Did I understand Fig 2 correctly ?

Yes. We will further clarify this.

What is 1D PLR?

Sorry for this confusion. Our proposed temporal emb is specifically designed for the timestamp, we choose a PLR emb applied to the single timestamp input as a baseline.

Are SoTA models able to learn ...

Without timestamp information, models cannot learn the order and periodicity of samples. The challenge is incorporating temporal information. Fig 8 shows that treating timestamps as numerical features causes performance drop, aligning with the common practice of removing timestamps. Only by combining temporal emb with periodicity knowledge can temporal patterns be effectively incorporated.

Due to the limit, some responses may lack detail. Please feel free to raise any further concerns!