Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings

Thank you for the thoughtful and well-engaged review. We appreciate your interest in the core motivation behind our work, especially your questions around how the gating networks behave and whether the two encoders prefer different types of features. We're also glad you found the paper clear and technically solid, and we appreciate your recognition of the strong empirical results. We're happy to address the insightful points you raised.

W1: Evaluating Feature Selection in Tabular Data

We thank the reviewer for raising the important question of whether existing feature selection or elimination methods are sufficient for inducing complementary representations in tabular learning. We address this in two parts: by evaluating global feature selection with trees, and by comparing TANDEM (our method) to local masking-based self-supervised (SSL) approaches.

1. Global Feature Selection with Trees:

To explore the reviewer’s suggestion that partitioned feature sets could support effective ensemble, we evaluated whether applying explicit feature selection before training improves performance in a tree-based setup. Specifically, we trained a decision tree (DT) alongside a simple one-layer softmax model under two key settings: one using all input features, and one using only a subset selected by a SOTA NN-based model for feature selection (STG [1]). In both cases, ensembling improved performance over the tree alone. Notably, the variant with explicit FS yielded further gains, showing that even simple models benefit from selecting relevant features. While this setup achieved promising results competitive with strong classifiers like XGBoost and CatBoost, it still fell short of TANDEM.These findings support the reviewer’s intuition and highlight the value of combining diverse inductive biases through architectural integration.

Table 1: Accuracy of Decision Tree, FS-gated MLP, and their ensemble, along with Jaccard similarity between their top-10 selected features.

2. Evaluating Local Masking-Based Feature Selection Methods:

We aimed to evaluate how existing local feature selection methods in SSL compare to our approach. To this end, we considered two representative SSL methods that rely on masking: VIME, which applies noise to random subsets of features and trains the model to predict the corruption mask [2], and SubTab, which reconstructs inputs from masked subsets [3]. To include a stronger baseline that incorporates learned feature selection, we also evaluated a self-supervised autoencoder with feature selection (SS-AE with FS using STG [1]), which learns to suppress irrelevant features during pretraining.

As shown in Table 2, both VIME and SubTab underperform compared to TANDEM, demonstrating the limitations of random or task-agnostic masking used for local feature selection in SSL. In contrast, SS-AE with FS performs significantly better than the other local feature selection methods, highlighting the value of learned, sample-specific gating in self-supervised settings.

Still, TANDEM consistently outperforms all alternatives, suggesting that its combination of gating with diverse encoder types is key to learning strong, complementary representations.

Table 2: Mean accuracy and mean rank across datasets for VIME, SubTab, SS-AE with STG, and TANDEM.
All methods were evaluated on the same datasets and under the same evaluation protocol as used in the paper, with identical pretraining data. Due to space constraints, detailed results for all evaluations in this response will be included in the appendix.

W2 + Q3: Complementary gating behavior:

To further analyze how the neural and tree components differ in their learned feature preferences, we computed several metrics across four datasets with a moderate proportion of categorical features (Table 3 , further details on the datasets categorization are shown in the response to reviewer Meqn). This analysis focuses specifically on how the gating network of the decision tree and the gating network of the neural network differ in their feature selection behavior over categorical features. We report three interpretable metrics. Gate Overlap measures the proportion of categorical features that are simultaneously activated by both gating networks. Gate Correlation captures the alignment between the gate strength values assigned by the two networks using Pearson correlation. Gate Variance Ratio (Tree / NN) reflects how sharply each gating network modulates feature importance by comparing the variance in their gate strengths. We observe moderate agreement between the gating networks in terms of overlap and correlation. Furthermore, the tree’s gating network consistently shows higher variance, indicating it makes more decisive distinctions between categorical features. In contrast, the neural gating network applies smoother, more conservative gating. These differences support our core design goal in TANDEM: to encourage the two components to attend to complementary aspects of the input.

Table 3: Comparison of gating behavior between the neural and tree encoders, evaluated only on the categorical features of the datasets.

Q1: Feature relevance in OpenML datasets

We did not find official documentation labeling irrelevant or nuisance features in the OpenML datasets. Globally, most features appear task-relevant; however, we believe some features act as local nuisances, i.e., unhelpful for specific samples even if not globally irrelevant. This is supported by the behavior of our gating mechanism, which selectively downweights such features during inference. Similar observations have been made in prior work, where both global and local nuisance features were identified and mitigated([1], [4]).

Further analysis of feature quality is crucial to ensure these benchmarks remain both informative and realistic.

Q2: OSDT robustness to nuisance features

We refer to nuisance features as variables that may be irrelevant for the prediction task. For example, in genetics, when trying to predict a medical condition, often only a small subset of variables carries information, while the rest are nuisance features. In our current notation, we treat nuisance and irrelevant features interchangeably.

To clarify our claim that tree-based methods are more resilient to nuisance features, we refer to prior work [5], which shows that decision trees are inherently less sensitive to uninformative features due to their greedy, axis-aligned splits and implicit regularization. To further illustrate this property empirically, we injected 100 purely irrelevant Gaussian features into the CO dataset. As shown in Table 4, XGBoost’s performance remained stable, whereas the MLP experienced a significant drop in accuracy. This validates the notion that nuisance features can degrade the performance of NNs and underscores the robustness of tree-based approaches, like OSDT, in attenuating the influence of nuisance features.

Table 4: Impact of adding 100 Gaussian nuisance features to the CO dataset.

We appreciate the feedback and would be happy to further clarify any open issues.

[1] Yamada et al. Feature Selection using Stochastic Gates.

[2] Yoon et al. VIME: Extending the success of self- and semi-supervised learning to tabular domain.

[3] Ucar et al. SubTab: Subsetting features of tabular data for self-supervised representation learning.

[4] Yoon et al. INVASE: Instance-wise Variable Selection using Neural Networks

[5] Grinsztajnet al. Why do tree-based models still outperform deep learning on typical tabular data?

We thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of the modular and conceptually simple architecture, the interpretability and adaptivity enabled by the per-dimension gating mechanism, as well as the model’s ease of implementation and extensibility. We appreciate that the core contributions of integrating complementary local and global feature representations in a principled and interpretable way are viewed as valuable and technically sound.

We appreciate the reviewer’s concerns and address them through new regression experiments, further analysis of gating behavior, and clarification of the consistency loss. We also extend our evaluation to include additional SSL baselines such as VIME, SubTab, and Scarf. These additions broaden the empirical scope and reinforce the method's effectiveness and generalization.

W1: Architectural contribution - parallel fusion of neural networks and tree-based encoders

We appreciate the reviewer’s point regarding architectural simplicity. While the neural and tree-based components are individually standard, the core novelty lies in their parallel integration within the same representation space. Unlike prior works that combine trees and neural networks in a sequential manner , for example, by feeding the output of one into the other ([1]), our approach processes features independently through both encoders and then fuses the resulting representations directly. This parallel fusion enables the model to learn from the complementary inductive biases of trees and neural networks without one dominating or reshaping the other's input space.

W2 + Q1 + L1: Extension to Regression Tasks

To test the generality of TANDEM beyond classification, we evaluated it on 13 regression datasets using 400 samples per task, all from the benchmark from [2].

The results, shown in Table 1, demonstrate that while less decisive than in the classification evaluation, TANDEM also achieves the best average performance (lowest mean MSE and mean rank) compared to all baselines, We believe that results could be improved by using other losses for NN (as shown in [3]), and consider this an important area for future work.

Table 1: mean MSE (MMSE) and mean rank of various models across 13 regression datasets. Best results are bolded.

Due to space constraints, detailed results for all evaluations in this response will be included in the appendix.

W2 + Q3: Effect of consistency loss on representation diversity

We appreciate the concern regarding the potential collapse of diverse representations due to the consistency loss. To investigate this, we conducted a sensitivity analysis over a wide range of values for both components of the consistency term: 1) alignment loss weight and 2) latent representation similarity (LRS) loss weight. Results on five representative datasets are summarized below.

Table 2-3: Accuracy for varying consistency hyperparameter values in parentheses represents feature dim.

Table 2: Sensitivity to Alignment Loss Weight

Table 3: Sensitivity to latent representation similarity weight

These results show that across a wide range, the consistency terms either improve or preserve performance without collapsing the dual encoders. Only at excessively large values do we observe degradation due to dominance of one branch, but this is rare and easily avoided with light tuning.

W4 + L2: Clarifying the role of the gating mechanism

To provide a deeper analysis of the gating mechanism, we computed summary statistics across four datasets with a moderate proportion of categorical features (Table 4 , further details on the datasets categorization are shown in the response to reviewer Meqn) to capture how gate values fluctuate across both features and samples. Specifically, we report the mean and variance of the activations from the two gating networks, one connected to the neural encoder and the other to the decision tree.

As shown in Table 4, both gating networks exhibit relatively high variance, highlighting that gate values are not static but dynamically adapt per sample and feature. The tree-based gating shows higher variance, reflecting its more discrete, thresholded behavior, while the neural gating is smoother yet still responsive.

Table 4: Mean and variance of gate activations across samples and features.

W5 + Q3 + L2: Complementary gating behavior and its role in generalization

To examine how the complementary gating behavior of the neural and tree components contributes to generalization, we computed several metrics on the same datasets analyzed previously.

This analysis focuses specifically on how the gating network of the decision tree and the gating network of the neural network differ in their feature selection behavior over categorical features, potentially leading to improved generalization. We report three interpretable metrics. Gate Overlap measures the proportion of categorical features that are simultaneously activated by both gating networks. Gate Correlation captures the alignment between the gate strength values assigned by the two networks using Pearson correlation. Gate Variance Ratio (Tree / NN) reflects how sharply each gating network modulates feature importance by comparing the variance in their gate strengths.

While the two gating networks show moderate agreement across all datasets in terms of overlap and correlation, the tree’s gating network consistently shows higher variance, indicating it makes more decisive distinctions between categorical features. In contrast, the neural gating network applies smoother, more conservative gating. These differences support our core design goal in TANDEM: to encourage the two components to attend to complementary aspects of the input, which helps with generalization.

Table 5: Comparison of gating behavior between the neural and tree encoders, evaluated only on the categorical features of the datasets

W6 + Q4 + L3: Comparison with SSL baselines

We've added SCARF, VIME, and SubTab to our benchmark, evaluating them on the same 19 classification datasets with 400 labeled samples per class.

As shown in Table 6, TANDEM outperforms all three methods in both mean accuracy and mean rank across datasets.
This confirms that our hybrid approach with dual inductive biases provides stronger representations than existing SSL alternatives in low-label regimes.

Table 6: mean accuracy and mean rank per over all datasets.

We appreciate the feedback and would be happy to further clarify any open issues.

[1] Luo et al. NCART: Neural Classification and Regression Tree for Tabular Data.

[2] Grinsztajn et al . Why do tree-based models still outperform deep learning on typical tabular data?.

[3] Stewart et al. Building Bridges between Regression, Clustering, and Classification.

We thank the reviewer for their thoughtful assessment and constructive feedback. We're particularly grateful for your recognition of the key strengths of our work, including the overall clarity and motivation of TANDEM’s architecture, the effective integration of tree-based and neural inductive biases via dual encoders, and the use of stochastic gating networks for per-sample feature selection. We're also grateful for the acknowledgement of our spectral analysis as a convincing tool to demonstrate the representational differences between the encoders and for noting the consistent performance improvements across a broad set of classification benchmarks. These points reinforce the motivation behind our design choices and the validity of our experimental conclusions. The reviewer raises important points about evaluation scope, efficiency, and hyperparameter sensitivity. We appreciate the opportunity to address them below.

W1 + Q3: Hyperparameter sensitivity and stability

We fully agree that one of the core strengths of tree-based models is their strong out-of-the-box performance, and we designed TANDEM (our method) to retain this practical usability. We clarify that although TANDEM includes several configurable parameters (e.g., number of trees, tree depth), in practice we observed stability across a wide range of values (see table and further analysis on the loss weights in response to uEwZ). Furthermore, TANDEM does not utilize a mask ratio parameter, and we did not perform tuning on the temperature parameter, which was fixed to a default value throughout all experiments. This further minimizes the tuning burden in practice.

To empirically assess optimization robustness, we include a cost analysis of performance over increasing trial budgets. The following tables report the mean accuracy and standard deviation computed over a moving window of the last five trials at each budget size, across different random seeds and hyperparameter samples drawn using Optuna. This experiment is based on the reviewer’s suggestion.

As shown in Table 1 and Table 2, TANDEM demonstrates stable accuracy across multiple hyperparameter trials, indicating that the method is not highly sensitive to hyperparameter tuning. This result holds for both datasets, where TANDEM is the best, and for those where it is not.

Table 1: mean Accuracy Over Trials on CP dataset (window of last 5 trials)

Table 2: mean Accuracy Over Trials on HI dataset (window of last 5 trials)

W2 + Q1: Hyperparameter transparency and computational efficiency

We clarify that all hyperparameter search ranges for TANDEM and the baselines are provided in the Appendix (Table E.1), ensuring transparency and reproducibility. To assess practical feasibility, we provide two tables below. Table 3 shows the mean pretraining time across all datasets for 50 epochs and 2000 samples per label on an L4 GPU. TANDEM is faster than all self-supervised baselines evaluated, except for the self-supervised autoencoder (SS-AE), which happens because of the added OSDT encoder and gating net, which are both smaller than the SS-AE (and TANDEM’s) encoder-decoder network.

Table 3: Pretraining Time per Model

Table 4 presents downstream training time per batch (128 samples) on an L4 GPU, where TANDEM behaves like a lightweight MLP. TANDEM's downstream training cost is minimal, reducing to a simple MLP plus one-time usage of a gating network, making it highly scalable and practical for real-world deployment, even on CPUs (in contrast to TabPFN). Its performance gains and representational power justifies the modest training overhead (just slightly more than the MLP).

Table 4: Downstream Training Time per Batch

W3 + Q2 + Q4: Evaluation beyond classification and across labeling regimes

We thank the reviewer for pointing out the limited scope of the initial evaluation. Following this valuable feedback, we expanded our experiments in two significant directions:

1. Extension to regression tasks

To test the generality of TANDEM beyond classification, we evaluated it on 13 regression datasets using 400 labeled samples per task, all from the benchmark from [1].

The results, shown in Table 5, demonstrate that while less decisive than in the classification evaluation, TANDEM also achieves the best performance (lowest MSE and mean rank) compared to all baselines. We believe that results could be improved by using other losses for NN (as shown in [2]), and consider this an important area for future work.

Table 5: mean MSE (MMSE) and mean rank of various models across 13 regression datasets. Best results are bolded.

Due to space constraints, detailed results for all evaluations in this response will be included in the appendix.

2. Varying labeled sample sizes:

We conducted additional experiments ranging from 50 to 1000 labeled samples per dataset on the classification task. As shown in Table 6, TANDEM consistently ranks at the top or remains competitive across all sample sizes. TANDEM achieves state-of-the-art performance with fewer than 800 labeled samples, and remains competitive even beyond that scale.

Table 6: mean classification accuracy for each model trained on different numbers of labeled samples. Bold values indicate the best performance for that sample size.

These additions substantially strengthen the empirical foundation of the paper and highlight the significance of TANDEM’s contributions. The model consistently outperforms alternatives in both classification and regression tasks, across a wide range of supervision levels. While many public tabular datasets are large, real-world domains such as healthcare and finance often face limited labeled data. Our results demonstrate strong performance precisely in these challenging settings, underscoring the relevance and practical value of TANDEM. We thank the reviewer for raising this point, which prompted a more comprehensive and impactful evaluation.

W4: Feature correlation and reconstruction objectives

We agree that when features are weakly correlated, reconstruction-based pretraining may carry limited signal. However, in real-world tabular data, dependencies often exist between features. To demonstrate this, we computed the sum of absolute Pearson correlations between a feature and all other features (on numerical features only) within each dataset. In the table below, we present the mean of these values, which indicates substantial inter-feature dependencies in practice. We note that there may be additional nonlinear dependencies that have not been evaluated here.

Table 7: Mean and median of sum of absolute correlations per feature averaged across all datasets (numerical features only).

We hope this addresses all your concerns, and we would be happy to further clarify open issues.

[1] Grinsztajnet al. Why do tree-based models still outperform deep learning on typical tabular data?.

[2] Stewart et al. Building Bridges between Regression, Clustering, and Classification.

We thank the reviewer for their thoughtful assessment and constructive feedback. We are particularly grateful for your recognition of TANDEM’s “genuinely novel architectural contribution” and its “paradigm shift” in self-supervision. Your appreciation of our well-motivated design, including the alignment and latent similarity losses, as well as the spectral analysis that validates the complementary inductive biases, is deeply encouraging. We also value your acknowledgment of the vital challenge our work addresses by bridging the gap between neural and tree-based models in tabular learning.

We found your comments and questions extremely helpful, and we address each of them below, providing additional experiments, clarifications, and analyses.

W1 + W3 + Q1: Experimental validation and scope

We thank the reviewer and want to clarify that our method’s contributions extend beyond specific dataset sizes or label counts. Nonetheless, we have significantly expanded the experiments in three key directions:

1. Recent SSL baselines:
   We added SCARF, VIME, and SubTab to our benchmark, evaluating them on the same 19 classification datasets with 400 labeled samples per class.
   As shown in Table 1, TANDEM outperforms all three methods in both mean accuracy and mean rank across datasets.
   This confirms that our hybrid approach with dual inductive biases provides stronger representations than existing SSL alternatives in low-label regimes.

Table 1: mean accuracy and mean rank per over all datasets

Due to space constraints, detailed results for all evaluations in this response will be included in the appendix.

2. Extension to regression tasks:
   To test the generality of TANDEM beyond classification, we evaluated it on 13 regression datasets using 400 samples per task, all from the benchmark from [1].
   The results, shown in Table 2, demonstrate that TANDEM also achieves the best average performance (lowest mean MSE and mean rank) compared to all baselines. We believe that results could be improved by using other losses for NN (as shown in [2]), and consider this an important area for future work.

Table 2: mean MSE (MMSE) and mean rank of various models across 13 regression datasets. Best results are bolded.

3. Varying the number of labels:
   We conducted additional experiments ranging from 50 to 1000 labeled samples per dataset on the classification task. As shown in Table 3, TANDEM consistently ranks at the top or remains competitive across all sample sizes.

Table 3: mean classification accuracy for each model trained on different numbers of labeled samples. Bold values indicate the best performance for that sample size (we report only the top 5 models here).

These additions substantially strengthen the empirical foundation of the paper and highlight the significance of TANDEM’s contributions. The model consistently outperforms alternatives in both classification and regression tasks, across a wide range of supervision levels. While many public tabular datasets are large, real-world domains such as healthcare and finance often face a shortage of labeled data. Our results demonstrate strong performance precisely in these challenging settings, underscoring TANDEM’s relevance and practical value (see e.g., [3],[4]). We thank the reviewer for raising this point, which prompted a more comprehensive and impactful evaluation.

W1 + Q3: Statistical significance comparison

Table 4: how many datasets (out of 19; 18 for TabPFN) TANDEM outperforms each baseline. Parentheses indicate statistically significant wins (p < 0.05, 100 trials).

W2 + Q2: TabNet baseline results

We understand the reviewer’s concern regarding the TabNet results. We used the pytorch-tabnet package and tuned its full pipeline (pretraining + classification/regression) via Optuna. In addition, we reproduced the official TabNet implementation from GitHub, including its embedding tuning. Both setups were tuned independently; the best achieved ~0.57 mean accuracy, which is below the performance of TANDEM. Prior work [5] also reported suboptimal TabNet performance on tabular data, including datasets from our benchmark, likely due to its sensitivity to tuning and limited handling of categorical features. We thank the reviewer for raising this concern. All implementation and optimization details appear in the appendix.

W2: Justification for hierarchical vs. shared Gating

In the tree encoder, each layer operates directly over the original input feature space. Because of this, gating must be applied at every depth, to determine which features are relevant for that layer’s decisions. Moreover, to preserve the oblivious structure where layers operate independently, we use a separate gating network for each layer. Each of these gating networks is tailored specifically to its corresponding layer, making the overall mechanism inherently hierarchical.

In contrast, the neural encoder applies gating once at the input, since subsequent layers operate sequentially on the resulting feature embeddings.

Q4: Computational cost and scalability

During inference, TANDEM is applied as a fully connected MLP, making it highly efficient and lightweight, suitable even for home CPU environments. Pretraining is under 2× the cost of the self-supervised Autoencoder, with the primary overhead from the tree encoder, which scales modestly with depth. Despite this, TANDEM remains significantly faster than SubTab, VIME, TabNet, and SCARF, both in training and inference.

Q5: Categorical feature ratio breakdown

To better understand where TANDEM excels, we grouped the datasets by their categorical feature ratio.

High (≥ 0.70): PW (1.00), BM (0.92), CO (0.85), AD (0.87)
Medium (0.30–0.69): RS (0.69), ALB (0.39), EY (0.33), CC (0.33)
Low (< 0.30): OG (0.10), CP (0.27), VO (0.18), EL (0.12), HI, HE, JA, NU, CA (0.00)

This characterization highlights that TANDEM performs particularly well on datasets with a moderate share of categorical features and also shows strong results on entirely numeric datasets with low-cardinality continuous features, which often behave like categorical inputs. Studying this effect further is an interesting direction for future work.

L4: Sensitivity analysis of TANDEM hyperparameters

We conducted a sensitivity analysis over five datasets, examining various ranges of loss weights, gating hidden dimension, tree depth, and number of trees. TANDEM showed stable performance across a wide range of values. Optimization trials using Optuna on two datasets (one where TANDEM excels and one where it does not) demonstrated stable performance. Additional results will be included in the appendix. The alignment and latent representation similarity loss weights sensitivity analysis is discussed in the response to reviewer uEwZ, and the optimization behavior is addressed in response to reviewer km3R. (Omitted here due to space constraints).

We hope this addresses all your concerns, and we would be happy to further clarify open issues.

[1] Grinsztajn et al. Why do tree-based models still outperform deep learning on typical tabular data?

[2] Stewar et al. Building Bridges between Regression, Clustering, and Classification.

[3] Arazi at al. TabSTAR: A Foundation Tabular Model With Semantically Target-Aware Representations.

[4] Hollmann et al. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second.

[5] Gorishniy et al Revisiting deep 318 learning models for tabular data.