Range-limited Augmentation for Few-shot Learning in Tabular Data

[Lack of theoretical analysis] Thank you for raising this important point. While a theoretical explanation would indeed strengthen the contribution of our work, defining task-relevant information or assuming that a specific augmentation preserves semantic information is far from trivial in tabular domains, unlike in image or text data. As mentioned in the manuscript (L47-53), assessing task labels for tabular data is challenging and often infeasible without domain expertise.

Instead of relying on potentially unrealistic definitions or assumptions about ground-truth labeling functions, we adopted two metrics — usable information and representation invariance score — to evaluate the preservation of semantic information, using a reliable proxy for the labeling function. In the revision, we will add theoretical implications derived from our empirical analysis in Section 4.1, including the relationships between these metrics and how they reflect the preservation of semantic information. As referenced in prior studies, these metrics provide valuable insights into the efficacy of augmentations in retaining task-relevant information.

[Lack of confusion matrix based metrics] As noted in the manuscript (L211), we evaluate not only accuracy but also AUROC and log loss in our benchmark, with results provided in the supplementary materials. Specifically, our method achieved an average AUROC of 0.653 in the 1-shot setup and 0.734 in the 5-shot setup, consistently outperforming other approaches. We appreciate the reviewer’s suggestion to include additional results, such as confusion matrices, to provide a more comprehensive evaluation of our method’s effectiveness. We will incorporate these results into the revised manuscript to enhance clarity and support our findings.

[Typos] Thank you for your feedback. We will carefully proofread the manuscript to address any remaining typographical errors and improve the overall presentation. If the reviewer has identified specific issues, we would greatly appreciate further guidance to ensure these are thoroughly corrected.

[Computational overhead] Thank you for raising this issue. As noted in the manuscript (L259) and Section 6.2, our method introduces computational overhead due to the need for sampling or shuffling within individual ranges. However, as summarized in Supplementary C.2, when comparing fitting time per epoch (excluding the impact of early stopping), our method exhibits lower training time than transformer-based approaches like SAINT, as it relies on a simple MLP architecture. Despite this simplicity, our augmentation method consistently enhances few-shot learning performance across diverse datasets.

We acknowledge that the computational overhead could present challenges for practical applications. As a potential optimization, leveraging the distributional properties of the ranges (which contain an equal number of samples) could allow precomputing shuffled or sampled values for each range, reducing runtime by reusing indexed augmented samples. We will explore and implement this optimization in the provided code to improve efficiency further.

[Benchmark code availability] Absolutely. The reproducible benchmark is already included in the supplementary materials and will be made publicly available through a GitHub repository following publication.

[Future research plans] Thank you for your interest in our research. Augmentations are not only essential for contrastive learning but can also serve as powerful tools for supervised learning and data amplification in scarce data scenarios, such as anomaly detection. Building on our findings that range-limited augmentations effectively preserve semantic information in tabular datasets and enhance few-shot learning performance, we plan to extend this approach to other applications, including supervised learning tasks and anomaly detection systems.

[Generalizability to other data domains] We respectfully disagree with the reviewer’s assessment. In recent years, a substantial number of studies have been dedicated to tabular learning, underscoring its importance in real-world applications. Tabular data is ubiquitous across domains such as healthcare, finance, and education, where high-stakes decisions rely on robust learning models.

Unlike image or text data, tabular data presents unique challenges, including heterogeneous feature types (numerical and categorical) and lack of spatial or sequential structure. Consequently, standard augmentation methods from other domains, such as random rotation or flipping in images, are not directly applicable to tabular data. Our approach addresses these challenges with a tailored solution that effectively preserves task-relevant semantics within tabular datasets. We strongly believe that tabular-data-specific approaches are neither narrow nor trivial but rather essential to advancing machine learning in this domain.

[Clarity issues in problem introduction] Thank you for raising this point. While we believe the current manuscript provides sufficient context to understand our target problem, we agree that additional clarity on the unique challenges of tabular data would be beneficial. In the revision, we will include a detailed discussion of the tabular-specific problem setup, including the diverse feature dimensionality, mixed feature types (categorical and numerical), and the lack of domain-specific inductive biases like spatial or temporal structure. We hope these revisions will further enhance the accessibility and clarity of our work for all readers.

[Limited novelty compared to SimCLR)] We respectfully disagree and believe the reviewer may have misunderstood our method. As noted in our official comment, other reviewers recognized the novelty of our range-limited augmentation technique, which preserves task-relevant information within feature-specific ranges — a critical requirement for tabular data.

To clarify the misunderstanding, let us address the specific examples provided in the review. The color jittering in SimCLR is similar to the sampling method proposed in SCARF, while our range-limited sampling applies sampling within feature-specific ranges, tailored to tabular data. Similarly, random cropping in SimCLR resembles the subset method used in SubTab but is entirely different from our approach. Our range-limited shuffling is unique to tabular domains, where values for a given feature are shuffled only within predefined ranges, preserving semantic consistency. While shuffling might loosely resemble permutation augmentation in image domains, it is fundamentally adapted for the unique structure of tabular data. As noted in the manuscript, the novelty of our method lies in introducing range constraints to well-known augmentation techniques, significantly enhancing semantic preservation in tabular datasets.

We hope this explanation clarifies the novelty of our augmentation method and its contribution to contrastive learning in tabular domains.

[Writing quality] We respectfully disagree with the reviewer’s point. One of the main contributions of our study is the introduction of FESTA, a new benchmark for few-shot learning in tabular domains. To establish the novelty of this benchmark, it is essential to explain the datasets and algorithms included and to provide a rigorous comparison. Furthermore, as there has been no prior systematic investigation of few-shot learning in tabular domains, detailing the experimental setup is crucial to ensure reproducibility and clarity. Other reviewers have noted that the paper is well-organized, with clear problem definitions, methods, and extensive evaluations across diverse datasets, making the contributions easy to follow.

If the reviewer identifies specific sections as unnecessary or unclear, we would appreciate further clarification and will gladly revise or condense these sections as needed.

[Use of GBDT methods] Thank you for raising this question. While it is true that gradient-boosted decision trees (GBDTs) are not inherently few-shot learning algorithms, they are widely recognized for their strong performance in tabular data tasks, often surpassing deep neural networks in many setups. Including GBDT methods in our evaluation allows for a fair comparison and aligns with practices in prior studies, such as STUNT, which used similar setups.

In our experiments, we followed the same task setups as those used in previous works, including STUNT, and included detailed configuration descriptions in the supplementary material. Additionally, the provided code includes all configurations and executable scripts to ensure reproducibility. If further explanation is needed, we can expand the supplementary material to describe how these methods were adapted for few-shot scenarios.

[Adaptation to categorical features] Thank you for your comment. As noted in the manuscript (L483), the current method is specifically designed for numerical features, and its extension to categorical data remains a limitation. Categorical features typically have limited discrete values, which can lead to unintended semantic changes during augmentation. (In this study, as outlined in L182, we followed the definition from [Lee et al., 2024b] to classify categorical features as those with fewer than 20 unique values.) While range-limited augmentations for numerical features, such as height (e.g., 140–160 cm, 161–180 cm), preserve task-relevant semantics, similar operations on categorical features like nationality may result in unrealistic or misleading variations. Therefore, we intentionally excluded categorical features from augmentation and will clarify this design choice in the revised manuscript.

[Limited novelty] We understand the reviewer’s concern regarding the perceived novelty of our range-limited augmentation method, as it builds upon existing sampling and shuffling techniques. However, as other reviewers (NQQr, qfF3) have noted, our approach introduces a significant advancement in contrastive learning for tabular data by preserving task-relevant information more effectively through range-limited augmentation. While the concept may seem straightforward, its impact is far from negligible, as evidenced by its consistent performance improvements.

Moreover, we believe our work provides meaningful contributions beyond the augmentation method itself: (1) introducing a novel augmentation technique tailored specifically to tabular data, (2) presenting FESTA, the first comprehensive benchmark for few-shot tabular learning, and (3) providing an empirical analysis of semantic preservation in different augmentation methods. Together, these contributions advance the understanding and practical capabilities of few-shot learning in tabular domains.

[Importance of few-shot learning tasks] We acknowledge the importance of full-shot tabular learning, which has been extensively investigated in prior benchmark studies, including WhyTrees [1], OpenML, TabZilla [D McElfresh et al., 2023], and TabRepo [D Salinas et al., 2023], where tree-based models often dominate on average performance. These benchmarks provide a solid foundation for evaluating full-shot learning, and substantial progress has already been made in this area.

In contrast, few-shot learning in tabular data remains underexplored, despite its critical importance in domains constrained by privacy limitations and the high cost of labeling, which often lead to data scarcity. As noted by other reviewers (ksSf, NQQr), few-shot learning directly addresses these challenges, making it an essential and practical research topic in tabular data applications.

Additionally, we have already presented results for a range of shot settings in Section 6.1, showing that our method consistently achieves robust performance gains across various numbers of shots. This underscores the practical value and adaptability of our approach to real-world scenarios with limited labeled data.

[1] Léo Grinsztajn et al., Why do tree-based models still outperform deep learning on tabular data?, NeurIPS, 2022.

[LLM-based baselines] We agree that LLM-based methods hold significant potential for few-shot tabular learning and have cited relevant studies in Section 2 (L127). However, as explained in the manuscript (L129), our study intentionally focuses on approaches that do not rely on auxiliary inputs, such as column names or descriptions. These inputs are often incomplete, inconsistent, or unavailable in real-world scenarios, which limits the practicality of LLM-based approaches in certain domains.

By excluding methods that depend on external metadata, we aim to provide a fairer comparison among approaches that operate solely on the data itself. Nonetheless, we recognize the potential for integrating range-limited augmentation with LLM-based methods and will explore this direction in future work.

[Suggesting a new SSL framework] Thank you for your suggestion. As noted in the manuscript (L256), we follow the contrastive learning framework proposed in SCARF. This deliberate choice was made to isolate and highlight the impact of preserving semantic information through augmentation methods and to demonstrate how this preservation contributes to improving few-shot learning performance. By keeping the overall framework consistent, we aim to emphasize the novelty and effectiveness of our range-limited augmentation approach.

That said, we acknowledge the value of developing a new learning framework to further leverage the proposed augmentation method. If the reviewer feels that the current focus on augmentation limits the perceived novelty, we will consider extending the framework as part of our future work.

[Adaptation to categorical features] Thank you for your insightful comment. As noted in the manuscript (L483), the current method is specifically designed for numerical features, and its extension to categorical data remains a limitation. Categorical features typically have limited discrete values, which can lead to unintended semantic changes during augmentation. (In this study, as outlined in L182, we followed the definition from [Lee et al., 2024b] to classify categorical features as those with fewer than 20 unique values.) While range-limited augmentations for numerical features, such as height (e.g., 140–160 cm, 161–180 cm), preserve task-relevant semantics, similar operations on categorical features like nationality may result in unrealistic or misleading variations. Therefore, we intentionally excluded categorical features from augmentation and will clarify this design choice in the revised manuscript.

Regarding homogeneous datasets, our method is applicable to datasets comprising only numerical features. In the FESTA benchmark, we identified 16 datasets (IDs: 22, 1475, 1489, 1492, 1497, 4153, 40499, 44125, 44131, 1462, 44126, 41168, 44091, 44123, 44090, 40922) that fall into this category. Across these datasets, our method consistently outperformed competing approaches, achieving an average accuracy of 0.534 with average rank of 2.125 in the 1-shot setup (second-best: MLP — Accuracy 0.504, Rank 9.813), and 0.656 with average rank of 2.625 in the 5-shot setup (second-best: MLP — Accuracy 0.640, Rank 6.938).

[Evaluation on other benchmarks] We appreciate this valuable suggestion. To the best of our knowledge, there has been no large-scale benchmark explicitly designed for few-shot learning in tabular data, despite its importance in domains where privacy and cost constraints often limit data availability. To address this gap, we introduced FESTA, the first comprehensive benchmark tailored to this scenario, comprising 42 dataset systematically selected based on criteria summarized in Section 3.2.

While OpenML-CC18 is a well-known collection, it is not suitable for evaluating few-shot learning algorithms due to its inclusion of datasets that violate some necessary conditions. For instance, some datasets have fewer samples per class than the number of shots (5 in our setup), while others correspond to flattened image datasets, such as MNIST or CIFAR-10, which are not representative of the unique characteristics of tabular data. To address these limitations of existing tabular benchmarks, we built the FESTA benchmark to evaluate few-shot learning algorithms in tabular domains while enhancing generalizability.

That said, we acknowledge the importance of benchmark diversity in enhancing generalizability. To address this concern, we conducted additional experiments on 29 tabular datasets from OpenML-CC18 that meet our selection criteria. Our method (CL+BinShuffling) consistently demonstrated superior performance, achieving an average 1-shot accuracy of 0.523 and AUROC of 0.723, compared to the second-best methods, CL+Shuffling (accuracy: 0.518) and SCARF (AUROC: 0.716). We will incorporate these results into the revised manuscript and include additional datasets, such as credit-g, to further validate the generalizability of our findings.

[Application to regression tasks] As detailed in Section 6.3, we have already extended our analysis to include few-shot regression tasks and demonstrated that the proposed method performs consistently well. In specific, our range-limited augmentation methods (shuffling and sampling) achieved superior performance compared to strong baselines like XGBoost and MLP across multiple regression datasets.