LLM-Guided Self-Supervised Tabular Learning With Task-Specific Pre-text Tasks

1. Limited novelty. The core idea is essentially using LLMs for feature engineering with prompt design, then using these features as self-supervision signals. Using LLMs as feature generators has been explored in many other fields - this paper simply applies it to tabular data.

2. At the methodology level, the framework is quite straightforward and lacks deep technical innovation. The feature selection and multi-task learning components use basic methods without proposing new improvements.

3. Lacks theoretical foundation. There's no analysis of why LLM-generated features help improve performance; no guarantees on feature quality; the whole method feels more like an empirical attempt.

• The proposed training method tailored for specific downstream task limits its application into broader spectrum.

• Lack of comparision against existing pretraining methods on tables (e.g. TUTA, TabLLM, XTab, TabPFN). As the model is trained with the specific objectives tailored for downstream task, the learned representation in itself demonstrates better performance against other representation learning methods. Adding comparision against other existing pretraining methods will provide a thorough examination.

• The small-scale training restricts the broader applicability of the proposed method. The training datasets are limited in comparison to existing self-supervised training approaches.

1. While the paper compares TST-LLM to several self-supervised learning methods, it overlooks some highly competitive supervised learning approaches (e.g., XGBoost [1], CatBoost [2], TabR [3], FT-Transformer [4], ModernNCA [5]), as well as general learning methods like TabPFN [6], Tp-Berta [7], and XTFormer [8]. These methods, along with LLM-based feature engineering techniques (CAAFE [9], OCTree [10], FeatLLM [11]), are already highly effective in tabular data tasks. A comparison to these methods would be necessary to justify the need for self-supervised learning with TST-LLM.

2. TST-LLM seems to mainly rely on features generated by LLM-based methods like CAAFE [9], OCTree [10], and FeatLLM[11] and applies self-supervised learning on them. This approach does not offer significant novelty in comparison to these existing methods, as it mainly repurposes the generated features for a different training objective.

3. The ablation study does not fully explore the potential of automatic feature engineering methods like OPENFE, which can generate a large number of features within the same time as LLM-based methods. A more fair comparison would involve using TST-LLM's feature selection approach on the most informative features generated by OPENFE, rather than randomly selecting features.

4. The method appears to be more effective on datasets with clear semantic relationships that can be easily articulated by LLMs. This limits the applicability of TST-LLM to domains where such semantic relationships are less obvious or harder to define, restricting its generalizability compared to other tabular data methods.

[1] Xgboost: A scalable tree boosting system

[2] CatBoost: unbiased boosting with categorical features

[3] TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023

[4] Revisiting Deep Learning Models for Tabular Data

[5] Modern Neighborhood Components Analysis: A Deep Tabular Baseline Two Decades Later

[6] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

[7] Making Pre-trained Language Models Great on Tabular Prediction

[8] Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data

[9] Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering

[10] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning

[11] Optimized feature generation for tabular data via llms with decision tree reasoning

• Lack of Novelty: The idea of tailoring pre-text tasks to specific downstream objectives has been previously explored in various frameworks, making the contributions appear incremental rather than groundbreaking. Additionally, the reliance on existing techniques such as supervised contrastive learning further diminish the uniqueness of the approach, as many prior works have already demonstrated similar strategies. Overall, the integration of known methods does not provide a significant advancement in the field, leading to questions about the paper's originality.

• Complexity of Implementation: The task-specific design may require more intricate setup and understanding compared to simpler, task-agnostic approaches.