Contemporary Continuous Aggregation: A Robust Categorical Encoding for Zero-Shot Transfer Learning on Tabular Data

Methodology

1. Moment Estimation for New Categorical Variables
   When introducing new categorical variables, it is essential to estimate the first, second, and higher-order moments (mean, standard deviation, minimum, and maximum) of all continuous variables associated with the new categorical variable. However, the paper seems to assume that the test set is provided offline in its entirety, calculating these moments accordingly. In an online setting, where the number of samples for newly introduced categorical variables is likely to be limited, this estimation could be highly inaccurate. The authors should provide a more detailed explanation of how their method can accurately estimate these moments in an online scenario where the test set is introduced sequentially and may be small.

2. Theoretical Analysis of Theorem 4.1
   While Theorem 4.1 presents a theoretical analysis, it is derived from prior work (Corso et al., 2020), offering no theoretical novelty. Additionally, the discussion lacks depth in addressing the practical implications of the theorem. The authors could enhance this section by discussing how their findings extend or build upon the existing literature.

3. Versatility of the Proposed Method
   The proposed method's applicability is limited in the absence of continuous columns. In the section titled Encode Correlation with Other Categorical Features, the authors suggest using ordinal encoding for categorical features. However, the mean, standard deviation, minimum, and maximum values become meaningless in this context, which undermines the validity of the proposed approach. An empirical verification should be conducted to assess the effectiveness of this method when applied in practice.

Experiments

1. Incremental Test Stream Evaluation
   The experiments should be conducted in an environment where the test stream is introduced incrementally and online. Furthermore, evaluating the method's performance in scenarios where target categories increase over time, as is typical in online settings, is crucial.

2. Evaluation Metrics
   The evaluation metric is currently limited to AUPRC. To strengthen the paper's claims, the authors should also report additional metrics such as AUROC, accuracy, and F1 Score.

3. Baseline Comparisons
   The number of baseline methods is insufficient. Even if scalability issues prevent certain baselines from being evaluated on larger datasets, the authors should have tested them on smaller datasets. Presenting the performance of methods like Ordinal Encoding or One-Hot Encoding, even if they perform poorly, would provide valuable context for the results.

4. Dataset Variety
   The number of datasets utilized in the experiments is too small. Current best practices suggest evaluating methods across nearly 100 datasets for comprehensive validation purposes.

5. Scope of Downstream Tasks
   The focus on binary classification (e.g., anomaly detection) is a limitation. The authors should consider testing the method on multi-class classification and regression tasks to showcase its versatility.

6. Validation on Neural Networks
   Although gradient-boosted decision trees (GBDTs) are commonly used in industry, it is important from a research perspective to validate the method on deep neural networks, particularly large language models (LLMs), to demonstrate broader applicability.

7. Marginal Improvement
   In Table 4, the proposed method shows only a 0.4% improvement over SDV, which is marginal. The authors should provide a more substantial analysis explaining the significance of this improvement.

8. Empirical Analysis of Performance Gains
   The paper lacks sufficient empirical analysis to explain why the proposed method outperforms others. For example, it does not clarify the specific reasons behind the performance gains achieved through the proposed categorical encoding method.

[1] Yan et al. "Making Pre-trained Language Models Great on Tabular Prediction." ICLR 2024.

• Poor Presentation and Clarity: Section 4, which contains the core methodology, is challenging to follow. The presentation of key definitions, such as the multiset encoding process, lacks clarity and could benefit from better-organized equations and explanations.

• Limited Original Theoretical Contribution: Although the paper claims to provide a theoretical basis for CCA, much of the theoretical groundwork appears to be based on the Principle Neighborhood Aggregation (PNA) framework from prior work. This reliance limits the novelty of the theoretical contributions claimed by the authors.

• Insufficient Experimental Validation: The experimental section lacks breadth in evaluating CCA across diverse types of tabular data, as only 14 datasets are used. Additionally, comparisons with baseline methods are limited, as various encoding methods (e.g., ordinal and one-hot encoding with an "unknown" class) are not tested.

• Lack of In-Depth Analysis of Encoding Properties: The paper would benefit from a more detailed analysis of the unseen category encoding itself, beyond simple performance comparisons. A deeper examination of the encoding's inherent properties and theoretical soundness would help validate the method’s utility and justify its application to unseen categories.

• Synthetic-Only Experiments for Unseen Categories: While unseen categories commonly occur in real-world datasets, experiments for zero-shot transfer were conducted only in synthetic environments, which limits the real-world applicability and validity of the results.

• While I agree with the authors that GBDTs are the best baseline to compare with, I believe results on deep learning algorithms can better contextualize the applicability of CCA. Because GBDTs naturally scale better to larger feature counts, I worry whether such performance gains are also applicable to deep learning algorithms, specifically given the rise of powerful ICL tabular learning algorithms[1,2,3,4,5].

• "Ordinal Encoding and One-hot Encoding are not considered since their encodings will generate out-of-distribution values on unseen categories, thus lacking extrapolation ability." What if you take the average ordinal encoding or the average one-hot encoding in such situations?

• The method is motivated by its ability to handle out-of-distribution data, but there is not any studies directly measuring CCA's scalability and sensitivity to out-of-distribution or NaN features.

• Is there any reason why you did not test on standard benchmarks such as [1]?

[1] When Do Neural Nets Outperform Boosted Trees on Tabular Data?

I am somewhat unfamiliar with the current state of the art in this sub-field, I yeild those concerns to other reviewers. I found the preliminaries/method section to be a bit difficult to parse, especially with the mixture of equations/relations and text, I would urge the authors to consider a more simplistic presence in the main paper and divert the more lower level technicalities into the supplementary. I additionally noticed the absence of figures throughout this manuscript. I personally believe that having a methodology figure could help better illustrate the contribution/motivation, I would recommend that the authors consider this entering the discussion phase -- a simple diagram could include an existing dataset, with an avenue for more incoming categories, and how CCA can expand it's encoding generalizability to support this setting. I think this would significantly help in the readability and understanding for a larger audience.