D2R2: Diffusion-based Representation with Random Distance Matching for Tabular Few-shot Learning

Thanks for your comments!

Q1.

We plan to add more explanations about shortcomings of existing methods as follows.

Firstly, tabular data comprises heterologous features, which underscores the importance of simultaneously modeling continuous and categorical features. The current SOTA method STUNT[28] in our paper, and other existing few-shot learning methods designed for images or texts, treat all features as the same type, neglecting the unique information in different features. Failing to address heterogeneous features hinders the model's ability to capture complex data patterns and relationships. Moreover, the heterogeneous features make tabular data lacking in straightforward augmentation techniques, which are easily implemented in images, as demonstrated by UMTRA[21].

Secondly, unlike images and texts, tabular data lacks strong spatial and sequential relationships between features. While STUNT[28] generates pseudo-labels based on the assumption that substitutable features, such as "occupation" acting as a proxy for "income", may not universally apply to tabular data. Besides, the arbitrary permutation of columns in tabular data highlights the model's robustness, which is ignored by existing methods.

Thirdly, the influence of column permutation and multimodal behavior within same class are all ignored by existing methods.

Q2.

We will add more explanations about motivations as follows.

Existing methods failing to address issues in Q1. thus we propose a novel approach, D2R2, to tackle above challenges in tabular data. Considering the first and second challenges above, we avoid to rely on proxy methods or augmentation methods, which are limited by the lack of relationships or augmentation techniques in tabular data. Instead, we create an information bottleneck for extracting semantic knowledge, named D2R2, which leverages the strong expressiveness of the diffusion model and distance information from pairwise comparison. Notably, we handle numerical and categorical data types separately. For example, we introduce two distinct noises and generate two distinct random projections for different feature types. Adding small noise to the input also enhances the model's robustness.

For the third challenge in Q1, we propose an instance-wise iteration prototypes classifier, which can construct stable prototypes while revealing the multimodal behavior within same class.

Q3.

Based on the unique characteristics of tabular data, as detailed in Q4, we address scenarios where there is an unlabeled training set and a very limited labeled support set (K-shot) to help predict class labels for a testing query set. This follows the problem definition in [28] and is stated in Section 3 of our paper. Such scenarios are common in critical applications like credit risk assessment[1] and diagnosing patients with rare or new diseases[2]. Based on our research, three types of methods can address these scenarios: few-shot learning, semi-supervised learning, and self-supervised learning. Meta-learning is one of the most common techniques in few-shot learning. The SOTA method [28] for few-shot tabular learning in our paper follows this scheme. On the other hand, our method innovates within the prototype scheme by enhancing embedding generation and prototype classification. In semi-supervised learning, a model is trained on labeled data then to predict the unlabeled data, where predicted labels are treated as true labels in further training rounds. This method aligns with our problem definition; however, it requires more labeled data compared to few-shot learning. Self-supervised learning excels in generating robust representations. Since our approach involves generating representations, we compare our method with the SOTA self-supervised learning methods[54,4,47] in the tabular domain. The mentioned methods serve to position our work within the broader context of existing methodologies that address similar problems. We have already outlined the relevant methods in Section 2. We will include the above more detailed discussion in the updated version.

[1]Chen N, Ribeiro B, Chen A. Financial credit risk assessment: a recent review[J]. Artificial Intelligence Review, 2016, 45: 1-23. [2]Cereda D, Tirani M, Rovida F, et al. The early phase of the COVID-19 outbreak in Lombardy, Italy[J]. 2003.09320, 2020.

Q4.

Our problem setting follows the definition provided in [28], where there is no variation of categories between the training (base) and test (novel) datasets, but the training set is unlabeled, which is a more common scenario in tabular data. Tabular datasets consist of numerical or categorical inputs that have specific and explicit meanings, where classes are well-defined and consistent across the dataset. it is rare to encounter new categories based solely on existing features. New categories usually emerge only with the introduction of new features, which is different from domains like image or text data, which have spatial and sequential relationships. For example, new image categories can be effectively classified based on the learned spatial structure patterns between pixels. Our study focus on the most common scenario in tabular few-shot learning, as mentioned in Section 3 of our paper and supported by the problem setting in [28].

However, we also conducted experiments to test the generalization of our method regarding the variation of categories in the training (base) and test (novel) datasets. For the dataset for 10-class classification 'optdigit', we randomly removed one classes from the train set and validated the effectiveness of the test results. The experimental results are as follows, demonstrating the robustness and effectiveness of our method.

Thanks for your comments!

Q1. What's the relationship and key novelty in the method compared to existing methods? Existing papers about representation learnings are all designed for image data. Paper [1] introduces a "latent Denoising Autoencoder" (LDAE) architecture where the learned representations are used for Denoising Autoencoder. Unlike LDAE, we learn a latent z that supports image diffusion process, rather than Autoencoder reconstruction process, which is less suitable for tabular data as discussed in [4]. The success of the representations in paper [2] depends on the quality of the generated images, meaning accurate image generation is crucial for performance. Generating high-quality data is especially difficult for tabular data due to its varied features and the absence of strong spatial or sequential relationships [5]. Our approach, however, does not rely on data quality for results, making our model more effective with tabular data. Paper [3] extracts representations from noisy data at a specific time step t in the diffusion process, using a student model trained with task labels. Our method differs by deriving representations directly from the original feature space without requiring a student model, eliminating the need for labels during training. This is especially critical for tabular few-shot learning where labeled data is scarce.

Moreover, different from existing papers, our work effectively modify diffusion models according to tabular data characteristics, including heterogeneous features, lack of strong spatial and sequential relationships between features, the influence of column permutation, and multimodality within the same class. We design a conditional diffusion process with distance matching to extract representations from original space, which is not influenced by the new generated samples' quality. Besides, tabular columns can be arbitrarily permuted without altering the underlying information, thus models for tabular data need to be robust to such permutations. We perturbed original data with two distinct types of relatively small noises for numerical and categorical input respectively (Section 4.1), which enhanced the robustness of the performance. Moreover, deriving representations solely from the "diffusion-only" model is not effective enough, as shown in our manuscript's Ablation study (Table 2). Therefore, we align the comparison of pairwise distances between the embedding space and a randomly projected space with the diffusion training process, using different projection matrices for various feature types. Additionally, we propose instance-wise iterative prototype to address multimodality for tabular data.

[4]Nam J, Tack J, Lee K, et al. Stunt: Few-shot tabular learning with self-generated tasks from unlabeled tables[J]. arXiv preprint arXiv:2303.00918, 2023.

[5]Xu L, Skoularidou M, Cuesta-Infante A, et al. Modeling tabular data using conditional gan[J]. Advances in neural information processing systems, 2019, 32.

Q2. Will other noise levels be more useful for few-shot learning? Or will a combination of different noise levels be better?

The timestep t = 1, ..., T indicates noise levels in diffusion models. In our experimental analysis, we have noted a trend where performance sees an improvement with an increase in t during the initial stages. However, this trend of performance enhancement plateaus after the first few timesteps, which tends to stabilize with minimal fluctuations, as shown in the following table. We speculate that this is because, as t approaches 0, the noise levels decrease, potentially causing only the fine-grained details to be lost. Hence, the representation learns to keep the information needed to recover these fine-grained details. On the contrary, as t approaches T, noise levels rise, and the mutual information between xt and x0 begins to decrease. In these situations, effective denoising requires that all information about x0 be thoroughly encoded, which contains the most information about the data class. Thus, as stated in line 200 of our manuscript, "We focus on larger timesteps thus extract the low-frequency semantic information rather than details." In our experiments, we selected the last step (T) for all datasets to ensure stable performance. We will include a discussion on this observation in the updated version of our manuscript.

In the updated version, we will include the above discussion following your valuable comments.

Thanks for your comments!

Q1. The paper lacks a clear description of the input format for tabular data.

In our study, "tabular data" refers to the dataset organized in tables, which is a structured format that presents information in rows and columns. It is defined as $D = \{\boldsymbol{x} _ i\} _ {i=1}^n\subset \mathcal{R}^{d}$ consisting of $n$ instances (rows) and $d$ dimensional features (columns). Each data instance $\boldsymbol{x} _ i = (x_i^1, x_i^2, ... , x_i^d)$ may or may not hold strong relationship among features. The feature types of tabular data can be a mixtures of numerical numbers or categorical indicators. For example, the "income" dataset classifies individuals' incomes based on features like "age" (numerical)，"education" (categorical) and so on. Table 3 in Appendix provides a detailed information on each dataset.

Tabular data is widely used in practical applications, and few-shot learning is particularly important in the context of tabular data, as limited labeled tabular data is inherently common in many real-world applications, such as financial fraud detection [1], disease diagnosis [2], and social science [3]. Different from CV and NLP, tabular data contains a mixture of numerical and categorical features; tabular data lacks spatial and sequential relationships between columns, making it more challenging to extract semantic knowledge; tabular column order can be arbitrary permuted without affecting the tabular information; tabular embeddings within the same class may exhibit multiple modes, as demonstrated in Figure 3 of our manuscript. The significance and challenges of tabular few-shot learning motivate us to conduct this work.

Q2. Whether the diffusion model is effective across all types of tabular data?

From the experimental analysis, our datasets (seven in the paper and two additional datasets in the response to Review wij1's Q1) involve various types of tabular data. These include varying sample sizes (from 768 to 48842), feature dimensions (from 8 to 24482), feature types (numerical, categorical or the mixture of both), and number of classes (from 2 to 10). Our experimental results demonstrate the effectiveness of our proposed method and its scalability across various types of those tabular data.

From a theoretical analysis perspective, reasons that the designed diffusion model can extract semantic knowledge come from twofold. Firstly, the diffusion model with powerful expressiveness encodes the information needed for denoising. Specifically, in conditional diffusion models, the noise reconstruction loss $\mathbf{E} _ {t, \mathbf{x} _ 0, \mathbf{x} _ t} ||\epsilon_\phi(\mathbf{x} _ t, t,c) - \epsilon||^2_2$ trains the noise prediction function $\epsilon_\phi(\mathbf{x} _ t, t,c)$ to predict the true noise $\epsilon(\mathbf{x} _ t, t, \mathbf{x} _ 0)$ given the noisy sample $\mathbf{x} _ t$, the knowing $t$ and the condition information $c$. If $c = \mathbf{x} _ 0$, we could expect $\epsilon_\phi$ can almost perfectly recover $\epsilon$. By replacing the conditional information $c$ by a function $z_\theta(\mathbf{x} _ 0)$ that maps to a embedding space with lower dimension than $\mathbf{x} _ 0$, we introduce an information bottleneck to the noise reconstruction process. This forces $z_\theta$ to extract effective information for denoising from $\mathbf{x} _ 0$, leading to representation learning through the noise reconstruction loss.

Q3. what is model's computational costs？

The model complexity is manageable through our framework's utilization of two neural networks: the embedding network and the noise prediction network, both of which are demonstrated as 3-layer MLPs in our paper. These settings (hidden dimensions, embedding dimensions and model structure) can be adjusted to meet different needs. In terms of computational efficiency, The main time-consuming factor after the embedding model is the calculation of the instance-wise iterative prototype. For an N-way K-shot problem with L iterations and a query set of size Q, the computation complexity is O(NKLQ). In few-shot learning, where N, K, L are small, the computation complexity is linear in the size of the query set. In the updated version, we will include the above discussion following your valuable comments.

[1]West J, Bhattacharya M. Intelligent financial fraud detection: a comprehensive review[J]. Computers & security, 2016, 57: 47-66.

[2]Kim S J, Choi S J, Jang J S, et al. Innovative nanosensor for disease diagnosis[J]. Accounts of Chemical Research, 2017, 50(7): 1587-1596.

[3]Hicks D. The four literatures of social science[J]. Handbook of quantitative science and technology research: The use of publication and patent statistics in studies of S&T systems, 2004: 473-496.

Thanks for your comments!

Q1. Does the proposed method scale with larger datasets or those with higher feature dimensions?

In our experiments, we utilized seven datasets that are widely used in tabular data research, as referenced in popular papers [28], [2], and [54] in our paper. Among these, the "income" dataset has the most samples (48842), and the "pixel" dataset has the highest feature dimension (240).

To further investigate the scalability of our method, we added two new datasets to our experiments. The "nomao" dataset has a larger size of 34465 samples, and the "breast" dataset has a significantly higher feature dimension of 24482. The following table shows their results. Our method consistently outperforms other baselines. These results will be included in the updated version of our paper, demonstrating that our method scales well to larger and high dimensional datasets.

Q2. Could the authors provide more empirical evidence or analysis on the pseudo-label validation scheme？

In the few-shot learning scenario, particularly in 1-shot classification, there are no labeled samples available for validation, which leads most researchers to rely on fixed parameters. However, we argue that a better set of hyper-parameters can always be identified for a specific dataset. To this end, we generate pseudo-labels for a validation set using the fundamental principles of the soft k-means algorithm. Firstly, achieving complete consistency between the validation labels and the true labels is indeed challenging. Nonetheless, existing studies have demonstrated satisfactory performance despite minor discrepancies between the validation and test sets [1]. It is important to note that we have consciously avoided over-tuning the hyper-parameters on the validation set, limiting our adjustments to only three parameters, as detailed in the appendix. Secondly, we posit that raw features possess inherent clustering properties, and the soft k-means method has been proven effective and widely adopted in various contexts, as supported by references [3][4]. Our empirical evidence further substantiates this claim.

In our experiments, the accuracy of pseudo-labels was evaluated on several datasets, all showing relative high accuracy. This leads us to believe that pseudo-labels serve as a relative reliable proxy for true labels during validation. For the dataset "income" , where accuracy was comparatively lower compared to the classes number, we conducted experiments without using a validation set but with fixed hyper-parameters. The results indicated that selecting parameters using a pseudo-label validation set achieved better performance than using fixed hyper-parameters. The robustness of our method, demonstrated across multiple datasets, enables our approach to be applicable to a wide variety of tabular datasets.

Table 1: Accuracy of pseudo-labels

Table 2: Fixed hyper-parameters VS Tuned hyper-parameters

[1]Peng X, Usman B, Kaushik N, et al. Visda: The visual domain adaptation challenge[J]. arXiv preprint arXiv:1710.06924, 2017. [3]Ferraro M B, Giordani P. Soft clustering[J]. Wiley Interdisciplinary Reviews: Computational Statistics, 2020, 12(1): e1480. [4]Singh V K, Tiwari N, Garg S. Document clustering using k-means, heuristic k-means and fuzzy c-means[C]//2011 International conference on computational intelligence and communication networks. IEEE, 2011: 297-301.