Handling Learnwares from Heterogeneous Feature Spaces with Explicit Label Exploitation

Q1: Clarify the relations between each technical contribution and the heterogeneous feature spaces.

Thank you for your comments. We will provide a more detailed discussion in the revised version. The following is a brief explanation:

Subspace learning is a standard method for handling heterogeneous features. The critical aspects are from what to learn and how to learn it. The learnware framework protects model providers' privacy without accessing their raw data and avoids collecting auxiliary data across feature spaces. Without raw and auxiliary data, how can the learnware dock system effectively learn a unified subspace?

In the learnware paradigm, subspace learning can be performed based on the RKME specification of the model. The RKME specification compresses the marginal distribution $P(X)$ through a small number of weighted sample points. However, previous methods that lack label information rely on unsupervised learning techniques, leading to suboptimal subspace learning results. Without label information, categories from different tasks may overlap in the subspace, impacting model recommendation and reuse effectiveness.

To address this, we introduce supervised information into subspace learning to better align samples from each class of different tasks within the subspace, enhancing the heterogeneous model hub's performance. Thus, we extend the original RKME specification representation $\\{\beta\_m,\boldsymbol{z}\_m\\}\_{m=1}^M$ to include label information, obtaining $\text{RKME}\_\text{L}$, represented as $\\{\beta\_m,\boldsymbol{z}\_m, y\_m\\}\_{m=1}^M$. We propose two methods: compressing the marginal distribution first and then generating the label, or simultaneously compressing both the marginal distribution $P(X)$ and the conditional distribution $P(Y|X)$. By incorporating label information, we add supervised term for subspace learning, obtaining a subspace with better properties and corresponding mapping functions to original feature spaces.

In summary, our main techniques are designed to enhance subspace learning.

---

Q2: In the experiments, the paper should compare with more SOTA methods which handle the heterogeneous features. In current version, most compared methods are not designed for heterogeneous features.

Thank you for your comments. Indeed, most of the compared methods are designed for heterogeneous features and are SOTA methods. We will describe the contenders more clearly in the revised version.

This paper explores using knowledge from tasks with heterogeneous feature spaces to improve user task performance.

First, we evaluate user self-training performance using LightGBM with the same training setup as the learnware preparation for fairness.

Next, we compare our approach with recent SOTA methods that use the heterogeneous model hub for the user task:

• Align_unlabel (KDD 2024): Minimizes MMD distance between the user task and learnware task based on RKME. We modified it to traverse the best model in the hub, improving performance.
• Align_label (KDD 2024): Builds on Align_unlabel by using user-labeled data for feature augmentation, further improving performance. We also traverse the hub for the best model.
• Hetero (IJCAI 2023): Generates a subspace for model recommendation/reuse using basic RKME without label information, creating a linear mapping function.

These solutions learn feature space mapping without label information. Our paper fully utilizes label information, resulting in significantly better performance.

We also compare with other SOTA methods that use raw heterogeneous task data, compromising privacy:

• TabPFN (ICLR 2023): Uses a Transformer to fit the posterior predictive distribution from synthetic tasks and applies it to real-world tasks. It outperforms other deep methods in the small data paradigm.
• Transtab (NeurIPS 2022): Uses feature descriptions and values to generate a unified subspace and trains a shared Transformer backbone, pioneering deep network training across heterogeneous tasks.
• Xtab (ICML 2023): Trains a shared backbone for different tasks with a specific input processor and prediction head, without using feature descriptions, offering flexibility in real-world scenarios.

In summary, the contenders are highly relevant to heterogeneous feature spaces. We compared not only with SOTA methods that exploit the heterogeneous model hub but also with other SOTA methods that use raw data from heterogeneous tasks for new user tasks.

---

Q3: Conduct experiments on at least one real-world heterogeneous data.

Thanks for your suggestion, we add two real-world projects to demonstrate the efficacy of proposed method, please see author rebuttal for details.

---

Q4: In Eq.(2), there is a term Lsimilar, but in Section 5.2, there is a term called Lcontrastive. Are them the same? Is this a typo or are they two different losses?

We apologize for any confusion caused. Thanks for your careful reading.

In short, $L\_{\text{similar}}$ is a general term for similarity loss, and $L\_{\text{contrastive}}$ is a specific implementation.

In Section 4.2, we define the general objective for subspace learning as Equation (2), based on all model specifications $\\{\boldsymbol{s}\_i^{\text{dev}}:=\\{(\boldsymbol{\beta}\_{ij},\boldsymbol{z}\_{ij},y\_{ij})\\}\_{j=1}^{m\_i}\\}\_{i=1}^N$: $\min\_{\\{h\_k,g\_k\\}\_{k=1}^Q} L=\alpha\_1L\_{\text{reconstruction}} + \alpha\_2L\_{\text{similar}} + \alpha\_3L\_{\text{supervised}}.$ The similarity loss ensures that embeddings of different slices of the sample $\boldsymbol{z}_j$ are similar. This can be implemented in various ways, such as through contrastive loss, manifold loss, etc.

In Section 5.2, we provide a detailed implementation of the loss function, specifically using contrastive loss to measure similarity, which is a quite popular loss for self-supervised learning.

Q1: To my knowledge, there have been some works aiming at handling learnwares from heterogeneous feature spaces. However, these works are only cited in reference list while not adequately discussed.

Thanks for your careful reading. We will discuss more related papers in the revised version. Here is a brief discussion:

These works focus on constructing and utilizing heterogeneous learnware markets in specific scenarios with relatively fixed feature combinations, such as relational databases constructed from multiple related tables. For a specific task, the complete feature space consists of several blocks, and the user and learnware feature spaces are combinations of these blocks.

• MLJ 2024: The first work to consider heterogeneous learnware. The original training data is accessible, and auxiliary data across the entire feature space is collected. The paper assigns specifications to heterogeneous models, learns the subspace using the model's original data and auxiliary data, and then generates the RKME specification based on the projection results.

• IJCAI 2023: This study explores organizing and utilizing a heterogeneous model hub without accessing the original data or auxiliary data across the feature space. It performs subspace learning via the RKME specification. Developers upload models and specifications to the learnware market. The market learns the subspace and mapping function based on the specification, aligning learnware specifications with user requirements.

• This paper: This work identifies the limitations of the RKME specification, noting its inability to effectively support subspace learning and model recommendation due to the lack of label information. Unsupervised subspace learning can mix data from different tasks and categories, leading to poor model recommendation and reuse. The RKME specification only describes the marginal distribution, matching models based solely on this without considering their capabilities. This paper embeds label information and model capabilities into the specification to improve the heterogeneous learnware market mechanism. This allows subspace learning to incorporate supervised information, better aligning samples from each class of different tasks. In model recommendation, the model's classification boundary is matched with the user distribution, enhancing the performance of the heterogeneous model hub.

---

Q2: Discussion of newly proposed specifications

Thank you for your advice. We will revise the abstract and provide a detailed discussion on the newly proposed specification in the revised version. Below are the major differences between the two specifications:

• Summary: Specifications describe a model's ability without exposing raw data. $\text{RKME}$ (TKDE 2023) sketches the marginal distribution of the training data but lacks label information, which is insufficient for handling heterogeneous learnwares. We extend $\text{RKME}$ to $\text{RKME}_\text{L}$ by adding label information to better encode the model's ability through its conditional distribution.
• Representation Form: The original $\text{RKME}$ specification includes minor weighted samples $\\{\beta\_m,\boldsymbol{z}\_m\\}\_{m=1}^M$. The extended $\text{RKME}\_\text{L}$ specification includes minor labeled weighted samples $\\{\beta\_m,\boldsymbol{z}\_m,y\_m\\}\_{m=1}^M$, where $\beta\_m$ is the weight, $\boldsymbol{z}\_m$ is the generated sample, and $y\_m$ is the label.
• Generation Mechanism: $\text{RKME}$ outlines the marginal distribution $P(X)$ of a dataset. In contrast, $\text{RKME}\_\text{L}$ includes label information, outlining both the marginal distribution $P(X)$ and the conditional distribution $P(Y|X)$. $\text{RKME}$ uses minor weighted samples to approximate the kernel mean embedding of the original data, with distance measured by MMD distance. $\text{RKME}\_\text{L}$ can either compress the marginal distribution $P(X)$ first and then generate labels or compress both $P(X)$ and $P(Y|X)$ simultaneously based on model predictions.
• Impact on Subspace Learning: With $\text{RKME}$, the lack of label information can result in poor model recommendation and reuse as different task categories may overlap in the subspace. In contrast, $\text{RKME}\_\text{L}$ allows for better alignment of samples from different tasks within the same class, leading to improved performance of the heterogeneous model hub.
• Impact on Recommendation Mechanism: $\text{RKME}$ only recommends models with a similar marginal distribution to the user task, ignoring the model's ability. $\text{RKME}\_\text{L}$ enables the system to match the model boundary with the user task boundary by comparing the conditional distribution $P(Y|X)$, resulting in better model recommendation outcomes.

---

Q3: It is mentioned that "the recommended heterogeneous learnware significantly outperforms user self-training with limited labeled data", it is similar to semi-supervised learning. Is it fair to compare the proposed method with user self-training with limited labeled data?

We apologize for any confusion caused and appreciate your careful reading. We will address this in the revised version.

Our problem setup aligns with supervised learning; however, the user's training data may be insufficient. In such cases, training a model with a small expected loss is challenging due to the lack of labeled data. By leveraging the heterogeneous learnware dock system to identify and reuse existing well-trained models, users can significantly enhance task performance, as demonstrated in the paper experiments.

---

Q4: Extension for heterogeneous label space

Thanks for your suggestion. Please see author rebuttal for a brief discussion.

Q1: The potential challenges or limitations of the proposed model are suggested to be given.

Thanks for your advice. We discuss the limitations of the proposed method in the checklist (see Q2: limitations).

In this paper, we consider scenarios where the feature spaces of models in the learnware dock system are heterogeneous. However, real-world applications may also have heterogeneous label spaces. We will add more discussions on this in the revised version and please see the author rebuttal for a brief discussion.

---

Q2: The paper does not provide a thorough analysis of the computational complexity and scalability of the proposed framework, which could be important for large-scale, real-world deployments.

The process of the learnware paradigm is outlined as follows:

• Building the Learnware Market: Developers generate specifications, package them with models into learnware, and submit them to the learnware dock system. This system organizes the learnware and assigns system-level specifications to the models.
• Utilizing the Learnware Market: Users generate requirements and submit them to the system, which then recommends appropriate learnware. Users can then reuse the learnware.

Below, we analyze the complexity of each process. The relevant symbols are defined as follows: the size of the raw dataset is $n$, the number of categories is $c$ (1 for regression), the dimension is $d'$, and the size of the specification is $m$. There are a total of $N$ learnwares in the learnware dock system, and the complete feature space contains $K$ sub-blocks with a total dimension of $d$.

1. Specification/Requirement Generation

• Initialization: k-means clustering, with a complexity of $O(nmcd')$.
• Iterative Optimization: Achieved through alternating optimization, with the bottleneck being the inverse matrix computation of the kernel matrix, which has a complexity of $O(cn^3)$.

The total complexity is $O(c(nmd' + T\_s n^3))$. Typically, the number of iterations $T\_s$ is small, around 10, and the size of the specification $m$ is much smaller than the dataset size $n$. When the sample size $n$ is large, the complexity of specification generation is $O(cn^3)$, scaling cubically with the size of the raw dataset $n$. However, with GPU acceleration, the time required for specification generation is quite small. For the dataset openml__volkert__168331 with shape (58310, 180), the generation time of specification is 2.826s (A100 80G*1).

2. Subspace Learning

• Single Epoch, Single Specification: The complexities for calculating the contrastive loss, reconstruction loss, and supervised loss are $O(m^2 K^2 d)$, $O(mK d^2)$, and $O(mcd + md^2)$ respectively, totaling $O(m^2 K^2 d + mK d^2)$. The complexity of updating the model using the loss functions (both are two-layer fully connected networks) is $O(m d^2)$. Thus, the complexity for subspace learning with a single specification is $O(m^2 K^2 d + mK d^2)$.
• All Epochs, All Specifications: Considering all specifications and the number of iterations, the complexity for subspace learning is $O(N T\_{sub} (m^2 K^2 d + mK d^2))$.

This complexity is linearly related to the number of learnwares $N$ in the market and quadratically related to the size of the specifications $m$，the dimension $d$ of the feature space and the number of feature blocks $K$.

3. Learnware Recommendation

The time complexity is $O(Ncm^2)$, linearly related to the number of learnwares $N$ in the market.

4. Model Reuse

When reusing a model, it only requires mapping the raw data through a two-layer fully connected network to the feature space corresponding to the heterogeneous learnware and making predictions using the learnware, which has low time complexity.

5. Summary

The most time-consuming procedure of the learnware paradigm is the subspace learning for heterogeneous learnware recommendation. Even with a large number of learnwares $N$, the total training data $Nm$ for subspace learning remains manageable due to the small size of each specification (typically $m = 50$). Furthermore, our methods train an encoder and decoder for each feature block instead of each model. Although the number of models can be quite large, the number of feature blocks is relatively small. An extreme case involves $K$ blocks, resulting in $2^K$ feature spaces and $2^K$ models for each feature space, our method trains $K$ encoders and decoders, not $2^{K}$.

Besides the theoretical analysis, we will also provide the actual running time in the attached PDF to demonstrate the efficiency of the proposed framework. Our method have much less time for preparation (construct the learnware market) compared to the pre-training methods, and the time for utilization (recommend and reuse learnware) is also efficient. Please see author rebuttal for details.

---

Q3: The experiments are conducted on a single dataset, and a more diverse evaluation of different types of tasks and datasets would strengthen the generalizability of the findings.

We have conducted experiments on real-world projects, please see author rebuttal for details. Thanks for your suggestion.

Q1: The method seems complex. Thus, a running time comparison is necessary to show its efficiency.

Thank you for your suggestion. In the author rebuttal, we provided a brief discussion; here is a more detailed explanation.

1. Evaluation Setup

The evaluation setup includes the preparation and utilization phases. Since the preparation phase is done once by non-user entities, we focus on the utilization phase time required for user tasks. The evaluation setup for each method is as follows:

• lightgbm: Utilization phase only, training from scratch on user-labeled data.
• TabPFN, Transtab, Xtab: Both phases. Preparation involves obtaining a pre-trained model; utilization involves fine-tuning on user data.
• Align_unlabeled, Align_labeled, Hetero, Our_unify, and Our_cls: Both phases are involved. Preparation includes constructing and organizing a model hub, while utilization involves generating task requirements, recommending models, and reusing models on user data.
  • Our_unify requires a self-trained model for pseudo-labeling during specification generation and is influenced by the model type and training parameters. Therefore, $**Our**_{**unify**}$ records the time assuming a pre-existing self-trained model, whereas $**Our**\_{**unify**}^{\text{lightgbm}}$ includes the entire process, including model training time using the time-consuming lightgbm method. Users can opt for a simpler model to reduce time costs.

2. Summary of Evaluation Results

• Classification Tasks:
  • $**Our**\_{**unify**}$ ranks 1st, $**Our**\_{**cls**}$ ranks 3rd
  • $**Our**\_{**unify**}^{\text{lightgbm}}$ ranks 7th (lightgbm ranks 5th).
• Regression Tasks:
  • $**Our**\_{**unify**}$ ranks 2nd
  • $**Our**\_{**unify**}^{\text{lightgbm}}$ ranks 4th (lightgbm ranks 3rd).

3. Analysis of Utilization Phase

Compared to other methods with $**Our**_{**unify**}$:

• lightgbm: Requires training a model from scratch, whereas our methods reuse relevant heterogeneous models, making them faster.
• TabPFN, Transtab, Xtab: Our methods do not need the entire labeled user data for fine-tuning, making them quicker.
• Align_unlabeled, Align_labeled: Our methods do not traverse the model hub but directly find the most suitable model, making them faster.
• Hetero: Similar process, but Hetero might recommend multiple models, whereas our methods recommend only one, making them faster.

Among our methods, $**Our**\_{**unify**}$ is faster than $**Our**\_{**cls**}$. $**Our**\_{**unify**}$ compresses X first, then uses the model to predict on the reduced set, generating the specification quickly if user model is already prepared. $**Our**\_{**cls**}$ predicts first, then compresses X and Y distributions simultaneously, making it slower.

By efficiently generating user task requirements and quickly reusing heterogeneous models, our method reduces the utilization phase time compared to other methods.

4. Analysis of Preparation Phase

• TabPFN: Pre-training from simulated datasets generated by structure causal model, learning the posterior distribution $P(y\_{test}|x\_{test},D\_{train})$ and applying it to small-scale real tasks. Pre-training time: 20 hours (8*RTX2080Ti).
• Xtab: Learns a shared backbone network from many tasks; pre-training time unknown.
• Transtab: Uses feature descriptions and values from related tasks to generate a unified subspace, training a shared Transformer backbone. Average pre-training time: 672 seconds (classification) and 1425 seconds (regression).
• Align_unlabeled, Align_labeled: No model library organization time.
• Hetero: Organization time: 2.44 seconds (classification) and 2.55 seconds (regression).
• Our_unify: Organization time: 41.39 seconds (classification) and 40.78 seconds (regression).
• Our_cls: Organization time: 45.69 seconds (classification).

Times are based on a single A100 GPU.

Compared to methods using pre-trained models, our methods significantly reduce preparation time by not requiring training on complete datasets but using specifications, which are smaller. We also do not train a unified backbone network, only performing simple subspace learning.

---

Q2: I wonder if the method can be used for larger data to show its scalability.

Thank you for your question. Indeed, our experiments test datasets of various sizes and show that even with more labeled data, our paradigm enhances user performance. We also discuss the scalability of the learnware dock system with numerous heterogeneous learnware in the author rebuttal.

1. Our experiments test datasets of various sizes

We tested datasets with varying sizes. For classification tasks, sample sizes range from 1,000 to 58,310, feature dimensions from 7 to 7,200, and classes from 2 to 10. For regression tasks, sample sizes range from 418 to 108,000, with feature dimensions from 8 to 128.

2. Our method boosts performance even with more labeled data

Tables 1 and 2 in our paper show that with only 100 labeled samples, using the recommended heterogeneous model significantly outperforms user self-training. Figures 4 and 5 illustrate that combining the heterogeneous model with user self-trained models improves performance across different labeled data amounts. On average, with 500 labeled samples, reusing the recommended model still outperforms self-training. With 2,000 labeled samples, the heterogeneous model continues to enhance performance. Even with 5,000 labeled samples, learnware improves 21% of classification cases and 50% of regression cases. In some cases, like kin8nm, even using the entire training dataset, the heterogeneous model improves user performance.

3. The cost of organizing a learnware dock system with numerous heterogeneous learnware is relatively small

Please refer to the author rebuttal for details.