On Transferring Expert Knowledge from Tabular Data to Images

We appreciate your recognition of our paper and your valuable comments. We will answer the questions below and we hope this clears up your concerns.

Q1: The paper has a lot of mathematical notations that are not defined or ambiguous.

A1: Thank you for your suggestions. Specifically, the transform numerical and categorical values into an embedding are as follows: $T_j = b_j + f_j(x_j) \in \mathbb{R}^d$, where $b$ is the feature bias, $f$ is implemented as the element-wise multiplication with the vector $\boldsymbol{W}$. Overall $\boldsymbol{x_{num}^T} = b_{num} + \boldsymbol{x_{num}} \boldsymbol{W_{num}}$ is for numerical attributes and $\boldsymbol{x_{cat}^T}= b_{cat} + \boldsymbol{x_{cat}} \boldsymbol{W_{cat}}$ for categorical. By denoting different attributes distinctly, we can effectively distinguish between numerical and categorical variables.

We have also corrected some remaining errors, including the representation of vectors and matrices, the modification of argmin, and the specification of the Frobenius norm. Thank you for your generosity in pointing out our problems.

Q2: A comparison with the case where, instead of the tabular alignment in the method, a fully connected layer or an MLP dedicated to each attribute prediction task is used is necessary.

A2: Thank you for pointing out our typo. Thank you also for pointing out the shortcomings of our comparison methods. In fact, we have taken this situation into account in our Appendix B1. We use the self-attention method to weigh the image channels differently and learn each tabular attribute separately, which can be seen as an enhanced MLP. The self-attention method constructs a learnable mask for each attribute and learns each attribute separately based on the extracted feature.

The result shows that the self-attention method did not perform as well as our method on all datasets. Specifically, on the DVM dataset, which involves a complex downstream task of 129 classification tasks, the attention method constructed different attentions for different attributes, which confused the backbone network and led to a decrease in overall task performance. This finding highlights the impracticality of using the attention mechanism alone to integrate the abundant information in tabular data into the image model. In other words, if we directly let the image predict every attribute of the table, this does not transfer the knowledge from the table to the image, but instead clutters the image model and causes it to learn without focus.

However, in our approach, we automatically find out the tabular attributes associated with the image and pay more attention to the learning of the relevant parts, which serves to guide the image for more detailed learning, making our approach perform better.

Q3: The second term of the loss in Eq. (6) does nothing to the performance.

A3: The second term in Eq. 6 is to update the tabular model. The tabular model is updated in order to get a more accurate representation of each tabular attribute. As shown in Eq. 5, we need to use the feature similarity matrix of each attribute of the table when calculating the cost matrix of OT. Therefore the tabular model is updated to get a more accurate representation of each attribute of the table, so as to compute a more accurate transfer matrix of the OT, which lays the foundation for a specific channel to learn a certain attribute of the table.

As we stated in Appendix A2, we choose to update the cost matrix every 5 epochs, striking a balance between updating them without stable knowledge and minimizing the computational burden. An update of the cost matrix is essential. As the model is updated, the representation of each attribute and channel is more and more accurate. Therefore, the second term of loss in Eq. 6 is essential.

To prevent any confusion regarding the formula, we acknowledge that there might have been some oversight on our part. Initially, we mistakenly assumed that this method was universally understood by everyone. We would like to rephrase our statement in the hope of providing clearer comprehension. Thanks again for pointing out our typo. We will also make this clear in the final version.

We appreciate your recognition of our method and your valuable comments. We will answer the questions below and we hope this clears up your concerns.

Q1: Phi has been first used in page 4, equation 3 but hasn’t been explained until page 6.

A1: Thank you very much for pointing out our problem. Phi is the feature extractor to extract the embedding of the images. We will explain phi in more detail under equation 1 in a future version.

Q2: Consider other matrices for checking the model performance.

A2: Thank you very much for your advice. We have experimented on the SUN dataset based on your suggestion.From the results, we can see that all of our methods achieve either optimal or suboptimal results. The different metrics provide more evidence of the robustness of our methods in different ways.

Q3: Limitations have not been discussed in the paper.

A3: We talk about some of the limitations of the change of method at the end of the Appendix. For example, our method relies on the mutual information of tables and images. On the other hand, the annotation of expert data often comes at a huge cost penalty, which challenges our method for knowledge transfer with a small amount of expert annotation. We will make it clear in the final version.

Q4: some human evaluation of the method.

A4: Thank you for your advice. We are working with hospitals to try to apply our method to the task of infection prediction. However, it is difficult to give a quantitative result to demonstrate the performance of our method due to the privacy of the users and the special requirements of medical treatment. If there is further result from expert evaluations, we will update it in time for later versions. Thanks again for your suggestion.

Q5: A clear explanation on the different types of tasks.

A5: Thanks for your suggestion. Due to the limitations of the paper's length, we will explain the use of our model in the different tasks in further detail in Appendix.

CHARMS is a general method for both classification and regression tasks, in detail, we use cross entropy loss for classification task and mean square error loss for regression task. We achieved an improved image representation by employing the CHARMS method, which leverages the guidance of tabular data on the image data. Specifically, for the classification task, our approach facilitated the representation with a more discerning distribution over the target categories. On the other hand, the regression task enabled us to learn an image representation that better approximated the target values during prediction. The fact that our method performs well on both tasks underscores its generalizability and effectiveness.

In summary, our approach demonstrates its versatility by excelling in both classification and regression tasks, showcasing its ability to enhance image representations using guidance from tabular data.

Q6: What is “attention image” referred to above equation 6?

A6: We use A to weigh the features of the image after extracting them, which can be seen as an attentional approach and pays more attention to the channels corresponding to the tabular attribute. More information about attention is in Appendix B1. We will make it clear in the final version.

Q7: The proposed model underperforms the LGB and RTDL.

A7: In some datasets, the performance after cross-modal transfer is not as good as the performance of the tabular model may be due to the fact that the attributes in the tables that can not be transferred are critical for the task. In these datasets, the tabular data is strong modality therefore the tabular data performs better.

On DVM dataset, the tabular model demonstrated promising results, primarily because a significant portion of the knowledge contained in the tabular data, such as engine parameters and prices. These attributes play a crucial role in the classification task.

In summary, it is possible that our approach can outperform the tabular modality even if it is a strong modality. We've already stated this in result in Sec 5.1, and we'll make it clearer later.

Q8: Is the code for this method openly available?

A8: Yes, our code is being organized and will be open-sourced later for reproducibility.

We appreciate your recognition of our idea and your valuable comments. We will answer the questions below and we hope this clears up your concerns.

Q1: The presentation of the paper.

A1: Firstly, we would like to thank you very much for your valuable suggestions for modifications. We will carefully correct these problems in the later submitted version. Secondly, we will also do our best to clear up some of your doubts. For example, for W1.6, we find it important to transfer knowledge from tabular data to images, as this can lead to image modeling enhancements. But little attention has been paid to this before. With our approach, we aim to demonstrate the feasibility of transferring knowledge from tables to images. Therefore, we consider this as one of our contributions.

Q2: An empirical argument before the method but the section was confusing.

A2: Thank you for question. Firstly, here we would like to point to Figure 2. We have used this Figure 2 again in the appendix to make it clearer for the reader to read and analyse in conjunction with Figure 9.

Secondly, we calculated the mutual information with tabular data versus the image data to construct Fig. 2. We want to illustrate that the mutual information is not the same for different methods and different modalities. Then, we found that the mutual information of and image can be improved by distillation of morphological data. Finally, we conclude that different attributes in tabular data have different effects on the image and we need to explore the useful knowledge in tabular data and transfer it to the image.

Finally, we will also describe the figure in more detail in the article so that readers can capture the information in the diagram more clearly.

Q3: There is some information need to be present. A description of the experimental protocol. Also, the LGB, RTDL, and Resnet methods are listed and some of these models seem to outperform knowledge transfer methods.

A3: Thanks for your reminder. We describe our experimental setup in Sec 5 and Appendix A. Specifically, there are experimental hyperparameter settings, as well as experimental details. We will select some important information to put in the main paper if space permits in the final version.

In Table 1 we show the performance of them is to tell the reader roughly where each modal's performance would be. This allows us to clearly see how performance changes after modal knowledge transfer, e.g. if there is a negative transfer. We think it is necessary to show the performance of the tabular method here because we can see the difference in the amount of information in the tabular data and the image data. The amount of change in performance after transfer shows how much knowledge from the tabular data is actually transferred.

Thank you for pointing out that we should explain the fact that our method does not perform as well as the tabular model in some datasets. In our setting we expect to transfer knowledge from the tabular data to the images, but not all tabular knowledge is transferable. In some datasets, the performance is not so good may be due to the fact that the attributes in the tables that can not be transferred are critical for the task. In these datasets, the tabular data is strong modality therefore the tabular data performs better. For example, in the Pet dataset, the pet's vaccine information is important for pet adoption, but this information is not displayed in the image.

In summary, we expect to transfer useful, image-related knowledge from tables to images to improve the performance of image models.

Q4: The use of "channel".

A4: In the AlexNet's paper[1], the authors indeed mentioned that the image consists of RGB channels. Similarly, in the case of ResNet[2], the authors described the image processing as being conducted channel by channel. We think it is common to use channel to represent high-level or low-level expressions of image information.

Q5: What are the quantities between which you are measuring the mutual information?

A5: In Appendix A3 of the paper, we explain in detail the exact meaning of this figure and the calculation of mutual information. Specifically, we calculate the mutual information between the representation obtained from our model and the representations of both the optimal table model and image model with MINE method. This allows us to quantify the amount of shared information or knowledge between these representations. By measuring mutual information, we can assess the extent to which our model captures and transfers knowledge from both the table and image domains. We may have misjudged the audience for this paper and put the relevant text in the appendix, this is a problem and we will try to reorganize the writing.

[1] Imagenet classification with deep convolutional neural networks, 2012.

[2] Deep Residual Learning for Image Recognition, 2016.

We appreciate your recognition of our setting and method and your valuable comments. We will answer the questions below and we hope this clears up your concerns.

Q1: Limited exploration of extremely large tabular datasets.

A1: Thank you for bringing this to our attention. Our objective is to transfer knowledge from tabular data to images. Through the use of POT, we are able to automatically match the corresponding tabular attributes to the images. Consequently, even with a substantial number of tabular attributes, our method performs admirably. For instance, in our SUN dataset, which contains 101 tabular attributes, our method outperforms all the compared methods. We have made significant efforts to locate a suitable tabular-image dataset, and it has proven to be quite challenging. We will continue our endeavors to conduct experiments on larger tabular datasets.

Q2: There are many uninformative features.

A2: Thank you for your valuable advice. In our approach, the inclusion of POT enables automatic selection of image-related features for knowledge transfer. As you rightly mentioned, in the dataset we are utilizing, certain attributes such as the pet's health or vaccine information are not visually depicted in the image. Therefore, these attributes can indeed be considered as uninformative features, just as you suggested. For instance, in Pet dataset, we can treat these attributes as uninformative features that do not significantly contribute to image representation and so as other datasets. Your suggestion of adding noise features to verify the performance changes is very reasonable. We conducted experiments based on your suggestions, but the results were basically unchanged.

Q3: Work on complex vision tasks.

A3: In complex tasks like segmentation and detection, the objective is to accurately construct bounding boxes or differentiate between objects with distinct semantics. When dealing with tables that contain semantically specific attributes, we hypothesize that our approach can do this. However, due to the inconsistency of the task goals, when using our method to the detection task, we may need to do extra processing on the bounding box to make it adapt our method. We appreciate your suggestion, and it indeed aligns with potential future work. Thank you for your valuable question.

Q4: Hyperparameter tuning and pre-trained weights.

A4: First of all, our comparison is fair. We have performed parameter tuning for all baseline methods. Specifically, we adjusted the temperature coefficients in the KD method, as described in Appendix A2. In the case of the FMR method, we modified the knockdown_num parameter to regulate the amount of knowledge transferred to the image model at each iteration. Similarly, for the MMCL method, we fine-tuned the temperature coefficient in the contrastive loss. Furthermore, it is important to note that all methods utilized the same weight for the image model. We appreciate your observation and will make it clear in the appendix.

Q5: New backbones like CLIP.

A5: Thank you for pointing out the comparison with the CLIP method. We have compared with the CLIP method in Appendix B2 and here we will also describe to you again our comparison and some conclusions.

The setting of this paper is more general. We expect to transfer the tabular knowledge to the image modality during training to cope with the absence of expert knowledge during testing.

CLIP is pre-trained on a large amount of text and image pairs, which makes it able to map from text to images. The details of the experiment can be found in Appendix. From the experiments, we can see that the performance of CLIP is not ideal. This is probably due to the fact that in tabular data, each column holds its own distinct meaning, and directly utilizing it as input to CLIP can lead to the loss of certain information. From another perspective, previous work[1] has pointed out that there is a modality gap in the CLIP's embedding space.

In summary, the loss of information and the modality gap can hinder the performance of the CLIP method in our setting. However, our method addresses these challenges by automatically discovering and establishing the matching relationship between the two modalities, which is a more general method.

Q6: Related work about tabular learning.

A6: Thank you for your suggestion. We conducted a comparison between the traditional tree-based approach and the FT-Transformer approach to gain a preliminary understanding of tabular data performance. While the methods you mentioned can achieve comparable results, they cannot be directly applied in our specific setting. When referencing tabular models, a comprehensive understanding of the domain will enhance the completeness of the paper. We will update the related work.

[1] Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. 2022.

Dear Reviewers,

Thank you for providing insightful feedback to help us improve our work. We hope that our responses can address your concerns and are helpful in improving your rating. If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you! Thank you again for the review.