Q1: I do not see why the results in the paper are only specialized to CLIP models instead of general estimating functionals of high dimensional (features embeddings) inner products, ..., Could the authors please elaborate more on this as this is really confusing to me about the real contribution of the paper?

Thank you for the question. We appreciate the comment about the interpretation of our method beyond CLIP as general estimating functionals of high-dimensional inner products. Nonetheless, the main motivation for the proposed method is to provide an alternative method to solve the CLIP loss and some theoretical understanding of CLIP from the RKHS perspective instead of the neural network. We acknowledge the complexity of thoroughly analyzing the intricate neural architecture of CLIP, which is why we have focused on solving the loss function of CLIP and attempted to give an alternative interpretation of the essence of CLIP via the kernel method. Therefore, we believe our proposed MTMK method should be considered a novel methodological contribution to the CLIP community.

Q2: It is mentioned in the Related literature that no theoretical analysis of MKL has been conductied in the multi-task setting. Could the authors justify that as there is a big literature on the subject?

We appreciate your concern and the reference provided. We apologize for the imprecise wording and the lack of citations in our initial submission. While there is indeed a substantial body of literature discussing kernel constructions, optimization, and computation aspects of MKL in the multi-task setting, we intended to highlight the scarcity of works that specifically analyze the optimal learning rate in multi-task multi-kernel learning. The statement "no theoretical analysis" refers to the limited research on the analysis of the statistical rate. In the revised version of our paper, we will include a discussion on the general theoretical research line in multi-task multi-kernel learning, along with relevant citations such as [1-6]. We will also ensure that the wording is accurate and precise.

[1] Micchelli, C., & Pontil, M. (2004). Kernels for Multi--task Learning. Advances in neural information processing systems, 17.

[2] Dinuzzo, F., Ong, C. S., Pillonetto, G., & Gehler, P. V. (2011). Learning output kernels with block coordinate descent. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) (pp. 49-56).

[3] Ciliberto, C., Mroueh, Y., Poggio, T., & Rosasco, L. (2015, June). Convex learning of multiple tasks and their structure. In International Conference on Machine Learning (pp. 1548-1557). PMLR.

[4] Ciliberto, C., Rosasco, L., & Villa, S. (2015). Learning multiple visual tasks while discovering their structure. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 131-139).

[5] Jawanpuria, P. K., Lapin, M., Hein, M., & Schiele, B. (2015). Efficient output kernel learning for multiple tasks. Advances in neural information processing systems, 28.

[6] Pentina, A., & Ben-David, S. (2015). Multi-task and lifelong learning of kernels. In Algorithmic Learning Theory: 26th International Conference, ALT 2015, Banff, AB, Canada, October 4-6, 2015, Proceedings 26 (pp. 194-208). Springer International Publishing.

Q3: In the experiments section (section 7.2), the authors do not specify which embeddings are used (are they CLIP embeddings?), justifying again my concerns of why CLIP as a motivation of the paper?

Thank you for the question. We perceive $\phi(x)$ as the embedding of $x$ and derive the corresponding inner products between samples via the method described in Section 4. Therefore, the embedding derived for images and phecodes in Section 7.2 corresponds to the estimated mapping of our MTMK method. As for the initial word embeddings for phecodes, we use the ones provided in the MIKGI dataset described in the third paragraph.

Q4: More CLIP related experiments?

Thanks for the question. To further illustrate the performance of the neural network-based CLIP, we will conduct some new experiments to further showcase the empirical performance. We hope the additional results (to be uploaded) resolve your concerns.

Thanks for the constructive comments, we address the concerns below.

Q1: How do we interpret the contribution of this work?

Thank you for the question. We expect that our work offers some theoretical understanding of CLIP from the RKHS perspective rather than the standard neural network perspective. In essence, we do not aim at proposing a new method to train a better CLIP model. Instead, we attempt to provide an alternative method to solve the CLIP loss, which is parallel to the standard backpropagation training scheme from a higher level. Therefore, we believe our proposed MTMK method should be considered a novel methodological contribution to the CLIP community instead of a remedy to the existing limitations of CLIP.

Q2: The methods used in this paper is too heuristics.

We appreciate the reviewer for bringing up this concern. We acknowledge that there is a certain degree of relaxation between the original loss and the objective we have derived. However, we argue that such relaxation is necessary to statistically characterize the complexity and learning rate of the model, as presented in Section 6.

Q3: Experimental section can be improved by adding more datasets and baselines.

We appreciate your suggestion on improving the numerical evaluation. However, we would like to emphasize that this work mainly focuses on a theoretical interpretation of CLIP rather than proposing a new method. As CLIP has demonstrated excellent generalizability in broader domains, we believe our work can be extended to various tasks as CLIP. To further illustrate the performance of the neural network-based CLIP, we will conduct some new experiments to further showcase the empirical performance. We hope the additional results (to be uploaded) resolve your concerns.

Thanks to the authors for performing these additional experiments to show that the proposed method perform better overall compared to the traditional way of training CLIP. Questions: 1- What is the computational complexity of the MTMK method compared to the training of CLIP you have performed? 2- I would stress more as message in the paper that you are proposing a new way of training CLIP and a section on how to do that exactly in the form of an algorithm

2. Choose Different Kernels

Expanding from the foundational task set, we select three task groups, each encompassing six distinct tasks. Within these task sets, each individual task is associated with 1000 samples. Four sets of experiments were conducted to evaluate performance, utilizing the average mean squared error and standard deviation across five different random seeds for validation.

1. Experiment 1: Employing the original experiment's kernel list as the control group

   Model	Task Set 1	Task Set 2	Task Set 3
   ORACLE	0.0107 ± 0.0030	0.0159 ± 0.0035	0.0257 ± 0.0148
   MTMK mix	0.0201 ± 0.0135	0.0310 ± 0.0135	0.0422 ± 0.0150
   MTMK L1	0.0193 ± 0.0128	0.0259 ± 0.0116	0.0382 ± 0.0136
   MTMK L2	0.0251 ± 0.0141	0.0267 ± 0.0108	0.0504 ± 0.0155
   MTSK	0.1355 ± 0.0110	0.1295 ± 0.0115	0.3064 ± 0.0260
   STMK	0.0245 ± 0.0120	0.0296 ± 0.0117	0.0554 ± 0.0112
   STSK	0.1381 ± 0.0116	0.1309 ± 0.0081	0.3055 ± 0.0239

2. Experiment 2: Introducing three additional Neural Tangent Kernels

3. Experiment 3: Misspecified Models excluding the oracle kernels, in contrast to Experiment 1

   Model	Task Set 1	Task Set 2	Task Set 3
   ORACLE	0.0107 ± 0.0030	0.0159 ± 0.0035	0.0257 ± 0.0148
   MTMK mix	0.0245 ± 0.0145	0.0323 ± 0.0134	0.0410 ± 0.0159
   MTMK L1	0.0213 ± 0.0122	0.0308 ± 0.0140	0.0390 ± 0.0164
   MTMK L2	0.0277 ± 0.0143	0.0291 ± 0.0108	0.0516 ± 0.0182
   MTSK	0.1355 ± 0.0110	0.1295 ± 0.0115	0.3064 ± 0.0260
   STMK	0.0314 ± 0.0157	0.0387 ± 0.0143	0.0405 ± 0.0212
   STSK	0.1381 ± 0.0116	0.1309 ± 0.0081	0.3055 ± 0.0239

4. Experiment 4: Misspecified Models removing oracle kernels and introducing new NTKs

   Model	Task Set 1	Task Set 2	Task Set 3
   ORACLE	0.0107 ± 0.0030	0.0159 ± 0.0035	0.0257 ± 0.0148
   MTMK mix	0.0257 ± 0.0153	0.0306 ± 0.0128	0.0422 ± 0.0163
   MTMK L1	0.0230 ± 0.0137	0.0265 ± 0.0108	0.0357 ± 0.0147
   MTMK L2	0.0306 ± 0.0145	0.0298 ± 0.0132	0.0561 ± 0.0172
   MTSK	0.1355 ± 0.0110	0.1295 ± 0.0115	0.3064 ± 0.0260
   STMK	0.0341 ± 0.0150	0.0366 ± 0.0154	0.0439 ± 0.0230
   STSK	0.1381 ± 0.0116	0.1309 ± 0.0081	0.3055 ± 0.0239

Thanks for the constructive comments. As an overview, we expect that our work provides an alternative tool to standard training schemes of neural networks with some novel insight from the RKHS perspective. Now we address the concerns below.

Q1: It is not clear about the focus of the paper. While the primary motivation has been understanding CLIP, the latter part of the paper doesn't offer any discussions, ...

Thank you for the question. The main focus of the paper is to provide an alternative method to solve the CLIP loss and thereby some theoretical understanding of CLIP from the RKHS perspective instead of the neural network. We acknowledge that we focus more on the theoretical analysis of MTMK in the latter parts while the discussions are mainly offered in Section 2. The reason why we organized in this fashion is that we believe once the equivalence between standard CLIP and our proposed method is built, the theoretical analysis (Section 6) and experiments (Section 7) can be applied to CLIP from a theoretical viewpoint. In the revised version, we will include a more comprehensive discussion in later sections.

Q2: The notation itself is a bit confusing to understand. While CLIP uses pseudolabels, which have n different values for n samples, this paper has a notation T for the classes and n for the samples. It would be easier to have a clear writeup on the dataset construction before proceeding to the next equivalances.

Thank you for the comment. The $T$ used in this paper refers to the classes that appeared in the whole dataset for ease of analysis of the learning rate. In the revised version, we will include further clarification with regard to the dataset construction.

Q3: It is not very clear how equation 2.1 represents CLIP. Since in CLIP, the inner product is computed between a text representation of sample i and image representation of sample j. Here, it is assumed that \phi captures both. It would be nice to provide a concrete reference to show that CLIP objective is max_H C^D_H, as mentioned in the paper.

Thank you for the comment. We use the same $\phi$ for notation simplicity in the following analysis, and extending to different $\phi$ for texts and images should be straightforward (for example, using an indicator function to indicate whether it is a text or image sample). Thus, it should be clear now that texts and images are pushed through $\phi$ separately. We will include the above clarification in the revised caption. In addition, equation (2.1) is just the mathematical abstraction of the original CLIP's idea, which is "To do this, CLIP learns a multi-modal embedding space by jointly training an image encoder and text encoder to maximize the cosine similarity of the image and text embeddings of the $N$ real pairs in the batch while minimizing the cosine similarity of the embeddings of the $N^2 − N$ incorrect pairings." [1]

[1] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

Q4: It is difficult to understand the correlate the final reduction in page 5 (the equation on the top) to that of CLIP. Because CLIP labels are defined for pairs, and here the label is defined for a sample.

Thanks for raising this question. Given the complexity of the theoretical analysis of learning rate, we assume the pseudo labels of sample pairs used by CLIP can be treated as labels in our analysis (indicated on page 3 below equation (2.1)). Therefore, in our notation, for a sample pair $\boldsymbol{x}_i, \boldsymbol{x}_j, \forall i,j \in [N]$, $y_i = y_j$ denote a correct pairing while $y_i \ne y_j$ denote an incorrect pairing.

Q5: ..., neither there are any experimental comparisons to that of CLIP. The experimental section is not in line with the initial claims of the paper.

Thanks for the question. To further illustrate the performance of the neural network-based CLIP, we conducted some new experiments to showcase the empirical performance. We hope the additional results in the official comments resolve your concerns.

Q6: What happens if we assume that the CLIP objective (2.1) has dot(\phi_I(x_i), \phi_t(x_j)) corresponding to the image and text embeddings. Are the results discussed in this paper hold without any loss of generality ?

The results hold without any loss of generality as we can directly write $\phi(x) = \mathbb{I}\\{x ~ \text{is an image}\\}\phi_I(x) + \mathbb{I}\\{x ~ \text{is a text}\\}\phi_t(x)$.

Q7: In 2.7, Is y_ti one hot encoding of the ith sample ?

We apologize for the confusion. In line with $y_i$, $y_{ti} \in \\{+1, -1\\}.$

Dear Reviewer pvdu,

As the author-reviewer discussion period ends soon, we would appreciate it if you could check our response to your review comments. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends. If our response resolves your concerns, we kindly ask you to consider raising the rating of our work. Thank you very much for your time and efforts!