Learning Task-Agnostic Representations through Multi-Teacher Distillation

We thank reviewer 3Aq4 for their thorough and insightful review. We appreciate that they found our method principled and our experimental setup thorough and convincing. We added additional baselines to our vision experiments and we are thankful to the reviewer for pointing out the unclear reference to the MSE training stability, as well as the mistaken reference in the conclusion. We will correct these in the revised version of the paper.

MSE distillation stability

We would like to clarify that our reference to the instability of Mean Squared Error (MSE) was drawn from previous works in the field of reinforcement learning $1, 2, 3$ , where stability is context-specific. Our intention was to use this as motivational background for our work, rather than making a direct claim about stability within our own research. We acknowledge that this distinction was not clear in our initial presentation, and we will revise those references. Most importantly, we mistakenly referenced stability in our conclusion, which we will correct in the revised version.

Additionally we will add a dedicated section comparing the MSE loss and our NLL loss (with the training curves):

We observed that when training with the MSE loss, the loss reaches a minimum in only a few epochs (~40), but the distilled students achieve lower performances on downstream tasks. This could be due to the fact that the NLL loss is more expressive, and harder to optimize (see below). As a result the student learns more informative features compared to when trained with the MSE loss. (training curves will be included)

We can provide a theoretical insight to explain this phenomenon. Training using the negative log-likelihood over a Gaussian kernel is a simple generalization of the MSE. For a given multivariate Gaussian kernel parameterized by \mu and \Sigma, we have:

\-log (p\_{\\mu, \\Sigma}(x)) \= \-\\log C \+ \\frac12 \\det \\Sigma \+ \\frac12 (x-\\mu)^T \\Sigma^{-1} (x-\\mu)

Minimizing the MSE loss boils down to minimizing this equation over $\\mu$ only, with \\Sigma \= I. Therefore, minimizing the negative log-likelihood of a Gaussian kernel is strictly more expressive than minimizing the MSE directly, which could account for the performance gains we observe.

$1$ Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
Jesse Farebrother and al. 2024

$2$ A Distributional Perspective on Reinforcement Learning, Marc G. Bellemare and al. 2017

$3$ Regression as Classification: Influence of Task Formulation on Neural Network Features
Lawrence Stewart and al. 2022.

Extension to multi-modal setting

The method can indeed be extended to a multimodal setting, where modality specific encoders would share the same backbone to embed different types of data. The challenge we would like to address is training a distillation model when limited cross-modal labels are available, to enable pretraining on larger datasets. So far, we did not observe a significant advantage compared to single modality training, and we are running additional experiments to answer the question: “How much cross-modal information is required to distill multi-modal representation that outperform the single modality ones?”

Additional baselines

We have added relational KD (RKD) $Park et al., CVPR 2019$ , Correlation Congruence with Gaussian RBF, and Bilinear (normalized features) kernels (CC-grbf, CC-Bilinear) $Peng et al., ICCV 2019$ for vision. As shown in the following table, our method (NLL) has better performance compared to the additional baselines. Our intuition for the difference of accuracy is that both RKD and CC are proposed to work alongside the task loss, which could be an important signal for their optimization in practice.

Presentation Issues

Thank you for your thoroughness. You are absolutely right, we’ll update the figures with vectorized versions, and we’ll fix the formatting inconsistencies you pointed out.

Thank you again for your valuable feedback and suggestions, which have significantly improved our paper. Especially, thank you for pointing out the lack of clarity in our reference to the instability of MSE value estimation in reinforcement learning, that we have corrected.
We hope we addressed all of your concerns so that you might consider raising the score of your review.

We thank reviewer zNKm for their review, and we are glad they appreciated our work, and believe the paper should be accepted. We remain available to answer any new question if needed.

We thank reviewer dFG5 for their insightful review and are pleased that they found our work interesting. Following the reviewer's advice, we will include a section dedicated to the limitations of our approach, referencing some results of other sections.

Limitation discussions

While we discussed some limitations in the appendices and experimental section, we agree our work would benefit from a dedicated limitation section. We will add a specific section covering the following limitations of our method:

• Application Scope: Our method develops student embedding models primarily for diverse, unknown tasks. For single pre-defined tasks, task-specific distillation approaches might be more suitable.
• Overhead in Distillation: Like any distillation setting, especially multi-teacher distillation, there is an overhead due to the distillation of large teachers into a smaller model. This overhead can be computational (if teacher embeddings are obtained by running inference for each training) or memory-related (if teacher outputs are precomputed and used during training). We opted for the latter as it drastically speeds up student training, initially storing all necessary embeddings on disk. For text (our most computationally demanding application), this amounts to approximately 100GB of embeddings for the largest teacher.
• Teacher Quality: Our approach requires high-quality teachers relevant to the downstream tasks. In Section D.4, Table 26, we explore the impact of training a student embedder with task-specific teachers (classification, object detection and segmentation). We demonstrate that while task-specific teachers may offer limited benefits outside their domain, they do not negatively impact the students’ learning when used alongside task-relevant teachers.
• Structural relationship: As pointed out in Section 4.2, our metric only optimizes the mutual information between the student and the teachers, it does not directly enforce any structure on the embedding space, which could harm the performances of our models for clustering tasks for instance. For textual embeddings benchmark, we observe clear gains in classification tasks (where a small classifier is trained on top of the embeddings), but the gains are less clear for clustering and STS tasks that rely on the dot product between embeddings to assess text similarity (See full results in App. C.2.2).
• Representative dataset: To effectively embed data for future tasks, our method requires a training set that is representative of the data distributions of these tasks. This limitation, however, is common to all embedding models, which all require a diverse and representative dataset for training.

Comparison of 8-teachers to 2-teachers on the BBB benchmark.

The student trained in molecular modeling with 8 teachers indeed shows slightly lower performance on the BBB (blood-brain barrier) benchmark compared to students trained with 1 or 2 teachers. The BBB benchmark data distribution significantly differs from our training set, representing a domain shift compared to the training set of the students. Furthermore, it is one of the benchmarks where teacher performance is most tightly packed, with variations within 1.45 times the average standard deviation of the results. This could explain why training with 8 teachers performs closely to 1 or 2 (the differences in AUROC being half the standard variation), as all teachers demonstrate comparable performance on this specific task. We believe this explains the slightly lower average performance of the 8-teacher student compared to the 1 or 2-teacher students.

Thank you again for your valuable feedback. We believe these clarifications and updates will address your concerns and enhance the quality and relevance of our work. We remain available to clarify any additional points if needed.

We thank reviewer Aw1B for their thorough review, and we are glad they appreciate the novelty in our proposed method. We politely disagree our experiments focused on 'really small models and small benchmarks', and we aim to provide a clear justification below.

Small Models

We trained models up to 300M parameters for textual embeddings, while this is not particularly large (<1B), it is a standard size for embedding models in text. Contrary to text generation where clear performance gain can be seen when using very large models, performances of embedding models in text can achieve very competitive scores, while having fewer parameters (e.g., the Stella-500M model outperforming several 8B models on the MTEB). This scale is comparable to other well-established and widely used embedders, such as Stella and GIST, which are recognized for their strong performance in the field.

Small benchmarks

We politely disagree with this statement, across the three modalities evaluated in our study we conducted comprehensive assessments on a total of 71 widely used datasets. These included 6 datasets for computer vision, 32 for molecular modeling, and 33 for text analysis (ie the Massive Text Embedding Benchmark, which constitutes the reference for textual embedders evaluation).

Motivation of the task-agnostic distillation

In our introduction, we aim to motivate task-agnostic distillation in two steps. First, we highlight the development of embedding models, which are inherently task-agnostic. These models compress objects into numerical representations, thereby facilitating a wide array of downstream tasks. Next, we argue that the diversity of embedding models available in each field can be leveraged to build an embedder that benefits from these diverse representations through distillation.

We will extend the paragraph motivating the use of embedding models by mentioning their current applications in a wide range of scenarios, including classification, clustering, and information retrieval. We will also emphasize their key advantage in terms of computational efficiency, particularly in scenarios with limited labeled data.

'The paper is a bit outdated'

We are not sure what part of the paper you are referring to when you say it is a bit outdated. Could you be more specific so we can update and improve our work accordingly? Nevertheless, we would like to make a few remarks as the field of text embedding models have gained a lot of focus in research lately:

• We chose our baselines based on their leading performance on the MTEB benchmark at the time of submission. Notably, Stella 400m v5 was the top-performing model in its weight category, only recently surpassed by the Qwen embedders released on June 15th 2025.
• We compared our students against the best models in each category, including GIST models, which were among the highest-performing embedders across most weight categories until early 2025.

In any case, we will update the MTEB benchmark tables in our paper to include the most recent results published.

Choice of baselines

We did not only compare our approaches with multi-teacher distillation approaches. While our analysis mainly focuses on the comparison of our method with multi-teacher distillation methods, we also compared its performances with several embedders for each modality (of the same weight category). The objective of our experimental section is to validate that our distillation approach is efficient in compressing the information of several teachers into a smaller student, thus our focus on distillation baselines. It is hard to provide a fair comparison of methods when comparing with other embedders since they have been trained with widely different settings, datasets, infrastructures and training objectives, whereas we provide distillation comparison in a valid controlled setting for fair comparisons to answer our initial scientific question.

We hope we have successfully addressed all your concerns, and we would be grateful if you could reconsider your score based on our revisions. Thank you again for your insightful review.

We thank reviewer Aw1B for their involvement in the rebuttal process.

We want to insist that we do compare with the most recent/modern embedders of similar sizes at the time of the submission in the main part of the paper (only the best one for each weight categories and we provide the full MTEB results in appendix C.2) and provide such a head-to-head comparison. And indeed, our method produces models that outperform all modern embedders in their size categories (for fair comparison), thus suggesting that the multiteacher setting indeed provides significant advantages. For comprehensiveness' sake, we include in this rebuttal the most recent results from the MTEB for the biggest/best models. (We had to trim the full table for it to fit in this answer, but we can provide any part of that huge table) compared to our own models. The most recent one is the Qwen 600M embedder, released only a few weeks ago.

We agree there is no harm in comparing with larger models (we provide the results here as well) however, it is important to keep in mind that it is not an apples-to-apples comparison. Models of different sizes and computational costs have different applications. Showing that we can achieve higher information density using our models has practical application for low-resource or on-edge deployment settings for which larger models are impractical.

Our medium model (109M) parameters are on par with models 5 times its size (Average performance 80.23), only outperformed by Stella 400M (still the best model of its category released early 2025 and included in the paper) and KALM (494M). The best performing and most recent model by far is Qwen 600M, which only outperforms our models by 5 points. Only models above 1B parameters achieve significant gains over our medium (109M parameters) model. If you have any additional specific model in mind that you deem more recent, please let us know we will add it if it's present on the MTEB benchmark. We will update the final version of the paper with the most recent version of the MTEB.

I thank the authors for their comments.

My biggest point of disagreement so far is that this paper does not compare its approach with modern embedding models. And that is the reason why I mentioned that the paper might be a bit outdated, since the embedding models that the paper compares to are increasingly outdated. Even if the authors feel that comparing it with modern embedding models is not a fair comparison, it would still be good to have those experiments in the paper. And that might directly question the motivation of the paper. The question is, "In light of how well modern embedding models perform, do we really need a multi-teacher distillation approach to train embedding models? And if so, could you motivate it by including head-to-head comparisons with modern embedding models?"

What's the harm in comparing it with models with a larger size and a larger context window?

Thank you for the updated experiments. I would still like to keep my current score, as i believe that the role of multi-teacher distillation is increasingly being replaced by more powerful models. While the authors focus specifically on tiny to small models, that are sub 0.5 to 1 B parameters (mostly sub 0.5B params), the SOTA embedding models are largely > 1B params. The cost of using these models is already very low and further going down. Further, it can be safely extrapolated from the current table that almost all, if not all, models of size > 1B params will outperform the Multi-Teacher Distillation strategy presented in this paper.