When is an Embedding Model More Promising than Another?

We warmly thank Reviewer 4AaU for their detailed review and the effort they put into evaluating our work.

Questions.

1. Orthogonal models and unique information. Yes, a model could contain unique information and still be competitive while not predicting the others. Our method allows the discovery of such situations by studying the communities in the predictiveness graph: a model containing only unique information would appear as a disconnected component, which should prompt further investigation. A larger set and a more diverse set of models naturally reduce this risk. (See Sec E.4 for a study of the robustness of the method for different sampling of reference models Figure 22 and Figure 23). In molecular modeling, both GPT-GNN and ContextPred appear unique and disconnected, but both of these models also end up being the worst-performing models. In our experiments, we found that models containing only niche information did not perform well, suggesting that good models have to contain, to some extent, a minimal amount of common knowledge.

2. Evaluation of embedding models. Our method aims to evaluate generalistic embedders that could be used for a host of possible tasks for a data distribution of interest, thus evaluating foundational embedding models. As you pointed out, it is useful to identify interesting candidates to be used as starting points to evaluate some downstream tasks, but also, more generally, one might not want to rely on different embedders for different tasks because of the potential computational overhead and be interested in the more “generally good” embedder for their data distribution of interest. Pursuing these goals, our method provides the ranking of embedders and community analysis, uncovering similar embedders and orthogonal communities that might bear different types of information. An interesting direction would be to explore if the different clusters discovered by our method perform differently on different tasks, supporting the hypothesis that they capture different types of information (we provided a short analysis of this question in sec C.3.5 and sec D.3).

3. Diversity of reference models. We considered the diversity of architectures, training distribution, and training objectives, and we suppose that these differences lead to different information being captured. We do not rely on performances on downstream tasks (which would indeed be circular), in Sec. E.4, Figures 22 and 23, we study the robustness of our method to the sampling of reference models and the number of sampled models.
   In NLP, 10 models suffice to reliably achieve strong correlations (~ 0.90) between the informativeness score and the performances of the model, and in molecular modeling, while with very few models, the correlations are still high (above 0.8), with 10 models we reliably achieve a correlation above 0.9. We computed the correlation scores for different subsets of reference models and reported the average and standard deviation of the correlation score. We will include this discussion in the body of the paper.

4. Vision Taskonomy. It is indeed very interesting work that we shall include in our related work, and it also appeared extremely relevant to the follow-up work we were conducting!
   However, it remains unclear to us how to directly connect it to this current work. We identify two possible directions.

   1. Frame the objectives of comparing embedders using their proposed methodology by adapting the notion of tasks in the paper of Zamir et al.

   2. Apply the ordinal normalization and the taxonomy discovery procedure to the predictiveness matrix we get using the information sufficiency.

      Both approaches are non-trivial to execute as it is unclear how to reproduce the transfer modeling for each models/tasks (especially considering the different modalities used) and what would be the optimization constraints for the taxonomy discovery (they are looking for the taxonomy that minimizes the overall amount of training data required to train all models to produce a “scheduling of training”).

   Pertaining to your question, we ran eigenvalues analysis and spectral embedding of the embedders based on the similarity matrix to extract communities and evaluate embedders' proximities and found similar results as those reported in the community analysis presented in our work (See Figure 1, in the PDF page attached to the general comment). We will include those in the appendices.

We extend our thanks to Reviewer 4AaU for their thorough review of our work, and we hope our answers address all their concerns so they can reassess our work’s quality.

We would like to thank Reviewer UcaH for their detailed account of our work and the efforts they put into their review.

Weaknesses.

1. Estimation of $I_S(U \longrightarrow Z)$ (See PDF in general comment). For a given dataset $D$, we generate the embeddings $(u_i, z_i)$. We then fit a Gaussian mixture on the embeddings $(z_i)_i$ and a parametric Gaussian mixture on the embeddings $(z_i)_i$, parametrized by $(u_i)_i$, i.e. the means and covariance matrices are estimated by a small feed-forward network taking $u_i$ as input, minimizing the negative log-likelihood of the samples $(z_i)_i$. We then compute the corresponding entropies using those trained distributions (see additional PDF, for the detailed algorithm, we will add this algorithm alongside the detailed description to the core of the paper using the additional page allowed to address the reviewers’ comments). In addition, the code source to perform this estimation is available as part of the supplementary material submitted alongside the paper.

2. Choice of Reference Data. We focused on global evaluation with a large and diverse set of reference data that would be representative of the data distribution of most of the evaluated downstream tasks (ZINC dataset in drug discovery). In Section C.3.6, Figure 10, we used different reference datasets to evaluate our metric. We found that evaluating on data that are close to the ones of the downstream tasks produces better correlation. The most striking example is the performance on the IMBD classification task (evaluating the sentiment of a film review). When our metric is evaluated on the AG News dataset (a dataset of news articles) the correlation with that task is significantly lower than when evaluated with the overall common set or with the amazon polarity dataset, which consists of amazon reviews that are arguably close to film reviews. We plan to extend our work to practical settings to evaluate models on different reference sets to find the best embedder for a given modality or subdistribution.

3. Implementation details in the paper body. We will use the additional page allowed to include reviewers’ comments to include details on the $I_S$ estimation and discuss our different choices (number of models, choice of the median, and embedding dimensions) to give more intuitions to the reader.

4. Practical utility. The main use case for our method is to compare different foundation models as it focuses on how well data are separated in the embedding space. In NLP, foundation models are trained on vast amounts of very diverse textual data, but it is unclear if it embeds well certain types of data (longer texts, simple reviews or question/answer passages etc…). In molecular modeling, since tasks are often extracted from wet-lab experiments, obtaining these labels is expensive, time-consuming, and they are often noisy. In molecular embeddings, the Information sufficiency graph and the identification of communities is helpful to see how the information encoded by a 3d model is inaccessible to the 2D models so far, even though these 2d models were trained to incorporate this information (see Sec D.2). We are currently working on applications of this work as follow up work to evaluate foundational models in medical computer vision: the goal is to evaluate the quality of the embedding models for the different modalities and data distribution they are supposed to handle.

5. Computational costs. We discussed the computational cost of our method in Sec. E.5. Our method is actually surprisingly cheap computationally. Computing the informativeness of a model requires computing $N$ information sufficiency, where $N$ is the number of reference models, each requires fitting two mixtures of Gaussians (the marginal and the conditional). In practice, if the embeddings are precomputed (~ $150$k samples per embedder), it took us less than $6$hours on a single GPU to compute the $45 \times 45$ (all the pairs) information sufficiency to evaluate all the text embedders. For reference, evaluating the whole MTEB benchmark for a model takes hours.

Questions.

1. Learning kernels. Yes, we need to fit a kernel per pair of embedders. It grows quadratically in terms of the number of embedders to build the whole graph. However, the kernels are small (3 layers feedforward networks to evaluate the conditional distributions), and the overall evaluation is quick, as discussed in our computational costs analysis (Sec E.5).
2. Baselines. We included 3 additional baselines: the model sizes (as you suggested), the embedding dimensions, and a less naive one: the reconstruction error of a cross-encoder, a simple feedforward network trained to transform an embedding into another directly, and we use the average l2 loss as a score. Our method consistently outperforms all baselines, reaching higher correlation scores on all considered benchmarks (using the l2 score, the Spearman correlation achieves -0.84 and -0.9 in NLP and molecular modeling, respectively, compared to 0.9 and 0.94 with our method; more details are provided in the general comment).

We renew our thanks to Reviewer UcaH for their thorough review of our work, and we hope our answers addressed all their concerns so they can re-assess our work’s quality.

We thank Reviewer o2cb for their review and the interesting questions they raised. We do our best to answer them below.

Weaknesses

1. Information sufficiency and deficiency. (See Official comment below for the proof) We can establish the following connection between deficiency and mutual information, where information sufficiency serves as an estimate:

$$\\delta(P_{U|X} \rightarrow P_{Z|X} ) \ge\inf_M \\mathbf{E}[\ell(X,M)] \\ge R_{X,\ell}^{-1}(I(Z,X;U))$$

Where $R_{X,\ell}$ is the rate-distortion function for $X$ and a loss function $\ell(\cdot)$. This is a non-increasing function in mutual information: when mutual information is low, the deficiency is high. This suggests that the mutual information $I(Z;U)$ should be as large as possible as a necessary condition to achieve lower deficiency. Due to character limits, the proof is provided in the following comment. We will include it in the final version of the paper.

2. Estimation of $I_S(U \longrightarrow Z)$. For a given dataset $D$, we generate the embeddings $(u_i, z_i)$. We then fit a Gaussian mixture on the embeddings $(z_i)_i$ and a parametric Gaussian mixture on the embeddings $(z_i)_i$, parametrized by $(u_i)_i$, i.e. the means and covariance matrices are estimated by a small feed-forward network taking $u_i$ as input, minimizing the negative log-likelihood of the samples $(z_i)_i$. We then compute the corresponding entropies using those trained distributions. (See additional PDF, for the detailed algorithm, we will add this algorithm alongside the detailed description to the core of the paper using the additional page allowed to address the reviewers’ comments). In addition, the code source to perform this estimation is available as part of the supplementary material submitted alongside the paper.
3. We will further improve the pagination and organization of the tables and figures.

Questions

1. Transitivity. The example provided in the review: If $I_s (U→Z)>=I_s (W→Z)$ and $I_s (W→Z)>= I_s (X→Z)$, then $I_s (U→Z)>=I_s (X→Z)$ is always true by the transitivity of the “>=” relationship as boils down to $(a \geq b \text{ and } b \geq c) \implies a\geq c$. A less straightforward transitivity property exists for the deficiency, indeed we can show that $\delta(P_{A|X}\rightarrow P_{C|X}) \leq \delta(P_{A|X}\rightarrow P_{B|X}) + \delta(P_{B|X}\rightarrow P_{C|X})$. This relation simply states that the reconstruction error of an embedder C by an embedder A is lower than the reconstruction of any embedder B by A, and the embedder C by B. We include the proof at the end of this rebuttal and will add it to the final version of the paper

2. Choice of Gaussian Mixtures. Gaussian mixtures are known to be universal estimators of densities, and to our knowledge, the KNIFE paper corresponds to the state of the art for this kind of information-theoretic estimations. Another option is to directly fit cross-encoders (feedforward networks) using the MSE loss and use this reconstruction loss as score of informativeness. Our method consistently outperforms this reconstruction loss strategy, reaching higher correlation scores (a Spearman correlation of 0.84 compared 0.9 in NLP and 0.9 compared to 0.94 in molecular modeling on all benchmarks) on every benchmark we considered (See attached PDF and general comment for full results and additional baselines).

3. Reference set requirements. The requirement is that the dataset used is sufficiently large to correctly represent the distribution of data that will be presented in practice (the ZINC dataset for molecules for example). In Section C.3.6, Figure 10, we evaluate the impact of evaluating the I_S on different subsets of data on the correlation with the downstream tasks performance. Evaluating broader data tends to help, but the performance is even stronger if the data used to evaluate the I_S are close to the data used for the downstream tasks. For example, evaluating $I_S$ on the Amazon polarity gives good insights on the performance on the imdb dataset whose task is related: the goal is to evaluate the sentiment about a movie.

4. Yes, this is correct. We will correct this typo.

5. Tightness of the bound. It might be the case, but we are not interested in actually estimating the bound but rather in the relationships between the terms of said bound since our goal is to compare embedding models. Our goal was to highlight the connection between the deficiency and the different risks: controlling and comparing the deficiency is still a good idea even if the bound is not tight.

We extend our thanks to Reviewer o2cb for their thorough review of our work, and we hope our answers address all their concerns so they can reassess our work’s quality.

---

Proof: Transitivity of the deficiency It is easy to check that Eq 1:

$$\\delta(P_{A|X}\\rightarrow P_{C|X}) =: \\inf_{M\\in \\mathcal{K}(C|A)} \\| M\\!\\circ\\!P_{A|X} - P_{C|X} \\|\_{\\text{TV}} \\leq \\inf_{M\\in \\mathcal{K}(C|B)} \\inf_{ M'\\in \\mathcal{K}(B|A)} \\| M\\!\\circ (M'\\!\\circ\\!P_{A|X}) - P_{C|X} \\|_{\\text{TV}}.$$

On the other hand, for any Markov kernels $M\in \mathcal{K}(C|B), M'\in \mathcal{K}(B|A)$:

Eq 2:

$$\\| M\\!\\circ M'\\!\\circ\\!P_{A|X} - P_{C|X} \\|\_{\\text{TV}}= \\| M\\!\\circ M'\\!\\circ\\!P_{A|X} - M\\!\\circ\\!P_{B|X} + M\\!\\circ\\!P_{B|X} - P_{C|X} \\|\_{\\text{TV}} \\leq \\| M\\!\\circ M'\\!\\circ\\!P\_{A|X} - M\\!\\circ\\!P\_{B|X} \\|_{\text{TV}}+\\| M\\!\\circ\\!P\_{B|X} - P\_{C|X} \\|\_{\\text{TV}},$$

and by data processing inequality on the TV norm: Eq 3:

$$\\| M\\!\\circ M'\\!\\circ\\!P_{A|X} - M\\!\\circ\\!P_{B|X} \\|\_{\text{TV}} \\leq \\| M'\\!\\circ\\!P_{A|X} - P_{B|X} \\|\_{\text{TV}}.$$

The desired inequality follows by applying inequality Eq.3 to Eq.2, and then taking the infimum at both sides over all Markov kernels $M\in \mathcal{K}(C|B)$, $M'\in \mathcal{K}(B|A)$, and using inequality Eq.1.

This is a complement to our answer to Reviewer o2cb regarding the connection between our estimator and the deficiency.

Review of the Distortion-Rate Function

The rate-distortion (RD) function of a random variable --the source-- $X$ for a given distortion function $\ell(\cdot, \cdot)$ is defined as [Cover 2006]

$$R_{X,\\ell}(D) \,\triangleq \inf_{\rule{0mm}{4.3mm}\substack{p_{\widehat{X}|X}:\\ \mathbf{E}[\ell(X,\widehat{X})] \leq D}} I(X;\\widehat{X}).$$

Furthermore, we assume that there exist $D>0$ such that $R_{X,\ell}(D)$ is finite. We denote the infimum of those $D$ by $D_{\min}$ and $R_{\max}\triangleq R_{X,\ell}(D_{\min})$ (or, more precisely, $R_{\max}\triangleq \lim_{D\rightarrow D_{\min}+}R(D)$). The following properties (see [Lem.~1.1, Csiszar 1974]) of the RD function will be used in what follows.

Theorem 1: The RD function $R_{X,\ell}(D)$ is a non-increasing convex function of $D$ on the interval $(D_{\min}, \infty)$. It is monotonically decreasing on the interval $(D\_{\\min},D\_{\\max})$ and constant with $R_{X,\ell}(D)=R\_{\min}$ on $[D_{\max},\infty)$ (here $D_{\max}=\infty$ and $D_{\min}=0$ are possible). The inverse function $R_{X,\ell}^{-1}(r)$ is well defined on $(R_{\min}, R_{\max})$ and monotonically decreasing.

The inverse function $R_{X,\ell}^{-1}(r)$ is known as the distortion-rate (DR) function of the random variable $X$ for the given distortion function $\ell(\cdot, \cdot)$.

Deficiency and Information Sufficiency

Assume two Markov (or transition probability) kernel between $\mathsf{U}$ and $\mathsf{Z}$ which is a mapping $M : \mathcal{B}(\mathsf{Z}) \times \mathsf{U} \rightarrow [0,1]$, and between $\mathsf{Z}$ and $\mathsf{U}$ which is a mapping $M^\prime : \mathcal{B}(\mathsf{U}) \times \mathsf{Z} \rightarrow [0,1]$. From the embedder definition, we have the Markov chain $Z \leftrightarrow X \leftrightarrow U$.

We begin by the deficiency $\delta(P_{U|X} \rightarrow P_{Z|X} )$. Let us define a suitable distortion function:

$$\ell(x, M) \triangleq \\| M \circ P_{U|X=x} - P_{Z|X=x} \\|_{\text{TV}}.$$

From which, it is easy to check that

Equation 1

$$\delta(P_{U|X} \rightarrow P_{Z|X} ) \triangleq \inf_M \\| M \circ P_{U|X} - P\_{Z|X} \\|_{\text{TV}} \geq \inf_M\mathbf{E}[\ell(X,M)],$$

where the last inequality follows by replacing the supremum over $x\in \mathsf{X}$ by the expectation. Using the data processing inequality and the definition of the RD function, we obtain the following bound for any Markov kernel $M$:

Equation 2

$$I(Z,X;U) \geq \inf_{\\rule{0mm}{4.3mm}\substack{p_{\widehat{X}|X}:\\ \mathbf{E}[\ell(X,\widehat{X})] \leq \mathbf{E}[\ell(X,M)]}} I(X;\widehat{X}) = R(\mathbf{E}[\ell(X,M)]).$$

For $\mathbf{E}[\ell(X,M)]\in (D_{\min},D_{\max})$, we can invert the RD function, and thus we obtain from it the fundamental bound $R_{X,\ell}^{-1}(I(Z,X;U))\leq \mathbf{E}[\ell(X,M)]$ or, equivalently,

Equation 3:

$$\delta(P_{U|X} \rightarrow P_{Z|X} ) \geq \inf_M \mathbf{E}[\ell(X,M)] \geq R_{X,\ell}^{-1}(I(Z,X;U)) ,$$

which follows from inequality Eq.1. For $\mathbf{E}[\ell(X,M)] < D_{\min}$, Equation 2 reduces to $I(Z,X;U) \geq +\infty$ which shows that in order to achieve an expected distortion below $D_{\min}$ the random variables $(Z,X,U)$ must have a joint distribution that is not absolutely continuous with respect to the product of their maginal distributions $(Z,X)$ and $U$. For $\mathbf{E}[\ell(X,M)] \geq D_{\max}$ we obtain the trivial bound $I(Z,X;U) \geq 0$. The lower bound in Equation 3 is a non-increasing function in the mutual information. This suggests that the mutual information $I(Z;U)$ should be as large as possible as a necessary condition to achieve lower deficiency. Similarly, we can obtain that the symmetric bound:

Equation 4

$$\rule{0mm}{5mm}\rule{3mm}{0mm} \delta(P_{Z|X} \rightarrow P_{U|X} ) \geq \inf_{M^\prime} \mathbf{E}[\ell(X,M^\prime)] \geq R_{X,\ell}^{-1}(I(U,X;Z)).$$

We will incorporate these suggestions into the final version of the paper.

References

T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, New York, NY, 2nd edition, 2006.

I. Csiszar, “On an extremum problem of information theory,” Studia Scientiarum Mathematicarum Hungarica, vol. 9, no. 1, pp. 57–71, 1974.

We warmly thank Reviewer id3c for their reviews.

Weaknesses:

1. We refer the reviewer to Questions below.
2. Computational costs. Our method is cheap computationally. We quickly discussed the computational cost of evaluating our method in Sec. E.5. Computing the informativeness of a model requires evaluating $N$ information sufficiency, where $N$ is the number of reference models, each requires fitting two mixtures of Gaussians (the marginal and the conditional). With precomputed embeddings (~ $150$k samples per embedder), it took us less than $6$hours on a single GPU to compute the $45 \times 45$ (all the pairs) information sufficiency to evaluate all the text embedders. As a result, in our examples, computing the informativeness score of a model takes less than 15 minutes.

Questions:

1. Estimation of $I_S(U \longrightarrow Z)$ (See PDF in general comment). For a given dataset $D$, we generate the embeddings $(u_i, z_i).$ We then fit a Gaussian mixture on the embeddings $(z_i)_i$ and a parametric Gaussian mixture on the embeddings $(z_i)_i$, parametrized by $(u_i)_i$, i.e. the means and covariance matrices are estimated by a small feed-forward network taking $u_i$ as input, minimizing the negative log-likelihood of the samples $(z_i)_i$. We then compute the corresponding empirical entropies using those trained distributions (See the attached PDF file in general comment for the detailed algorithm; we will add it alongside the detailed description to the core of the paper using the additional page allowed to address the reviewers’ comments). In addition, the code source to perform this estimation is available as part of the supplementary material submitted alongside the paper.
2. Better estimators of the deficiency. The information sufficiency is admittedly a proxy for the actual deficiency we wanted to evaluate. Our decision was guided by tractability and numerical stability arguments (we expand more on this in our answer to Reviewer o2cb). There is still work to be done to find a proven estimator. For example, we were able to show a more direct connection between the deficiency and the critic loss of a GAN. However, we could not make it work in practice: this more grounded approach is too numerically unstable and leads to very poor results (e.g., the information sufficiency yields correlations of $0.90$, while this more grounded approach yielded at most $0.70$). This opens an interesting line of research for the near future.
3. Performance for classification tasks and others. In Figure 3, table (a), the Kendall-Tau correlation between our metric and the performance on downstream tasks is significantly higher ($0.73$ ) for the classification tasks than for the other type of tasks ($0.69$ at best and closer to $0.65$ globally). Our interpretation of these results is that classification tasks rely on the training of an additional classifier on top of the embeddings, whereas the other tasks only rely on the dot product between embeddings to make decisions (retrieval, similarity and clustering). Our theoretical results give insight into what is doable when learning the best classifier possible, but they do not guarantee that the dot product / l2 distance in the embedding space captures any useful semantics.

Source code. We released the source code for this project: it is available as supplementary material attached to the submission and at [https://anonymous.4open.science/r/emir-B8D3/Readme.md]. A public repository will be shared upon acceptance. Our implementation only relies on pre-computed embeddings and thus can directly be used for any domain as long as they have been dumped. We will make sure to expand the Estimation Method section to provide more details on how we estimate information sufficiency in practice, and we will publicly release the library and code as a practical library to estimate our metric.

We hope we were able to alleviate your concerns and that our answers will allow you to improve your already positive assessment of our work, and we remain available for any additional information.