Conjuring Semantic Similarity

We thank the reviewer for their valuable feedback and suggestions.

Why do we need a new similarity metric?

This is because no existing metric is satisfactory (Stein et al., 2024), thus the evaluation of text-conditioned diffusion models remains an active area of investigation. In fact, no metric even exists to evaluate semantic similarity and alignment in text-conditioned diffusion models. Our work is the first to do so, by adding a new dimension (images) to the problem that had never been considered before. Having this also opens up avenues for evaluating semantic alignment with human annotators, as demonstrated in our experiments.

In paragraph 2, the authors provide some arguments about why comparing images is easier for humans than comparing text due to language and knowledge differences. But, that argument does not hold for models since the similarity between texts is often calculated by a single model. So, there is no difference in knowledge or judgment between models.

Indeed semantic spaces are uniquely defined for each individual model. However, this is a separate point from what we wished to convey in the last line of paragraph 2 -- that for humans, unlike images, comparing text requires a stronger shared knowledge base (for instance of the language in which the text is written). Nevertheless, we admit this sentence is debatable, and are willing to remove it from the final revision.

Moreover, even if this limitation is actually valid for text similarity measures based on dense vectors, it also applies to the proposed similarity metric. After all, just like there is model that is creating the dense vector, there is a model that is doing the denoising and image generation. So, there is still a model involved with all the specific and often unknown limitations and biases that come with any model.

The fact that semantic similarity is model/human-specific is precisely the motivation of our work -- which is to be able to quantify how semantically aligned different (image generation) models are with that of human annotators. We elaborate further in our response to W2 of Reviewer rCYr.

As the authors mentioned in Line 312, these generative models rely on some encoder to encode the text. So, all the limitations of the encoder model (i.e., cosine similarity between dense vectors) also apply to the proposed similarity metric. So, why should we add the extra level of complexity?

The "extra level of complexity" is actually a feature of our method: We perform comparison in the semantic space learnt by the generative model, rather than the text encoder itself. While, as mentioned in L312 and rightfully pointed out by the reviewer, the representations learnt are bottlenecked by the text encoder, this does not mean that all the information captured by the encoder is faithfully transferred to the generative model. We refer to the example given in the response to Reviewer rCYr -- a model trained to generate the same image for all text inputs can have a well-aligned text-encoder, but it would be inappropriate to conclude that this generative model is semantically well-aligned with humans.

Even beyond the motivation, the experimental results do not suggest that the new similarity metric is consistently better than the vector-based methods. The most interesting comparison is the performance of the proposed similarity metric compared to the performance of the text encoder that is used in the stable diffusion generative model, which is clip-vit-14. Comparing these two, it seems like in just three out of the seven datasets, the new similarity metric performs better, which is less than half the times.

I also have an issue with the lack of a baseline to measure the merits of the proposed similarity metric. This paper proposes two different concepts. One is the notion of text similarity through images, and the second is the proposed similarity metric to accomplish this.

The vector-based methods are included as points of reference, rather than for purposes of direct comparison. Not only are they used to evaluate completely different models, but none of them can be used to capture semantic similarity for these text-conditioned image generative models.

What can be directly compared, however, are the new baselines included in the revision. We refer the reviewer to the updated results, where we show that our method outperforms all comparable baselines across all datasets.

We thank the reviewer for their detailed feedback and their interesting ideas on extending our work.

The empirical experiments are too weak. On the one hand, the performance is not as good as the previous methods. The baselines listed are slightly outdated. It would be interesting if OpenAI ada embeddings, Llama 3.1, and Gemma can be compared. Surpassing BERT-CLS is not very convincing.

Our method is in fact the first work that evaluates this aspect of semantic alignment for text-conditioned diffusion models. Consequently, there are no proper baselines, and comparison with text-based methods in Table 1 is somewhat reductive as we aim to explore a different dimension - the visual one. Indeed there is much more evaluation method to be done to probe the value of visual representations for comparing textual semantics, which is well beyond of the scope of a single conference paper that introduces the concept.

However, what can be considered comparable baselines are the new methods we included in Table 1, where we tested other possible approaches to extract semantic similarity from diffusion models. Our approach, which is theoretically grounded by considering the JSD between SDEs induced by various textual expressions, convincingly outperforms these baselines across all datasets.

On the other hand, STS may not be the best arena for the novel method the authors proposed, since there are a lot of examples related to paraphrasing, instead of conceptual relevance. It would be great if the authors study a specific slice that is more relevant to the idea, e.g., examples with a not-complex scene and the difference mainly comes from the concepts. Knowledge graphs (e.g., COMET, Atomic) or WordNet depth distance may provide better comparative performance or qualitative examples.

While we included some WordNet-based evaluations in Figure 2, we have added a stronger analysis at the end of Sec 4.4 studying semantic similarity between words, which can be stratified based on the parts of speech from which they originate. We thank the reviewer for their excellent suggestion!

This paper can be linked with more recent advances in this area in NLP, e.g., C-STS (https://arxiv.org/abs/2305.15093) discusses the conditional semantic textual similarity in addition to STS; Instructor (https://github.com/xlang-ai/instructor-embedding) and PIR (https://arxiv.org/abs/2405.02714) discuss the changes in semantic embeddings under different instructions. The related work and experiment suite can help further improve the draft in discussing the semantic similarity.

We thank the reviewer for the references, and will add a discussion to the paper on how our paper can extend to these areas after perusing them carefully.

It would be good if further content is added to the paper especially since there are 7.5 pages in the current draft. For example, the authors can include experiments on potential performance improvement over various downstream tasks: Z-LaVI (https://arxiv.org/abs/2210.12261), for example, explores how visual imagination helps improve model performance on tasks such as word sense disambiguation in a zero-shot manner. Similar tasks can be included in the draft.

Indeed our work opens up many possibilities for evaluating the semantic alignment of image generative models in future work. We hope that our new experiments and analysis should address the reviewers concern regarding paper content, while the reviewer’s valid and interesting idea will be explored in future work.

Is there a specific set of STS that you found the proposed method works?

A low score implies low level of semantic alignment, while high scores simply imply high levels of semantic alignment of the specific diffusion model under evaluation. Compared, however, to naive diffusion baselines for extracting semantic similarity, we have updated Table 1 to show that our method greatly outperforms baselines on all sets of STS and SICK-R. The newly added Table 2 also analyzes alignment on word pairs, stratified across different parts of speech. Interestingly, we found that semantic relations between nouns are largely preserved across the diffusion training process, but this comes at the expense of adjectives and verbs.

We are very grateful for the reviewer's constructive feedback, and incorporated their suggestions into our revision. We address them in detail below:

While the authors cited weaknesses with the approach, the authors could do a better job motivating possible applications opened up by their method.

We have modified our paper, especially the last part of Sec 5, to incorporate this suggestion, thank you.

The paper qualitative experiment is a little shallow. Expanding it and doing error analysis ( where does the method perform well/poorly ) on results would help add more clarity into the method.

To address both the qualitative and quantitative aspect, we have incorporated a range of new experiments. We refer the reviewer to the overall comment for the full changelog.

The paper could have done an ablation compared their symmetric JSD approach to just using KL-Divergence to show the boost obtained.

We thank the reviewer for the suggestion and incorporated this into Table 1. However, we stress our goal is not to beat the benchmarks for semantic similarity, but to explore an alternate dimension, not (yet) well represented in existing methods, in the context of image generative models. While KL-divergence is also a valid method to do so, it is fundamentally asymmetric and fails to capture the fact that semantic similarity (at least for humans) is largely a symmetric property.

The interpretability angle of the paper is also a little lacking and under explained. Outside of the few examples given (one in the paper and two in the appendix ), is there anyway to automate interpretability results or use them for error analysis in a useful way.

Our method produces both a semantic similarity score, as well as an interpretable explanation in the form of generated images. The score can be used for analyzing alignment with human annotations in a quantitative manner. The justification for this score can then be visualized if desired, as shown in the qualitative evaluations.

One possible way for error analysis would be leveraging additional meta-data in the evaluation data. We have added an experiment on this on the SimLex-999 dataset to quantify errors across different POS (part of speech) classes in Sec 4.4, and thank the reviewer for their valuable suggestion!

The last line of the 2nd paragraph ( about pixel values not depending on distant knowledge or cultural background ) seems debatable/unnecessary since a text passage (about democracy, celebration, a feast, etc ) could be portrayed visually in different ways depending very much on knowledge/cultural background while still referring to the same semantic concept

We wished to convey the point that text needs to be interpreted before it can be semantically, or even synthetically, compared, while images require significantly less processing to compare directly. However, we do acknowledge that this is highly debatable, and could remove it in the final revision.

on line 238, "denoising with either y1 and y2" is a little confusing in that "either" seems to imply an exclusive OR while "and", Algo 1 and the definition d_ours(y1,y2) show the need for both? I could be mistaken, but I think the later is the case, in which case you should change "either" to be "both"

Indeed the latter is the case. "Either" was used to refer to what is being done in a single Monte-Carlo step, but we realized that using "both" in the context presented in L238 might be more appropriate. We have corrected this in the revision, thank you for catching this.

The line starting at 257 was a little unclear to me. Specifically "we note that they do not usually correspond exactly" ? nit: 313 "is" -> "are"

Fixed, thanks!

We thank the reviewer for their helpful feedback, and address concerns below:

W1: The premise of the paper–obtaining a similarity measurement based on the imagery texts evoke–is unique and interesting, but I don’t understand what use-case is it tailored to address. If the use-case is no different than measuring similarities in text-only environments, I don’t see why it is preferable over methods that are inherently text-only, are easier to scale, and are equipped to represent abstract notions, which are difficult to visualize. I find that conclusion is supported in section 4.2 and table 1, too. The paper will be improved if such motivation will be clarified, with an experiment to demonstrate this use-case.

Our work aims to explore a different dimension of semantic similarity based on image generative models. There are (at least) two use cases: One is self-evident, which is to provide alternate methods to measure semantic similarity in text, by opening a new modality of analysis - which is visual. The other is to provide methods to evaluate and compare text-conditioned diffusion-based image generation models - although this is largely open for investigation and well beyond the scope of a single conference paper whose goal is to introduce the concept. Importantly, none of the existing text-only methods pertain to either of these use-cases.

However, what can be empirically tested are baseline methods for extracting semantic similarity targeted towards image diffusion models. While we are the first work to formalize this notion of semantic similarity/alignment for such models, we introduced several new comparable baselines in our revision (see overall comment) to address the reviewer's concern. We have also further modified Sec 4.3 in our paper to incorporate this discussion on use-cases.

W2: Isn’t the method hindered by whatever the text encoder does not represent well, or what the diffusion process did not accurately learn? Seeing as these are inherent components, this seems like a significant drawback, which effectively nullifies possible advantages of this method. You touch on this matter in lines 312–317 and section 5. I would appreciate a clarification here.

Conceptual or semantic similarity is intrinsically subjective or model-dependent, as it can be proven that there exist no canonical notion of semantic information, and therefore semantic similarity (shared information) -- see Achille et al., 2024. The fact that the method can reflect what is learnt by both the text encoder and the diffusion process is precisely our advantage. While the semantic alignment of modern diffusion models are indeed "upper-bounded" by representation structures captured by text-encoders, this does not imply equality. For instance, one can easily train a model to map all encoded text to a white background. Even if the original text encodings are highly informative, it would be inappropriate to claim that the resulting model is semantically well-aligned. We also refer to our new experiments in Sec 4.4, which demonstrates a case where diffusion models preserve semantic relations between only certain categories of words.

This also relates to the reviewer's next question:

Is it possible to detect poor representations automatically in the underlying text encoder using this similarity measure?

It is possible, but the result will be conflated with the representations learnt from the diffusion training process (it would be better to independently evaluate the text encoder unit on its own). However, what is enabled by our method is the detection of how well semantic relations captured by the text encoder are transferred to the trained diffusion model, as our experiments in Sec 4.4 demonstrates.

I had a hard time understanding Figure 1, consider redesigning it or breaking it down to two figures

We have redesigned Figure 1 to incorporate the reviewer's suggestion, thank you for the suggestion!