CLIP in Mirror: Disentangling text from visual images through reflection

W1&Q1: Does … still work … text and its mirrored version? The proposed … might be easily circumvented.

A1: Yes, the disentangling mask still works. Although there is a 10.22 drop (59.71 to 49.49) in performance compared to the accuracy with ordinary typography (See Table Ⅱ in attached PDF), MirrorCLIP still achieves disentanglement and defends against typographic attacks.

As shown in Figure Ⅱ(b), we constructed a dataset that contains the original and the mirrored text. Our results revealed that, after adding original and mirrored text, the cosine similarity between image features before and after flipping also exhibited a great decrease from 0.9855 to 0.8566, as shown in Table Ⅳ. As the core idea of our method is to leverage the lack of feature invariance in CLIP when flipping images, MirrorCLIP still can locate the textual components by comparing image features before and after flipping, as shown in the activation map in Figure Ⅱ(b). Moreover, according to Table Ⅱ, MirrorCLIP still achieves disentanglement with 9.73 improvements (39.76 to 49.49) compared to the baseline, and defends against typographic attacks. We suspect that besides semantic information, the positional information of text may also have some impact on the disentanglement of MirrorCLIP. Yet, the performance experiences a decline compared to the accuracy with ordinary typography, due to significant interference from the original and mirrored text.

We somehow disagree. Because defense/circumvention is not our main focus, and our MirrorCLIP is primarily proposed as a disentanglement method. Compared to ordinary typography, typography with original and mirrored text is a targeted strong attack for our method, and this is not common in the real world. We sincerely appreciate your thorough insights, will add a discussion of this in the limitation section, and explore defense methods against such a strong attack in the future.

W2: The hypothesis … overly generalized … palindromic … "mom" … handwritten text …

A2: MirrorCLIP is capable of managing ordinary palindromes like "did" and "radar" or handwritten text, which change upon mirroring. However, it struggles to achieve disentanglement when dealing with special palindromes like "mom" and "wow". Yet, note that those special palindromes are extremely rare and hence basically have no impact on our hypothesis.

For the case of handwritten text, we have already conducted experiments on 3 real-world typographic datasets where the text is all handwritten and show excellent disentanglement results (Table 4 and Table 5).

For the case of palindromes, we categorized them into two types: ordinary palindromes, where the shape of the words changes before and after flipping ("did" to "bib"), and special palindromes, where the shape of the words remains basically unchanged ("mom" to "mom"). We constructed corresponding datasets: the ordinary palindrome dataset includes 26 words ("dad", "madam", "radar", etc.), while the special palindrome dataset includes 5 words ("wow", "noon", "mom", "nun", "minim"). Both palindromes are illustrated in Figure Ⅱ(c) and Figure Ⅱ(d) in attached PDF. The results are shown in Table Ⅲ in attached PDF. For ordinary palindromes, MirrorCLIP achieves disentanglement with 13.85 improvements compared to the baseline. This is a comparable improvement like other words. However, for special palindromes, MirrorCLIP struggles to achieve disentanglement and only improves the accuracy by 5.29. As special palindromes are quite rare compared to other words, according to the results in Table Ⅲ, their impact is limited.

Thanks for pointing out this. We will include a description of the special palindrome scenario in the limitation section.

W3-1&Q2: Why … not produce … semantics … typography?

A3: This issue is likely due to the limitations of the Stable UnCLIP model we used for feature visualization. It does not possess the capability to directly generate semantically relevant images when dealing with text-only images. The generated images are often meaningless characters, more examples are shown in Figure Ⅱ(a) in attached PDF.

As seen in Figure 7 first row, after disentanglement, images generated with visual features do not carry textual components, and images generated with textual features do not carry visual components. This shows the effective disentanglement of MirrorCLIP.

W3-2: Whether … disentangles the semantics … text-form visual features.

A4: Our method can disentangle the textual semantics. This is verified through text recognition. According to the results in Tables 5 and 6, with disentangled textual features, the accuracy of text recognition improved significantly. This indicates the excellent disentanglement capability of MirrorCLIP for features with textual semantics, not only text-form visual features.

Q3: In Table 6, why … 61.03 when … zeroed out? Does … disentangling … insufficient?

A5: There might be some misunderstanding. Text recognition accuracy when textual features are zeroed out is actually 23.18 (as shown after visual features (zero) in Table 6), not 61.03, 61.03 is actually the text recognition accuracy when visual features are zeroed out. We would like to clarify that the label (zero) denotes textual features or visual features obtained by performing Hadamard product of textual or visual masks with image features, as defined in L247. We will clarify the meaning of the label (zero) more explicitly in the revision to avoid any confusion.

The disentanglement of textual features is sufficient based on the large decrease (from 72.51 to 5.29) in text recognition accuracy in Table 6. The text recognition accuracy of textual features obtained with the textual filter is 72.51 while the text recognition accuracy of visual features is 5.29. The large decrease in text recognition accuracy is due to the efficient removal of textual information.

My primary concern remains whether mirrorCLIP can genuinely achieve semantic-level disentanglement, rather than merely a formal-level disentanglement. The current experimental results do not convincingly address this issue. My specific questions are as follows:

1. Could the authors provide accuracy results using typography semantics as ground truth (similar to Table II but with disentangled textual features)? This would allow for a more objective assessment of mirrorCLIP's performance in semantic disentanglement.

2. Regarding Figure 7, my question pertains to the "textual features results" in the top row, whereas your response focuses on the "image features results" in the bottom row. If the "textual features results" in the bottom row can generate semantically correct images, this would demonstrate the semantic capability of Stable UnCLIP. Consequently, the "textual features results" in the top row should also produce images with specific, corresponding semantics rather than meaningless text.

3. Text recognition appears to address formal-level (texture-like) disentanglement, which does not necessarily demonstrate semantic-level disentanglement. While text recognition can be a preprocessing step for semantic understanding, it does not itself provide semantic-level features.

4. The results in Table II indeed show that mirrorCLIP has some effectiveness against mirrored text attacks, but the reasoning behind this remains unclear. Could the authors provide further analysis and explanation of this experimental result?

5. In Table IV, could you provide the similarity results for the normal typographic attack?

We need to clarify that in our original paper, we have never claimed semantic-level disentanglement as our core contribution. In fact, our core contribution is the observation that CLIP's image embeddings do not exhibit horizontal flip invariance for typography, and we proposed the zero-shot framework MirrorCLIP to achieve the disentanglement of visual and textual embeddings based on this observation. The idea of semantic-level disentanglement was introduced during the rebuttal phase to address your question about whether MirrorCLIP truly disentangles the semantics of text or merely separates text-form visual features, and our experiments confirm that MirrorCLIP is indeed capable of disentangling the semantics of text.

Detailed answers are shown below.

1. There seems to be a clear misunderstanding on semantics of CLIP's representations. If CLIP's text encoder merely focused on the shape of text-like images, it would struggle to recognize images that do not visually resemble their textual labels-for instance, identifying an image of a cat, which obviously does not resemble the word "cat". Given this, our text recognition experiment is far from typical morphological analysis. Apart from the differences in ground truth categories and input embeddings, the process for our text recognition experiments is identical to that of image recognition. Both use CLIP's text encoder to establish the text embeddings of ground truth categories. Instead, it validates how effectively MirrorCLIP preserves and aligns semantic content with text embeddings, demonstrating that our disentanglement process maintains semantic integrity.

   Furthermore, to confirm that our text recognition task is based on semantic disentanglement, we conducted an experiment where images containing the typography of "little" were used as input to the image encoder, and the text ["little", "Iittle", "littIe", "IittIe"] was used as input to the text encoder, where "l" is the lowercase of "L" and "I" is the uppercase of "i", and their shapes are almost identical. We then recognize the content of typography by comparing the cosine similarity between the disentangled textual embeddings obtained from MirrorCLIP and the text embeddings from the text encoder. The final prediction probabilities are shown below.

   	"little"	"Iittle"	"littIe"	"IittIe"
   predicted probability	0.9536	0.0162	0.0298	0.0005

   Results of the same experiment with images containing the typography of "apple" and text ["apple", "appIe", "opple", "oppIe", "abble"] are shown below.

   	"apple"	"appIe"	"opple"	"oppIe"	"abble"
   predicted probability	0.9995	0.0002	0.0001	0.0002	0

   According to the above experimental results, it is evident that our text recognition experiments rely almost entirely on semantics rather than shape.

   In summary, we would like to emphasize that our approach differs significantly from standard text recognition methods, where we use CLIP's text encoder to establish the ground truth, an encoder known for capturing the semantics of input text rather than just morphological patterns [1,2,3,4].

   [1] Clip-forge: Towards zero-shot text-to-shape generation. CVPR. 2022.
   [2] Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022.
   [3] When does CLIP generalize better than unimodal models? When judging human-centric concepts. ACL workshop, 2022.
   [4] Clip also understands text: Prompting clip for phrase understanding[J]. arXiv preprint arXiv:2210.05836, 2022.

2. We highlight our conclusion here - due to the limitation of Stable UnCLIP and ours, we cannot generate expected results based on textual semantics and Stable UnCLIP. Actually, Stable UnCLIP is barely able to work in extreme cases. This does not indicate that our disentanglement of textual semantics is insufficient. Instead, we have shown the effectiveness of our disentanglement of textual semantics via contrasting disentangled textual embeddings from MirrorCLIP and text embeddings from CLIP's text encoder. Moreover, below are some key points that we need to clarify compared to [1]:

   • As described in Section 7.1 of [1], the image generation method used in [1] is entirely different from ours. The method in [1] is text-to-image generation with text prompt, whereas our method is image-to-image generation with no text prompt. And the generation model used in [1] was optimized with their proposed models, while ours is not.

   • It is obvious that the image generation experiments in [1] only can demonstrate that they achieved format-level disentanglement, instead of semantic-level disentanglement.

   • In the third row of Figure 7 in our paper, the images generated using disentangled textual embeddings contain semantically relevant visual components rather than typography. This clearly demonstrates that our method operates at the semantic level rather than the format level.

     [1] Disentangling visual and written concepts in CLIP. CVPR 2022.

   The examples of generated images in [1] demonstrate that they can only control whether they prefer to generate visual or textual components within the same semantic context using a trained projection matrix. This precisely indicates that their work only achieves format-level disentanglement rather than semantic-level disentanglement, as there is no second semantic input provided to the generative model. For example, as shown in Figure 1 of [1], when the input text prompt is "corn", the approach in [1] can only control whether to generate visual components of corn or the typography of the word "corn". This clearly demonstrates that only format-level disentanglement is achieved, not semantic-level disentanglement.

   Moreover, the generative model in [1] is optimized with their projection matrix as described in Section7.1 of [1], while ours is not. Despite this, the image quality produced by our method using disentangled visual embeddings is noticeably better than that produced by the "forget to spell" model used in [1]. Additionally, the model we used was trained for images with normal visual components and not specifically optimized for typography. As a result, our generated results are more susceptible to noise interference when dealing with textual embeddings of typography. It is evident that the textual embedding in the first row is much more affected by noise than in the third row of Figure 7, which explains why the first row struggles to generate semantically relevant images.

   Furthermore, in the third row of Figure 7, the images generated using textual embeddings of typography contain semantically relevant visual components rather than typography. This clearly demonstrates that our method operates at the semantic level rather than the format level.

I hope the above answers address your concerns. We sincerely look forward to your response.

W1: Detailed experimental results of the disentangling framework when dealing with images containing flipped text were not provided in Ablation Experiment.

A1: Thanks for your thorough review of the paper. We show the detailed experimental results with images containing flipped text in Table Ⅰ shown below. Based on the results of Table 6 and Table Ⅰ, we can see that the vision features obtained through MirrorCLIP achieve high accuracy in image classification tasks when handling both normal text or flipped text. The results will be added in the revision.

Table Ⅰ: Results of different features on image recognition with flipped text.

W2: More description of the potential applications of this disentanglement framework in practical tasks should be provided in conclusions.

A2: Thanks for your advice. We have initially explored object detection and text segmentation by combining MirrorCLIP with RegionCLIP and SAM. The results show the potential of MirrorCLIP for different downstream tasks or applications. Relevant examples are shown in Figure Ⅰ in attached PDF. By using MirrorCLIP to get the disentangled visual region features of RegionCLIP, we can reduce the influence of textual factors and get more accurate detection results. By using the textual features obtained from MirrorCLIP to generate prompts for SAM, we can achieve text localization within images and perform preliminary text segmentation. In our revision, we will include a description of the potential applications of MirrorCLIP.

W3: Cross multiplication ($\times$) is used throughout equations in the manuscript, which may cause misunderstanding.

A3: Thank you for pointing out the notation issue. We will correct it and thoroughly check all mathematical notations in the revision.

W1&W2&Q1: It would be better to have a discussion on whether MirrorCLIP can be explored for other downstream tasks or applications.

A1: To explore MirrorCLIP's applications and downstream tasks, we combined it with RegionCLIP and SAM, for detection and text region segmentation. Specific examples can be found in Figure Ⅰ in attached PDF.

For detection, RegionCLIP extends CLIP to learn region-level visual representations, allowing for detailed alignment between image regions and textual concepts. This capability supports region-based reasoning tasks, such as zero-shot and open-vocabulary object detection. However, the vanilla RegionCLIP is susceptible to textual components during object detection tasks. By using the MirrorCLIP framework to disentangle the region features of RegionCLIP, we can similarly reduce the influence of textual factors. In Figure Ⅰ(a), vanilla RegionCLIP mistakenly identified a price tag with text "papaya" as papaya. Moreover, after adding the text "television" on the laptop screen, vanilla RegionCLIP was misled and identified the laptop monitor as a television set. These errors were corrected by replacing the region features with the disentangled visual features obtained through MirrorCLIP. This highlights the potential of MirrorCLIP for applications in object detection.

For text region segmentation, by using the disentangled textual features obtained from MirrorCLIP to generate prompts for SAM, we can achieve text localization within images and perform preliminary text segmentation. Specific examples can be seen in Figure Ⅰ(b). This shows that disentangled features through MirrorCLIP can be used for downstream tasks such as image segmentation. Our future work will continue to explore the applications of MirrorCLIP in various tasks.

Dear Reviewer J5xD,

Thank you for your recognition of our work and constructive suggestions. We hope our additional experiments have addressed your questions. As the deadline for the discussion phase approaches, if you have any other questions or would like to discuss further, please let us know. We sincerely look forward to your response.

Best regards from all authors

W1: It would be beneficial to delve deeper into the differences by comparing this approach to the existing CLIP-based method for textual and visual disentanglement in related works.

A1: Thanks for your constructive advice. Compared to other CLIP-based works, our MirrorCLIP is the only training-free method without additional parameters and data, yet exhibits superior disentanglement performance. Moreover, MirrorCLIP remains the performance unaffected on the original datasets while others may degrade the performance.

Due to space constraints, we briefly introduced CLIP-based methods in L90. We will highlight the differences between MirrorCLIP and others in the revision. Specifically, Lemesle et al. introduced methodological tools from the cognitive science literature to assess the language biases of CLIP, and found that the textual and visual factors of an image do not share semantic representations in CLIP by presenting words that distort image classification across different categories levels. However, they cannot achieve disentangled representations of CLIP. Materzynska et al. disentangled visual and textual features by training different projection matrices and applying them to the CLIP outputs. However, it requires the introduction of additional model parameters and data for training; this also results in a performance decrease on the original datasets.

W2: Providing a preliminary introduction to CLIP would be better. Moreover, adding an image that introduces the concept of textual and visual objects of images will improve the clarity of the paper.

A2: Thanks for your advice. In our final version, we will add the preliminary of CLIP, along with a more straightforward presentation of visual and textual components of images to enhance the clarity of our work.

Dear Reviewer xWuF,

Thank you for acknowledging our work and putting forward precious suggestions. As the deadline for the discussion phase approaches, if you have any other questions or would like to discuss further, please let us know. We sincerely look forward to your response.

Best regards from all authors