Unlocking Compositional Understanding of Vision-Language Models with Visualization Representation and Analysis

However, I still have several concerns.

1. There are many vision-language models coming out recently, e.g., GPT-4V, Gemini. Before that, there are LLaVA and its variances. Simply researching CLIP may not provide up-to-date insights.
2. The literature review is out-of-date and some parts are missing. More recent works about compositionality should be included. Analysis works should also be included. In summary, I think the current version is not ready for a conference paper.

1. Given the depth of analysis, more nuanced conclusions would strengthen the paper. General statements like "This highlights CLIP’s limitations in handling semantically permuted yet textually similar inputs" and "CLIP tends to focus on individual entities rather than complex semantic relationships" are insights that can already be inferred from accuracy-based analyses alone. Such shallow conclusions could limit the paper's contribution in precisely identifying specific issues.

2. Representation clarity: The first item in the legend of Figure 4 is somewhat confusing. It appears that the red line refers to "Token Attention" and the green line to "Gradient." A simple fix would be to ensure "Token Attention Gradient" appears on the same line for better clarity.

• TL;DR: This paper may be more suitable for a demo track at another conference after further improvements.
• Limited Originality in Methods: The methods used, such as grid-based searches and t-SNE, are well-known and may lack the novelty expected in this context.
• Insufficient Experimental Details: The methodology could be better detailed; for instance, in line 171, the threshold selection for matching scores is mentioned but not elaborated upon.
• Lack of Quantitative Results: The paper relies mainly on case studies without quantitative analysis, which may limit the strength of its claims.
• Clarity in Writing: Certain sections would benefit from clearer structure and transitions, and some subjective terms, like "significant" (e.g., line 257), would be clearer with statistical evidence (e.g., p-values).
• Inconsistent Formatting: Symbols and citations need standardization, such as in line 028 with “composition understanding,” to improve readability.

Overall, this paper lacks technical novelty. All the techniques, approaches, and even the datasets are derived from existing works. The primary contribution is the engineering effort to integrate these elements into a visualization system. While this paper might be valuable for a demo paper, the contribution is not enough for research paper.

Some sections require further elaboration. For instance, the concepts of self-organizing maps and resource-controlled self-organizing maps are not clearly explained. Providing additional background and details would be better. Additionally, the motivation for using a grid representation is unclear. The authors should provide more intuition to this design choice and its implications.

Although the paper presents conclusions based on their analysis, these conclusions have already been reported in the ARO paper and other existing works. From my perspective, this paper merely offers an alternative way to show the same findings found in the ARO paper. None of new evaluation, benchmarks, techniques are proposed for a research contribution.

It would be good to review and reference (if you see appropriate) the literature on compositional models of language. Some of these have led to work on compositional VLM, e.g. see a recent paper https://aclanthology.org/2024.alvr-1.17/ which address a similar problem. Your approach differs from theirs, and this is good. But it would be good to have other solutions referenced in your paper.