Decoupling Semantic Similarity from Spatial Alignment for Neural Networks.

Why do the authors use Pearson correlation to examine the relationship between the Jensen-Shannon Divergence and the representational similarity? E.g. Kendall/Spearman correlation can be more robust. and "The use of some methods at work is not well justified (e.g. Pearson correlation)."

We have added a brief explanation in Section 4.3 regarding our choice of Pearson correlation: "We chose to use the pearson correlation, as it allows observing a direct linear behavior between representational similarity and predictive similarity." IWe believe this is particularly appropriate for cosine similarity and the radial basis function, which are bounded. However, we also provide Spearman correlation results in the appendix, along with a discussion. The corresponding table can also be found in the rebuttal PDF. Overall, we observe that Spearman correlation is mostly lower, with large ResNets being impacted strongly while ResNet-18 is notably consistent and stable in its correlations. ViTs interestingly show a much higher correlation for cosine and radial basis function similarities compared to Pearson correlations.

The authors should focus on the differences between the results obtained for CNNs and ViTs (see the comment in the weaknesses section).

We extended our discussion on CNNs vs ViT's in the appendix, going in depth into the differences of the two translation experiments provided.

The statement in the introduction “we argue that the spatial location of semantic objects does neither influence human perception nor deep learning classifiers.” is a little bit too bold - the paper does not examine human perception, therefore it would be better to leave only the deep learning here.

We cut the statement short to “we argue that the spatial location of semantic objects does not influence deep learning classifiers.”

The authors could provide more examples of their method for different networks to enable their better qualitative assessment which is now limited (e.g. in the appendix).

As displayed in the rebuttal pdf, we will add plenty qualitative visualizations with a discussion in the appendix for many models. Additionally, as we added the Cityscapes experiment, we will also provide qualitative results on this dataset.

Also, a minor thing is that some typos, grammatical and formatting errors can be found in the paper (e.g. the sentence starting in l79, l205: a RSM -> an RSM, l25: SAMor CLIPSeg ,the retrieval performance)

We want to apologize for this oversight, we have re-proofread our manuscript and pushed it through a spell-checker to fix any potential typos and grammatical errors.

They could also better highlight the limited data used in the experiments caused by the mentioned computational constraints.

Instead of highlighting this, we increased our overall sample size, as mentioned in the main response. We hope this to be a sufficient adaptation.

Find the similarity table with spearman's rank correlation below (N=20.000).

We appreciate the time and effort you invested in reviewing our manuscript.

Regarding your comment that "it would benefit from a more detailed discussion on the scalability of the proposed method to larger models and datasets and the approximation error."

We addressed a part of this regarding scaling to larger datasets in the general response already. Regarding the question of scaling to larger models, the scaling of our method is just dependent on representation size, while model size plays no role at all. We add section to discuss the approximation error at the end of the Section 4.4: We cite: "The fastest of the Batch-Optimal approximation methods shows $<8\%$% error while improving run-time $\times 36$ relative to the fastest optimal algorithm for spatial extent $4096$, while no spatial alignment shows $42\%$% deviation ["or approximation error"] from the optimal matching."

We hope that this helps to improve clarity on the approximation error quality of the approximations.

We would like to express our sincere gratitude for taking the time to read our paper and provide valuable feedback and constructive criticism:

1. “This paper lacks the experimental verification of specific downstream tasks, such as detection and segmentation, on semantic RSMs. I need to know which scenario is more suitable for RSMs and semantic RSMs.”

We believe this may be a misunderstanding. Semantic RSMs are not algorithms for detection or segmentation (which we do not claim), but rather they measure the similarity between neural network representations of images. If the concern is about applying our method to dense datasets like those used for object detection or segmentation, our EgoObjects experiment (Fig. 3) demonstrates its application to a dense dataset. Additionally, the results in Table 1 show our method applied to ImageNet.

2. "Lack of quantitative comparative data. It is suggested to add tables or charts to show specific performance comparison data between semantic RSMs and existing methods in different tasks (such as image retrieval, class probability similarity comparison, etc.), including accuracy, time complexity and other indicators."

We have added an additional retrieval experiment on the Cityscapes dataset, comparing semantic and spatio-semantic RSMs for this use case. Regarding the comparison of semantic RSMs with other retrieval methods, such as those using learned embedding spaces, we disagree that this is necessary. The aim of our paper is to find the best way to measure the similarity between representations of two samples. Parametrized methods that alter representations in non-trivial ways would make such comparisons unfair. We emphasize that the main purpose of our paper is to address the flaws in current RSM construction, which often requires spatial alignment. We demonstrate these flaws through retrieval and class probability similarity comparisons and show that our semantic RSMs alleviate these issues.

3. “It is recommended to add a detailed analysis of the experimental results to explain why semantic RSMs perform better on certain tasks, as well as possible reasons and limitations.“

We add a clarifying section at the end of the retrieval section, right before 4.3 discussing the retrieval results. We cite:

“These experiments display clearly that demanding spatial alignment can be a significant shortcoming when semantically similar concepts are misaligned. In Fig. 4, the network learned to represent the objects very similarly, despite a shift in perspective, but due the same objects not aligning anymore, spatio-semantic similarity fails to recognize this. This effect should generalize to other datasets where objects are not heavily centered. For datasets with heavy object-centric behavior, like ImageNet, this should be less pronounced.”

4. Typos We made sure the entire manuscript has the respective errors fixed.

Thank you for the detailed feedback. We can see that a great deal of time and thought went into it. However, we believe there may be a few misunderstandings that we would like to clarify:

Questions

Q1: How well does the method scale to very large datasets[...]

R1: The scaling behavior depends on the spatial extent of representations and the use-case in which the method is applied. Details on this can be found in the common response.

Q2: How well does the method scale [...] to more complex neural networks that handle highly varied or abstract visual content?

R2: We are not quite sure what you mean by "more complex neural networks of varied/abstract visual content". Our experiments already apply the method to various state-of-the-art networks (CLIP, DINOv2, SAM, ResNets, ConvNeXT), which are capable of handling diverse visual content. Additionally, we evaluate our method on standard ImageNet data as well as on EgoObjects, which we believe qualifies as a more complex dataset aimed at robotics.

Q3: How does the method perform under noisy conditions or when semantic parsing is imperfect due to occlusions or poor image quality?

R3: This is an interesting question! Poor conditions or visual ambiguity can degrade representations if the network cannot handle them, potentially harming, for example, the retrieval performance of our method. However, this is not a unique issue of our method but of all methods using learned representations, such as spatio-semantic RSMs. Even standard cosine similarity on global representations would likely experience similar degradation.

Limitations

1. "Apply semantic RSMs in specific use cases like medical imaging, satellite image analysis, and autonomous driving where ignoring spatial arrangements can be particularly detrimental or beneficial, providing a nuanced view of their applicability."

We added an additional experiment for autonomous driving using the Cityscapes dataset, demonstrating improvements of semantic RSMs over spatio-semantic RSMs in this context.

2. To better evaluate the impact of semantic RSMs, a set of diverse metrics should be established: Evaluate semantic RSMs against traditional spatio-semantic RSMs and other state-of-the-art similarity measures to highlight the improvements or shortcomings.

In this paper, we compared the current, most commonly used methods for constructing RSMs, as introduced in the CKA paper by Kornblith et al. (Inner Product and RBF spatio-semantic RSMs), with the addition of cosine similarity. We also introduced two metrics to assess the “goodness” of RSMs, which we believe are sufficient to demonstrate the benefits of our proposed method.

Weaknessses

1. [Runtime of Hungarian matching]: We clearly and transparently state the runtime as a limitation in our paper and provide some algorithms to alleviate this. Reviewer zPy6 even commends us on this. To put our compute time into perspective: In representation similarity analysis many much more slow to compute metrics exist, e.g. the IMDScore which takes >12 hours for a single layer-to-layer comparison between two representations of [10.000x2048] samples using 32 CPU cores we ran in a different project or the RSMNormDifference which can take multiple hours as well. The former was published in ICLR20. This is somewhat of a duplication to the common response, but we felt it necessary to reiterate.
2. [Visual ambiguity can negatively impact]: See Quesions R3
3. [Ignoring spatial context can be bad]: We really appreciated these thought experiments as they go into a lot of detail on how representations work.
   1. [Contextual cues]: Networks progressively combine/merge information to build semantic concepts. E.g. Ears, eyes and whiskers in a neighbourhood, could lead to expression of a cat's head and so on. Similarly we expect a network to learn associations between dishes and cutlery on a table to a meal setting. If the network learns this association, breaking the spatial location with our method is not an issue, as the semantic concept vector will not only carry <plate> but <plate, part_of_meal_table> information somewhere.
   2. [abstract content]: Distance and composition can carry meaning, but if the network learns to represent them similarly, this can be beneficial. For example, if you take a picture of an abstract painting, move back 5 meters, and take another picture, the spatial composition shifts despite the image being identical. Similarly, moving sideways changes the angle. If the network still represents the same semantic meaning despite these shifts, it can still be recognized.
   3. [object interactions] (Very related to context cues) We believe that networks merge semantic information hierarchically within a neighborhood, forming increasingly abstract semantic concepts. Due to this spatial proximity, representations of a cat on a mat are encoded similarly, leading to more frequent retrievals of cats on mats, even when spatial constraints are ignored. Evidence for this is provided by CLIP models, which use global aggregation of their outputs—disregarding spatial relations—to generate final embeddings. This method effectively identifies related images and captions, such as “a cat sitting on a mat,” and we assume this approach is applicable to our setting as well.

We hope that our responses have clarified the open questions and if we answered the questions sufficiently, it would be great if you could consider increasing your rating.

Thank you for engaging with our rebuttal. We are pleased that we have addressed most of your initial concerns. Regarding the remaining issues:

What are the derived insights from the proposal? e.g., Do ViT and CNN exhibit different behaviors? What can we learn from the results? A deeper analysis is needed beyond just showing that the proposal can predict output probabilities (e.g., Table 1).

We believe our paper provides a plethora of insights and is not limited to "just show[ing] that the proposal can predict output probabilities". In our paper, we:

1. Highlight that current RSM construction is flawed and has limitations of requiring spatial alignment, which has not been adequately discussed previously.
2. Introduce an algorithm that addresses this by utilizing set-matching, offering a novel approach.
3. Demonstrate the utility of our method through two applications, revealing significantly improved retrieval performance with five general-purpose feature extractors across the EgoObjects and Cityscapes datasets, and better correlation between the Jensen-Shannon Divergence of output probabilities and inter-sample similarity.

We believe our work underscores a critical gap in current RSM-based methods and sets the foundation for further exploration into RSM construction, effective inter-sample similarity measures, and their applications for retrieval.

There are counterexamples where neglecting location information leads to issues (e.g., visual anagrams), where semantically different images might not be distinguishable by the proposal since it matches features at a set level.

We understand your concerns regarding location information. However, i) our experiments on the EgoObjects dataset, which includes varied contextual scenarios such as the meal table example, indicate that our method outperforms existing RSMs. This suggests that issues related to spatial location might either be rare or of lesser importance. ii) Our method is based on learned representations. Should a visual anagram exist where similar representations are expressed by the network, we believe this is to be a failure of the model and not our approach. We don't believe a retrieval method should try to compensate for this. iii) There are other examples where spatial position is neglected entirely: E.g. the ViT paper [1] shows (in Appendix D.4) that ViT's are able to learn meaningful representations for classification without any positional embedding, making the input a bag-of-words.

Also, how can this proposal be applied in complex scenarios, such as when the data contains multiple objects of the same class?

Our method has already been successfully applied in scenarios involving multiple objects of the same class. Both retrieval datasets, EgoObjects (Figure 3) and Cityscapes (Rebuttal PDF) contain multiple objects of the same class. Additionally, one currently provided qualitative example in the paper and the additional qualitative examples in our rebuttal PDF illustrate our propose methods effectiveness in exactly such complex scenarios, highlighting retrieval on images with multiple instances of glasses or bowls.

Last but not least, the permutation matrix required for set-matching introduces high computational overhead, which could limit the applicability of the proposal to high-resolution data.

We acknowledge the computational demands introduced by the permutation matrix for set-matching. Nevertheless, we have applied our method to high-resolution images (1920x1080) from the EgoObjects and Cityscapes datasets, which demonstrates its feasibility on large-scale data. If your concerns are aimed towards retrieval, we want to emphasize that retrieval is generally a two stage process (see [2] Section 4), with the first being a ranking by global cosine similarity and the second being a re-ranking of the top 100-400 cases which are most similar through e.g. our method. Hence such a two-stage application can enable our method to scale easily to any number of retrieval dataset size. Aside from this, we transparently address this limitation in our paper and have expanded on this topic in the appendix during the rebuttal phase. Moreover we'd kindly refer you to our general response which goes into more detail on the scalability of our method and would be happy to answer any more specific concerns regarding the scalability of our method should there be any remaining.

We hope this addresses your concerns and look forward to any further questions you might have.

[1] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).

[2] Cao, Bingyi, Andre Araujo, and Jack Sim. "Unifying deep local and global features for image search." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16. Springer International Publishing, 2020.