From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication

We express our appreciation to the reviewer for their valuable comments and inquiries. We would like to provide clarifications on the following points:

Weakness 1 (RR definition): We sincerely thank the reviewer for highlighting that the way we formulated RR contains an abuse of notation. The corrected version has been incorporated into the PDF revision.

Weakness 2 (Model-stitching definition): We expanded the description of the stitching procedure, dedicating a specific paragraph after the experimental setting (Section 4.2) and Figure 10 describing the stitching procedure. The revised version is accessible in the PDF revision.

Weakness 3:

• Paper aim: Our paper aims to explore and understand the transformations that relate different latent spaces, introducing a method to achieve latent communication without relying on prior knowledge of the optimal invariance to incorporate. As further clarified in the general response, the purpose is not to enhance relative representation through an aggregating mechanism.
• Additional training: Our methodology does not require additional training in the stitching procedure. We have added a paragraph in Section 4.2 and Figure 10 to clarify this important aspect of our method. For further details, please refer to the response to Question 1 and the General Response.

We would also like to reply to the following questions:

Question 1 (Fine-tuned vs end-to-end): All the downstream experiments are performed in a zero-shot way (including Section 4.3), the learnable parameters are exclusively trained during the initial, end-to-end network training phase and are then employed in their trained frozen state in the stitched model (as part of the decoder model). The only exception to this setting is the one in Section 4.4, “Subspace selection”, where we want to assess the possible gains in integrating additional information into the subspace selection process. For further details, please refer to the general response, Section 4.2, and Figure 10 of the revision.

Question 2 (Other downstream tasks): To assess the applicability of our method to other downstream tasks, we conducted the zero-shot generation task on CIFAR100. Qualitative (Figure 14) and quantitative results (Table 24) demonstrate that our approach, incorporating all invariances (Cos, Euc, $L_1$, $L_\infty$) with the MLP+Sum aggregation (same setting of all the other experiments), consistently equals or exceeds the performance of the best single-invariance model. We would like to remark that this achievement is accomplished without prior knowledge of the most effective invariance and without any training or fine-tuning involved after assembling the pre-trained models into the stitched model. Results can be found in the PDF revision.

We hope to have clarified the reviewer's concerns and remain available to discuss any further concerns or questions.

I thank the authors for their efforts on the rebuttal and kind responds. My questions are all cleared up and I adjusted the score.

We thank the reviewer for providing valuable feedback, and we would like to offer further clarification on the following points:

Weakness 1 (Runs fluctuation): To ensure the robustness of our experiments, we deliberately employed a minimum of three different random seeds. We selected this limit not only to validate the reliability of our experiments but also to assess the applicability and generalizability of our method across a broader range of experiments.

Weakness 2 (Anchor selection): When referring to "randomly but fixed anchors," we mean that the anchors are randomly sampled uniformly from the training set but with a fixed seed value. In this regard, we conducted an ablation study on the number of randomly selected anchors for the stitching task using CIFAR100 across three different anchor seeds. The results reveal that varying the number of anchors leads to different transformations, indicating that a single projection function cannot incorporate the desired invariance. In contrast, our method enables the attainment of the highest score regardless of the number of anchors or the seed value employed for the uniform random sampling of anchors. We have introduced a new section (A.7) in the PDF revision for additional details, including Table 10 and Figure 9.

Weakness 3 (Large scale datasets and models): In our work, we intentionally aligned our choice of models with those used in Moschella et al. (2022), which are widely recognized in the field for their embedding capabilities [1,2,3,4,5,6,7,8,9,10,11]. While these models do not match the large-scale scope of systems like Stable Diffusion or GPT, they are nonetheless significant for their demonstrative value in capturing emerging behaviors. This decision is further substantiated by existing literature, which suggests that even super large-scale models exhibit similar phenomena of emergent similarities [12, 13, 14, 15]. Additionally, in this revision, we measure the performance of the proposed model on the ImageNet1k dataset, which is larger than the other employed datasets. Results in Table 23 show that the proposed methodology utilizing the Aggregation by sum consistently matches or outperforms the others.

We hope that our responses address all the concerns raised by the reviewer. Additionally, we have included supplementary experiments on ImageNet and reconstruction tasks, detailed in the General Response. If further clarification is needed or if there are additional concerns, we are more than willing to engage in further discussion.

Thanks for the authors's response and the concerns are addressed clearly, thus I will keep the score.

We thank the reviewer for the positive feedback, highlighting the extensive experiments and the formal writing style. We corrected the suggested typos, and we welcome the opportunity to address the raised concerns and offer clarifications on specific points:

Weakness 1 (Arbitrary complex):

• (a) Arbitrary distances and geometric intuition: The distances function chosen enforces invariances to distinct fundamental groups of transformations (e.g., orthogonal, conformal, and isometry groups with respect to different metrics: see Table 6 in the appendix for a complete picture). The aim is to combine the different sets of invariances as building blocks to capture more complex unknown transformations acting on real data. Each distance function will have its own specific properties: for instance, the choice of $L_{\infty}$ provides invariances w.r.t. the Chebyshev transformations, resulting in robust to bounded perturbations of the points. In general, our approach could employ arbitrary similarity functions: characterizing from a theoretical perspective which classes of transformations can be captured using the employed invariances as building blocks is a promising future direction.
• (b) Conformal map: The angle-preserving transformation we consider corresponds to the conformal orthogonal group: its elements are compositions of global orthogonal transformations and local rescalings, which depend on each coordinate. The cosine distance is invariant to this group of transformations. The Euclidean isometry group is not properly a subset of this group, as it contains translations, while the former isn’t (similarly, cosine distance is invariant to local scaling, while euclidean distance isn’t). Therefore we attribute the difference in performance to the fact that these two distance functions retain different invariances. To avoid confusion, we incorporated a precise definition of orthogonal conformal transformations in the manuscript.
• (c) Statement: We will re-modulate the statement in the paper and make it less bold. Still, our framework can capture transformations that the single latent spaces can’t capture (see qualitative example reported in Figure 14) where the compositions of all spaces lead to the best qualitative reconstruction), hinting at a major expressiveness in combining single invariances. In general, our framework could easily handle additional invariance other than the one considered in the paper, and as highlighted in the previous answer, it’s a promising direction to characterize theoretically which classes of transformations can be captured by combining different invariances as building blocks.

Weakness 2 (Assumption): We will incorporate in the assumption specifics that avoids degenerated case, as the one presented by the Reviewer.

• (a) The definition of the equivalence class of encoders depends entirely on the projection $\pi_\mathcal{M}$. In the example provided by the reviewer, if the encoder maps everything to the origin, still the projection will be the identity in this case, and the equivalence class will be with respect to the class of transformations $\mathcal{T}_1 \cup \mathcal{T}_2$. This is a degenerate case since the encoder will map everything to a constant; however, our definition still applies. To the best of our understanding, this should address the reviewer's concern. If we missed something, we remain available for further clarification.
• (b) The case where $\mathcal{M}$ corresponds to a single point is degenerate. Our definition still applies, but we agree with the reviewer that treating this specific case is not of great interest. $\pi_\mathcal{M}$ should ideally be a map invariant to all transformations $T \in \mathcal{T}$, but still retaining enough information in the latent spaces in order to solve the task at hand.

Weakness 3 (RR challenges): We have revised the sentence to clarify our statement. We recognize that invariances differ, and consistently depending on the cosine measure, might not be the optimal choice, even though it is sufficient.

Weakness 4 (Employed aggregation): We have added a statement in the "Aggregation functions" section specifying the aggregation function utilized in the experiment, namely the SumAggregation (MLP+Sum). Additionally, details have been included in the table's caption for clarity. The revised version is accessible in the PDF revision.

Weakness 5 (Subspace selection Sec. 4.4): We have incorporated additional details into the section, and we hope the following clarification provides a better understanding of our experiments:

• Experimental details: Section 4.4 focuses on utilizing the Self-attention aggregation described in the method section under "Aggregation functions," which involves a single self-attention layer. The distinction lies in using "SelfAttention" to denote our classic method, where a self-attention layer aggregates different projection functions. On the other hand, when referring to "SelfAttention+... opt" we indicate that there is a fine-tuning procedure at stitching time, which is unique to this particular experiment. In particular, "SelfAttention+MLP opt" indicates the fine-tuning is performed on the whole classifier head; meanwhile, "SelfAttention+QKV opt" indicates only the Query, Key, and Value are fine-tuned. This is an analysis to understand if we can select an optimal subspace (i.e., perform subspace selection) at a reasonable cost at stitching time. Please refer to the general response for further details on the zero-shot stitching procedure.
• Table results: The results do not match because we utilized a more expressive classifier in this experiment. We chose this approach to establish a less favorable scenario and illustrate that our method still performs better than using a single projection function. In our opinion, this is a less favorable scenario since we are fine-tuning more parameters in the "SelfAttention+MLP opt" setting. Following the reviewer's suggestion, we have moved the deep classifier results to Appendix A.6 for a more accurate comparison. In the main paper, we now showcase the results obtained with a simpler classifier, aligning with the data in the full appendix table and with the classifier adopted in all the other experiments. The updates are already present in the PDF revision.

We hope that our responses have addressed the concerns raised by the reviewer and that the modifications made to the paper have resolved any doubts. Additionally, we have included supplementary experiments on ImageNet and reconstruction tasks, detailed in the General Response. We are open to further discussions and available to address any additional concerns or questions.

We express our gratitude to the reviewer for their insightful comments and are pleased to note that their concerns have been adequately addressed.

Regarding the definition of equivalence class:

We concur with the reviewer's observation that varying $\mathcal{M}$ and $\mathcal{T}$ may identify overlapping sets of encoders (for instance, considering $\mathcal{T}_1$ as a subset of $\mathcal{T}_2$, such as $\mathcal{T}_1$ being rotations and $\mathcal{T}_2$ being orthogonal transformations).

However, our intention was to define the equivalence class as identified by the projection $\pi_\mathcal{M}$, i.e., fixing $\mathcal{M}$ and $\mathcal{T}$. In this context, for a specific $\pi_\mathcal{M}$, the set of possible encoders are partitioned in those invariant to $\mathcal{T}$ on $\mathcal{M}$ and those not. These two sets are disjoint by construction.

Different selections of $\mathcal{M}$ and $\mathcal{T}$ will lead to identifying distinct equivalence classes. We will clarify this point and refine the definition accordingly in our manuscript.

We remain available for further clarifications and kindly ask the reviewer to reconsider their evaluation in light of their belief that their concerns have been correctly addressed.

We thank the reviewer for the positive feedback, highlighting the clear focus, presentation, and extremely good execution. We welcome the opportunity to address the raised concerns and offer clarifications on specific points:

Weaknesses 1, 5 & Question 1 (Work scope and space for presenting RR failures): It is crucial to emphasize that our paper's primary goal is to delve into and comprehend the transformations that connect different and independent latent spaces. Building upon the observations that emerged with the analysis, we aim to achieve latent communication without prior knowledge of the optimal invariance to incorporate. The extensive discussion in the paper was undertaken as a first step to study the emerging similarities between latent spaces empirically. Please refer to the General Response for further discussion on our work scope.

Weakness 2 (Model-stitching definition): We expanded the description of the stitching procedure, dedicating a specific paragraph after the experimental setting (Section 4.2) and Figure 13 describing the stitching procedure. The revised version is accessible in the PDF revision.

Weakness 3 & Question 4 (Employed aggregation): We have incorporated a statement in the "Aggregation functions" section specifying the aggregation function used in the experiments. Additionally, details have been included in each caption for clarity. The revised version is accessible in the PDF revision.

Weakness 4 & Question 6 (Subspace selection details Sec. 4.4): We updated the section with additional details:

• Self-attention aggregation: Section 4.4 focuses on using the Self-attention aggregation described in the method section under “Aggregation functions”, which is a single self-attention layer. We want to remark that this is the only case in which we perform stitching-time fine-tuning since we fine-tune the Q, K, and V parameters responsible for blending the subspaces. The stitching is done zero-shot in all the other experiments, as detailed in sec 4.2. All the details were included in the PDF revision.
• Self-attention aggregation: We have included the results using the MLP+Sum (DeepSet) aggregation mechanism in Table 5. We believe this addition enhances the comprehensiveness of our study.
• Self-attention results: We acknowledge your doubt about DeepSet outperforming transformer architectures. However, in our case, it is only a single attention layer rather than a full transformer, and we believe this distinction is a crucial factor contributing to the performance.

Question 2 (DeepSet): We agree on the similarity with DeepSet and added the appropriate citation in the "Aggregation functions" section. The revised version is accessible in the PDF revision.

Question 3 (Anchors selection): In response to the reviewer's insightful question, which enabled us to further enhance the strength of the proposed method, we conducted an ablation study on the number of randomly selected anchors for the stitching task using CIFAR100 across three different anchor seeds. The results reveal that varying the number of anchors leads to different transformations, indicating that a single projection function cannot incorporate the desired invariance. In contrast, our method enables the attainment of the highest score regardless of the number of anchors or the seed value employed for the uniform random sampling of anchors. We have introduced a new section (Section A.7) in the PDF revision for additional details, including Table 10 and Figure 9. Given the importance of this finding, we also mentioned it in the main manuscript.

Question 5 (Aggregation ablation): We sincerely hope the experiments presented in the appendix effectively address the question and provide valuable insights. Tables 13 and 14 showcase the performances using image and text data, conducting an ablation study across all the aggregation methodologies described in Section 3. Additionally, Table 15 presents a comprehensive ablation on various numbers of distances, combining only one, two, or all four projection functions. While acknowledging resource and time constraints prevented a combinatorial exploration of all possible settings, we trust that the results offer valuable insights to the discussion. Your understanding and consideration are greatly appreciated.

We hope that our responses address the primary concerns raised by the reviewer, particularly the weakness regarding the perceived limited scope of the paper. Additionally, we have included supplementary experiments on ImageNet and reconstruction tasks, detailed in the General Response. We are more than willing to engage in further discussions if additional clarification is needed.

Changelist

We sincerely thank all the reviewers for their constructive feedback on our work, suggesting actionable modifications to improve the paper further.

Following their advice, we list here the main changes we adopted:

• Add qualitative (Figure 14) and quantitative (Table 24) performance comparison on zero-shot CIFAR100 reconstruction with AEs (pGUF).
• Added zero-shot classification results on Imagenet1k (Table 23) in Appendix (A.9) (8YvH).
• Added analysis on the number of anchors and on different anchors selections in Section A.7 (ay1h, 8YvH).
• Added experiment on QKV optimization with a linear classifier in Section 4.4 and moved the older in Section A.6 (NofE).
• Clarified the zero-shot stitching procedure in Section 4.2 and added Figure 10 to describe the method visually (ay1h, pGUF).
• Improve clarity and presentation of Section 4.4 (ay1h, NofE).
• Included MLP+Sum aggregation in Table 5 (ay1h).
• Added clarification in Section 3 that, unless specified otherwise, the results employ the Aggregation by sum (MLP+Sum), which is now also reported in the captions of the tables (ay1h, NofE).
• Included conformal transformation definition (NofE).
• Re-written statement on arbitrary complex transformation (NofE).
• Revised the sentence on RR challenges to clarify our statement (NofE).
• Re-formulated RR definition (pGUF).
• Fixed minor typos (NofE).

We provided details and specific replies as answers to each review.