Thank you for your insightful suggestions and positive feedback. Your input will help us refine this work, and comments such as 'The research questions are far-reaching' and 'The research methods are comprehensive and insightful' serve as strong encouragement to us.

1. What is the principle of linking OLA with syntactic knowledge? (w1,q1,q2)

Thank you for your insightful suggestions. Based on the two prompts below, we used GPT-4o to generate two sets of sentences. Group A contains 100 pairs of sentences with identical grammatical structures but opposite meanings. Group B contains 100 pairs of sentences with identical meanings but different grammatical structures. We used three CLMs to generate the OLA for these sentence pairs and computed their mean SSIM. As shown in Table 1 below, sentence pairs with identical syntax but different semantics exhibit significantly higher similarity than pairs with identical semantics but different syntax. This further supports the view that OLA captures syntactic knowledge.

1. Prompt for generating Group A (syntactically identical, semantically opposite): Please generate one hundred pairs of English sentences with identical grammatical structures but opposite meanings. The semantic opposition should be achieved by replacing one or more words in a sentence with words of the same part of speech but opposite meanings, resulting in sentences that convey contrary ideas. Each sentence must be exactly 50 words in length.
2. Prompt for generating Group B (semantically identical, syntactically different): Please generate one hundred pairs of English sentences with different grammatical structures but identical meanings. Each pair should convey the same semantic content, describing the same event or scenario, with each sentence strictly containing exactly 50 words.

Table 1: Image Similarity of OLA for Group A (100 sentence pairs with identical syntax/different semantics) and Group B (100 sentence pairs with identical semantics/different syntax). Values represent SSIM scores; higher values indicate greater image similarity (maximum = 1).

2. TOA’s Limitation in Applicability (w2)

While OLA exhibits cross-model consistency, it has not yet achieved a perfect feature representation. It primarily captures syntactic information and encodes semantic information weakly. Thus, it is well-suited for syntax-related tasks but remains ineffective for semantics-heavy tasks (e.g., text generation). We will continue refining it in subsequent work, aiming to uncover a shared semantic feature representation across LMs.

3. Does G-9b's OLA show significantly different performance in the dependency parsing task compared to other LMs? (q3)

Due to space constraints, not all models' syntactic dependency parsing performances were included in Table 3 in the main text. Table 2 below shows the performance of Q-7b, G-9b, and L-8b's OLA on syntactic dependency parsing. It can be seen that the performance of G-9b's OLA on this dependency parsing task is comparable to that of other LMs, showing neither unusually high nor low results.

The reason for TOA's weaker transfer performance on G-9b might lie in the similarity between G-9b's OLA and that of other models being slightly lower compared to other model combinations.

Table 2: Results of syntactic dependency parsing using OLA predicted by CLMs.

Thank you for your constructive suggestions and positive feedback. Your suggestions will help us refine this work, and comments such as 'addresses an interesting and relevant problem' and 'has clearly defined methods that appear reproducible with reasonable effort' serve as encouragement to us.

1. OLA is a reformulation of attention rollout from a multi-order perspective.(w1)

Although OLA is derived via order-level decomposition from Attention Rollout (Sec. 3.1), it differs from Attention Rollout and path tracing, which focus on quantifying input importance. Our focus, instead, is on uncovering common attention patterns across different LMs, an aspect not previously explored by these methods. As demonstrated in Secs. 3.2 and 3.3, identifying cross-model attention commonalities proves difficult using Attention Rollout. In contrast, OLA, achieved through this straightforward decomposition, reveals significant similarities. This highlights the unique contribution of OLA from a novel perspective.

2. Evaluation of OLA Similarity. (w2, w3)

Regarding the question of whether the OLA generated by different LMs are comparable or similar, the answer is similar. We visualized a large number of samples and observed that the OLA generated by different models for the same text do exhibit noticeable visual similarity (as shown in Fig. 3).

For quantitative evaluation, we employed two methods: one leveraging a visual model to assess the similarity of high-level features between different OLA, and the other evaluating low-level similarity using Image Similarity Retrieval (Sec. 3.3).

The rationale for introducing the visual model as a proxy for human evaluation is that it can assess the similarity of high-level features in images.

To more directly demonstrate the cross-LM similarity of OLA, we conducted the following experiments:

1. We replaced the SSIM metric used in the OLA similarity evaluation experiment based on image retrieval (Sec. 3.3.2) with Cosine Similarity and Mean Squared Error (MSE). As shown in Tables 1 and 2 below, using these simpler and more direct metrics, OLA still exhibit significant similarity.
2. Using the data from Sec. 3.3.2, we calculated the similarity metrics (SSIM, Cosine Similarity, MSE) between OLA generated by the source LM and target LM for the same text input. We then computed the mean value across all texts. In simpler terms: For each of the 1000 texts, we generated an OLA pair (one from the source LM, one from the target LM), and we calculated the mean similarity metric across all 1000 OLA pairs. As shown in Table 4 below, OLA generated for the same text by different LMs exhibit high similarity scores.

Table 1: Retrieval-based quantitative evaluation of first-order OLA cross-model similarity. Settings identical to Table 2 in the main text. The similarity metric has been replaced with Cosine Similarity metric.

Table 2: Retrieval-based quantitative evaluation of first-order OLA cross-model similarity. The similarity metric is MSE.

Table 3: Mean similarity of OLA for the same text under different LM combinations. Values in the table represent (SSIM/Cosine Similarity/MSE). Higher SSIM and Cosine Similarity scores indicate greater similarity (maximum = 1). Lower MSE values indicate greater similarity (minimum = 0).

3. The practical utility of TOA is not clearly demonstrated. (w4)

TOA's performance has not yet reached the level of current state-of-the-art (SOTA) methods. As such, its practicality does have certain limitations. Nevertheless, other cross-model knowledge transfer approaches, such as the method in [1], face analogous challenges. We maintain that this work represents a promising beginning and lays groundwork for future research.

[1] Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models.

4. Related work is incomplete. (w5)

Thank you for your feedback. We will supplement these references in the subsequent version and include a discussion on them.

5. What are the limitations of this focus on relatedness over representations? (w6,q3)

OLA fundamentally captures relationships between tokens, rather than token representations themselves. Our analysis (Sec. 3.4) demonstrates that OLA contains syntactic knowledge. We therefore utilize it as a specialized syntactic representation serving as input for TOA. Leveraging OLA's cross-model similarity, TOA enables direct cross-model transfer. The limitation, however, is that compared to features generated by traditional learnable encoders, OLA suffers from information loss. It primarily captures syntactic structural information, containing relatively limited semantic information. Consequently, this limitation restricts TOA's performance to semantics-dependent tasks.

6. Do tokenizers of different LMs require alignment? Should subtokens be aggregated? (q1)

Alignment of different LMs' tokenizers is not required, nor is aggregation necessary.

As evidenced in Fig. 3, despite Qwen and LLaMA having different tokenizers, their OLA for the same text remains similar.

We did not aggregate subtokens. The axes in Fig. 3 display raw subtokens.

7. Why does Sec. 3.3.1 use 4 source LMs instead of 1? (q2)

Employing four source LMs rather than fewer is intended to mitigate overfitting in the visual model.

We treat OLA generated from the same text by different LMs as belonging to the same class (line 183). This implies that the number of training samples per class equals the number of source LMs. If only one source LM were used, each class would contain just a single sample. This would lead to overfitting in the visual model, preventing it from learning comprehensive features of OLA. Using four source LMs helps mitigate this issue.

Table 4 below presents the results of cross-model similarity evaluation for OLA based on the visual model when using only a single source LM (analogous to Table 1 in the main text), which validates the above perspective.

Table 4: Results of quantitative evaluation on cross-model similarity for OLA based on visual model using only a single source LM.

Thank you for your insightful review and constructive feedback. ​​We are honored that our rebuttal resonated with you and received your positive acknowledgment.​​ In subsequent versions, we will:

1. Expand the Related Work​​ to thoroughly discuss higher-order explanations, contextualizing our contribution.
2. ​Clarify our focus​​ on syntactic structural information, emphasizing its scope and application relevance.

​ Your affirmation that the paper is ‘ready for publication’ is tremendously encouraging. We deeply value your support!

Thank you for your valuable feedback and kind recognition of our work. Your comments such as 'The paper offers a novel viewpoint,' 'Experiments are thorough and well designed,' and 'The proposed methods are simple yet effective' have been extremely encouraging to us.

1. TOA’s Limitation in Applicability. (w1,q1)

While OLA exhibits cross-model consistency, it has not yet achieved a perfect feature representation. It primarily captures syntactic information and encodes semantic information weakly. Thus, it is well-suited for syntax-related tasks but remains ineffective for semantics-heavy tasks (e.g., text generation).

We previously considered using OLA for text generation tasks and conducted exploratory experiments. When attempting to reconstruct original sentences using OLA, we observed unsatisfactory results. Consequently, we simplified the task by employing OLA for part-of-speech (POS) tagging of the original sentences, which yielded well-received outcomes. Since POS correlates strongly with syntactic structures, this realization directed our research toward syntax-oriented exploration. Although OLA currently primarily encapsulates syntactic knowledge, we will continue pursuing a unified semantic feature representation across diverse LLMs.

2. Discussion on Higher-Order OLA (q2)

In the main text, we primarily analyze 1st to 3rd-order OLA because they have already demonstrated significant cross-model similarity and possess lower computational complexity. The computational complexity of higher-order OLA is higher: for k-th order OLA, it is necessary to calculate $\binom{N}{k}$ paths worth of information aggregation matrices, where N denotes the number of LM layers. Therefore, we currently focus less on higher-order OLA.

In Table 1 below, we present the quantitative analysis results for fourth-order OLA cross-model similarity (settings mirror main-text Table 2). We discover that fourth-order OLA still reveals significant commonalities. In future work, we will explore higher-order OLA to formalize theoretical generalization boundaries.

Table 1: Retrieval-based quantitative evaluation of fourth-order OLA cross-model similarity. Entries indicate evaluation metrics Hits@1 / Hits@5 (unit: %)

Thank you sincerely for taking the time to thoroughly review our manuscript and for sharing your thoughtful feedback. ​​Your positive feedback is highly valued and encouraging for our research.​​

Thank you for your constructive suggestions and positive feedback. Your insights will help us improve the work, and comments such as 'a novel perspective' and 'strong breadth and depth' are a great encouragement.

1. The Innovativeness of OLA and its Relation to Interpretability Methods (w1, w2)

Although OLA derives from Attention Rollout via order-level decomposition (Sec. 3.1), it differs from Attention Rollout and path tracing, which focus on quantifying input importance. Instead, our focus is on uncovering common attention patterns across different LMs, a novel aspect unaddressed by these methods. As shown in Secs. 3.2 and 3.3, identifying these cross-model commonalities is difficult with Attention Rollout. In contrast, OLA, achieved through this straightforward decomposition, reveals significant similarities, highlighting its unique value from a novel perspective.

Regarding the relationship to interpretability methods: OLA originates from Attention Rollout, an interpretability method, and thus retains the ability to explain model behavior. However, this work is not aimed at enhancing model interpretability per se. Our primary goal is to discover latent commonalities in attention patterns across LMs. Consequently, we did not include interpretability methods like LogitLens or ACDC (which analyze individual model behavior and don’t study cross-model commonalities) in our comparative analysis.

2. Is there a trade-off between training cost and performance for TOA? (w1)

Yes, there is a trade-off between TOA's training cost and performance. As shown in Table 1 below, performance increases with more training steps, peaking at ~149,004 steps (taking ~7 hours). Beyond this, gains slow and performance eventually plateaus.

Table 1: TOA's F1 scores on NER with increasing training steps. (the source and target LM are Qwen2-1.5B)

3. Applicability of OLA to CLMs and Long Sequences (w1,q3)

As shown in Section 3, OLA reveals significant commonalities across both CLMs and MLMs. As demonstrated in Section 4, TOA achieves cross-model knowledge transfer on both CLMs and MLMs. This proves OLA's applicability to both CLMs and MLMs.

For long sequences, we conducted two experiments:

1. We validated OLA's cross-model similarity on long sequences. We sampled 1000 sentences of length 256 from CoNLL-2012 and repeated the experiment corresponding to Table 2 in the main text. Results are shown in Table 2 below. Retrieval accuracy improves with longer sentences (vs. main text Table 2), because increased sentence length enhances structural diversity, making OLA patterns between different sentences more distinct, thereby boosting retrieval success.
2. We validated TOA's transfer performance, training time, and inference speed on long sequences. For the NER task (Section 4.2), we created length-256 sequences by randomly concatenating original sentences. Results are shown in Table 3 below. TOA's F1 score shows comparable performance vs. main text Table 5, indicating its suitability for long sequences. Regarding training complexity and inference latency, Table 4 shows both increase with sequence length. This issue can potentially be addressed in the future through architectural optimization of TOA.

Regarding whether OLA overestimates the importance of distant tokens in long sequences: As observed in Fig. 3(b) and (d), OLA tends to overestimate the importance of distant tokens, particularly the first token, due to the attention sink phenomenon (Section 3.1). However, this does not appear to affect OLA's core commonality findings or TOA's performance, since OLA patterns still exhibit high similarity across different models - including this shared tendency to overestimate distant token importance.

Table 2: Retrieval-based quantitative evaluation of first-order OLA cross-model similarity using long-sequence.

Table 3: Cross-LM transferability of TOA on the NER task using long-sequence.

Table 4: TOA's training time and inference latency across sequence lengths on the NER task.

4. Supplemental Comparative Methods (w2,q1):

Our approach focuses on training-free adapter transfer, operating without accessing the target model during transfer. While the knowledge distillation methods you mentioned, such as hidden-state alignment [1] or DSKD [2], offer valuable insights (and will be discussed in future version), they require training student models, falling outside our training-free scope. Consequently, they were excluded from our baselines.

Regarding LoRA and P-tuning methods:

1. P-tuning: We adapted this as a baseline by directly transferring soft prompts trained on the source LM to target LMs. When encountering dimensional mismatches in word embeddings, we performed interpolation to resize prompts to the target LM's dimensions.
2. LoRA: It cannot be directly applied in cross-LM transfer due to architectural incompatibilities (e.g., differing layer counts and hidden dimensions between models).

P-tuning results (Table 5 below) show that transferred soft prompts achieved near zero-shot performance on Qwen models but failed completely on Gemma and LLaMA. This indicates that:

1. This poor performance, approaching zero-shot levels or failing entirely, confirms P-tuning's unsuitability for cross-model transfer.
2. This discrepancy may stem from tokenizer differences: Qwen's tokenizer lacks a BOS token, making it robust to noisy soft prompts occupying the first token position. In contrast, Gemma and LLaMA require a BOS token as the first token to function correctly.

Table 5: Cross-Model Transferability of P-tuning.

[1] IMPROVING LANGUAGE MODEL DISTILLATION THROUGH HIDDEN STATE MATCHING. [2] Dual-Space Knowledge Distillation for Large Language Models.

5. Overstatement of 'first' (w3)

Thank you for noting this. We are the first to identify the OLA similarity phenomenon. Regarding cross-model transfer, a limited number of studies have investigated this. We will refine our expression.

6. Discussion on High-Order OLA (w3)

In the main text, we primarily analyze 1st to 3rd-order OLA because they have already demonstrated significant cross-model similarity and possess lower computational complexity. The computational complexity of higher-order OLA is higher: for k-th order OLA, it is necessary to calculate $\binom{N}{k}$ paths worth of information aggregation matrices, where N denotes the number of LM layers. Therefore, we currently focus less on higher-order OLA.

Table 6 below presents quantitative results for fourth-order OLA similarity (settings match main text Table 2). We discover that fourth-order OLA still reveals significant commonalities. In future work, we will explore higher-order OLA to formalize theoretical generalization boundaries.

Table 6: Fourth-order OLA cross-model similarity retrieval scores. Entries: Hits@1 / Hits@5 (%)

7. OLA Similarity Analysis between CLM and MLM (q2)

Table 7 below shows first-order OLA similarity between encoder-only (MLM) and decoder-only (CLM) Transformers, mirroring main text Table 2's settings. Since CLM OLA matrices are lower triangular but MLM ones are not, we masked the MLM matrices to lower triangular form for SSIM calculation.

As shown, OLA similarity between CLMs and MLMs is lower than within CLM or MLM pairs, but still present in some degree. This indicates shared contextual information aggregation mechanisms across these architectures.

Table 7: Retrieval-based quantitative evaluation of first-order OLA between MLMs and CLMs.

8. TOA’s Limitation in Applicability (q4):

As stated in Sec 4.2, while OLA exhibits cross-model consistency, it has not yet achieved a perfect feature representation. It primarily captures syntactic information and encodes semantic information weakly. Thus, it is well-suited for syntax-related tasks but remains ineffective for semantics-heavy tasks (e.g., text generation). Developing cross-model unified semantic representations is a key direction for our future work.

9. Selection of the OLA Order (q5)

The OLA similarity does vary to some extent for different model pairs and orders, but the overall trend shows high similarity from orders 1 to 3 across most pairs (line 217), proving that OLA similarity is a widespread phenomenon.

To investigate how OLA order selection would affect TOA's performance, we tested TOA's performance under different orders for the NER Task. Results in Table 8 below show no substantial performance differences across orders 1 to 3. Therefore, given that order-1 OLA has the lowest computational complexity, it is generally sufficient in most scenarios.

Table 8: TOA Transfer Performance Under Different Orders. Source LM is Qwen2-1b5.

We appreciate your insightful suggestions.

1. TOA’s Limitation in Applicability (w1, q1):

As stated in Sec 4.2, while OLA exhibits cross-model consistency, it has not yet achieved a perfect feature representation. It primarily captures syntactic information and encodes semantic information weakly. Thus, it is well-suited for syntax-related tasks but remains ineffective for semantics-heavy tasks (e.g., text generation).

Developing cross-model unified semantic representations is a key direction for our future work. We're integrating the technique from [1] into OLA by representing each word's semantics as its relevance to a set of Anchor Words. This approach holds promise to enhance OLA's semantic capacity while preserving its cross-model consistency.

Moreover, a significant portion of our contribution is reflected in the discovery of OLAS, which reveals universal attention patterns across diverse LMs, a finding acknowledged by other reviewers.

2. Supplemental Comparative Methods and Further Performance Analysis for Zero-shot (w2, q2)

In the subsequent version, we will supplement comparative methods including Cross-Model Control (CMC) [2] and few-shot prompting. Below we present the partial results for the NER task (Sec. 4.2). The few-shot templates were constructed by adding the first five samples from the training set as demonstrations to the zero-shot templates (Sec. K).

Table 1: Comparative experimental results (F1 score). Best performance bolded.

Analysis of the zero-shot and few-shot performance:

1. Using NER as an example, LMs may not fully grasp certain nuances of the task. For instance (Cases 1 & 2), they often incorrectly predict pronouns like 'I' or 'you' as named entities. They also exhibit counterintuitive errors, such as labeling phrases like 'our perspective' as an 'EVENT'.
2. LLMs sometimes generate outputs that seem reasonable but diverge from dataset labels (Cases 3 & 4). This discrepancy resembles a mismatch with human labeling preferences. For example, where the dataset annotates 'Zhou' as 'Person', LMs prefer the full span 'Professor Zhou'. Similarly, LMs may classify 'mobile phone SMS' as 'PRODUCT', while the original labels do not. These represent comprehensible deviations from dataset conventions, rather than fundamental errors.
3. Few-shot prompting improves modestly over zero-shot, but gains remain limited. TOA typically outperforms both (Table 1). Smaller models (Qwen-1.5B, Gemma-2B, LLaMA-3B) benefit more from few-shot examples than larger counterparts (Qwen-7B, Gemma-9B, LLaMA-8B), likely because the limited information from only five examples adds negligible value to large models with extensive pre-trained knowledge.

case study:

For Case 1, we show the full prompt (including the template from Section K). For other cases, we display only the input sentence and model output; their prompts were generated by inserting the input after the template’s 'Sentence:'.

Case1（Q-7b）:

Prompt: Act as a named entity recognition tagging tool. Find all entities and their classes in a sentence according to these rules: 1.Choose the correct named entity class from $PERSON, NORP, FACILITY, ORGANIZATION, Geo-Political Entity, LOCATION, PRODUCT, DATE, TIME, PERCENT, MONEY, QUANTITY, ORDINAL, CARDINAL, EVENT, WORK_OF_ART, LAW, LANGUAGE$ . 2.Do not explain or add extra text. Sentence: Well , I especially want to know , ha , how the two of you found out the news on the day of the accident ? Response as tuples, and each tuple must have exactly two elements: first element is the named entity text (as a string), second element is the named entity class (as a string), e.g. (<entity1>, <class1>), (<entity2>, <class2>), ... Response:

Sentence: Well , I especially want to know , ha , how the two of you found out the news on the day of the accident ?

Lables: (two, CARDINAL)

Outputs: (I, PERSON), (two of you, GROUP), (the news, PRODUCT), (the day, DATE), (the accident, EVENT)

Case2（L-8b）:

Sentence: From our perspective as media , I think it was precisely by going through those breaking - in , trials that we could be so calm in handling such an emergency today , ah , and give everyone the most effective announcements and notifications , and instructions .

Lables: (today, DATE)

Outputs: ('our perspective', 'EVENT') ('as media', 'FACILITY') ('I', 'PERSON') ('it', 'PERSON') ('those breaking - in , trials', 'EVENT') ('today', 'DATE') ('ah', 'PERSON')

Case3（Q-7b）:

Sentence: Well , like Professor Zhou , I also received this news , ha , through a mobile phone SMS .

Lables: (Zhou, PERSON)

Outputs: ('Professor Zhou', PERSON), ('news', EVENT), ('mobile phone SMS', PRODUCT)

Case4（L-8b）:

Sentence: I think , in comparison to China , we should say that urbanization in foreign countries developed earlier and is more widespread .

Lables: (China, Geo-Political Entity)

Outputs: ('China', 'Geo-Political Entity'),('urbanization', 'EVENT'),('foreign countries', 'LOCATION')

Analysis of CMC Results:

1. CMC enhances an LM's performance by training a delta LM that concurrently reasons on the same input text alongside the source LM and integrates their output logits. Furthermore, this delta LM can be applied to enhance other target LMs, demonstrating promising results in instruction-following tasks. However, they utilized a delta LM (TinyLlama) and source LM (LLaMA2-7B) with identical tokenizers, and they only explored scenarios where the delta LM's tokenizer differed from the target LM's (e.g., LLaMA2-70B, Mistral-7B) via token mapping based on metrics like edit distance. We observed that when the source LM (e.g., Qwen-1.5B) and its delta LM (e.g., TinyLlama) employ different tokenizers, the knowledge transfer from the delta LM to target LMs faces challenges, leading to suboptimal performance. This performance gap likely arises from mismatches in tokenizer mapping between the source/target LMs and the delta LM, which can cause a distributional discrepancy in the tokens processed by the delta LM during both training and inference. Since the OLA of two LMs with different tokenizers remain similar, we did not encounter this problem.
2. We found that incorporating CMC may introduce previously absent formatting issues, such as nested parentheses, extra spaces, or superfluous commas. These artifacts somewhat lowered its overall score.

3. TOA require training independent adapters for downstream tasks. (w3)

Yes, but learning separate adapters for different downstream tasks is a common practice in adapter-based methods [3,4]. Moreover, once TOA is trained, it can be directly used by other LMs without retraining, making it efficient.

4. Performance Degradation After Transferring TOA. (w4)

Training-free cross-model transfer is profoundly challenging due to the absence of target model information. This inevitably leads to performance degradation after TOA transfer; nevertheless, its performance remains superior to the target model's zero-shot capability, indicating successful knowledge transfer to some extent. Notably, this limitation also persists in other transfer methods [2,5].

5. Unclear Expression (w5)

Thank you for the feedback. We will clarify this expression in subsequent revisions.

By 'outliers,' we refer to points within a row that are significantly larger than the others (Line 583). In the example in Sec. 3.2, only the maximum value in each row is considered an outlier. For ease of understanding, we explicitly stated to 'set the maximum value to zero'. In Sec. B, multiple outliers may exist per row. Therefore, we used the term 'outliers'.

We will remove the preposition 'of' from the phrase 'The diagonal features of…'.

6. Metrics for Measuring OLA Similarity (q3)

We used a 2D similarity metric (visual model, SSIM) instead of 1D vector metrics because different LMs employ distinct tokenizers. Even when resized to identical dimensions, corresponding pixels do not consistently represent the same key-value pairs (Figure 3). Thus, a 2D structural indicator is more appropriate than pixel-to-pixel metrics.

For comprehensive evaluation, we employed simpler vector similarity metrics in our image similarity retrieval experiment (Section 3.2.2), including the Cosine Similarity and Mean Squared Error (MSE). It can be seen from the results in Tables 2 and 3 below that all tested metrics reveal the cross-model similarity of OLA.

A more detailed discussion can be found in R4-#2.

Table 2: First-order OLA cross-model similarity using Cosine Similarity (Experiment settings match main text Table 2)

Table 3: Results using the MSE metric.

References:

[1] Relative representations enable zero-shot latent space communication.

[2] Cross-model Control: Improving Multiple Large Language Models in One-time Training.

[3] Scalable Language Model with Generalized Continual Learning.

[4] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters.

[5] Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models.

Dear Reviewer aRHQ,

Thank you for your detailed review and valuable suggestions. As the discussion period is ending soon, we warmly invite you to take a final look at our rebuttal.

To briefly summarize our response:

1. ​We discussed the limitations of applying TOA to semantics-dependent tasks and potential improvement methods.​​
2. We added few-shot prompting and Cross-Model Control as comparison methods; TOA still achieved the best performance. Furthermore, we analyzed the reasons for the LLM's poor performance through case studies.​​
3. ​We discussed that TOA requiring training an independent adapter for each task is a common practice. Moreover, due to its cross-model transferability, TOA only needs to be trained once and can be used directly by other models, demonstrating its efficiency.​
4. ​We discussed that as training-free adapter transfer is a very challenging task, some performance degradation after TOA transfer is difficult to avoid. However, TOA's performance being better than zero-shot demonstrates that TOA effectively achieves knowledge transfer.​​
5. ​We thank you for pointing out the unclear expressions in our writing; we will revise them accordingly.​​
6. ​We explained the rationale for using the 2D similarity metric (visual model, SSIM) and supplemented it with simpler vector similarity metrics. Experiments further confirm the existence of cross-model similarity in OLA.​​

Thank you again for your thoughtful feedback!

Best regards,

Paper #11941 Authors