Interfering with Interference: Blind Shuffling and Superposition for Better Multi-Model Compression

1. Improving Presentation and Proof for Layer Shuffling Optimality

We thank the reviewer for their thoughtful feedback on the existing math analysis and we have removed redundant proofs from the appendix.

Regarding the theoretical justification for Layer Shuffling's effectiveness: We recognize the challenge in providing a rigorous proof, as the method's performance inherently depends on learned parameter matrices, which is a product of the learning process. However, our empirical results, particularly in Figure 3(b) of the revision file, demonstrate that intra-model layer variation significantly exceeds inter-model variation for corresponding layers. This observation provides strong empirical support for Layer Shuffling's ability to enhance task vector orthogonality.

We developed this heuristic and used it to establish layer shuffling as a simple and effective algorithm. While we acknowledge there may be better random transformations for task vector decorrelation, and it may be possible to design training procedures to enhance layer shuffling's effectiveness, we look forward to exploring these interesting directions in future work.

2. The Dynamics Between Layer Shuffling and Task Vector Superposition

Thank you for your insightful observation. To have a better look at the interaction between layer shuffling and task vector superposition, we curated below the average accuracy of all three variants of our method across all our benchmarks:

This reveals three key patterns in the interaction between these two mechanisms:

1. Complementary Enhancement: On several benchmarks (8xViT-B/32, 8xFlan-T5-base-LoRA, and 14xViT-L/14), we observe a clear complementary effect. The combined approach (STA + Shuffle) achieves notably higher accuracy than either method alone, demonstrating that the two techniques can work synergistically.
2. Method Dominance: In other cases, we find that one method may be primarily responsible for the performance gains.
3. The Combined Approach Always Works: The combined approach (STA + Shuffle) demonstrates remarkable consistency, achieving either the best or statistically indistinguishable from the best performance across all seven benchmarks.

This interplay between the methods suggests a more nuanced relationship than simple additivity. Combining these two approaches does not necessarily lead to an additive effect. But when they are not additive, they are still complementary in that they seem to be more effective for complementary domains. Sometimes one dominates, sometimes the other. And the union always works.

We have a working theory for this phenomenon. Our theoretical analysis (see equation 5 and section 4) shows that both techniques fundamentally work by enhancing orthogonality between task vectors, thereby reducing interference. The "one-side-dominant" benefits we observed in some cases may be due to diminishing returns in orthogonality enhancement: once sufficient orthogonality is achieved through either method, additional increases may yield minimal improvements. We plan to expand this analysis in a future version of our manuscript to provide a more detailed theoretical framework for understanding these interactions.

1. Including Comparisons with Recent Task Vector Manipulation Methods

We appreciate the suggestion to compare our algorithm with recent task vector manipulation methods, specifically TALL-masks [1]. We have now evaluated our approach against TALL-masks on merging 8, 14, and 20 ViT-L/14 models. The experiment results and discussion are now included in [GR2] and the “Scalability Analysis” section of our revision.

2. Clarifying Memory Overhead and Performance Tradeoff with SMILE as a Comparison

Thank you for raising this important consideration about memory efficiency. As discussed in the "Amortizable Memory Overhead" section of our paper and in [GR1], our method requires only storing one additional task vector instance beyond the pre-trained model. Subsequent models can be merged by saving just their random seeds. This results in a constant memory footprint, which becomes increasingly advantageous when merging more and larger models, as elaborated in [GR2].

To address your concern about whether our memory overhead justifies the performance gains over baselines like SMILE [2], we have prepared a comparison between our method and SMILE across multiple benchmarks. The average accuracy and storage cost (in GB) are presented below:

Performance and Memory Footprint Comparison with SMILE Across Benchmarks

Our STA+Shuffle algorithm outperforms SMILE across all benchmarks, while using less memory on the 8xViT-B/32 and 8xFlan-T5-base LoRA tasks. Although we use more memory than SMILE on the 8xFlan-T5-base benchmark, our performance matches that of the fine-tuned baseline, effectively saturating this benchmark. We plan to compare our method with SMILE on more challenging scenarios (e.g., merging 20 ViT-L/14 models) and will include those results in the next version of our manuscript.

3. Comparing Accuracy and Memory Trade-offs with SMILE on LoRA Models

Thank you for raising this point. We discovered that in our initial experiment, we had not properly tuned the merging coefficient lambda, instead using a fixed value of 1.0. This was suboptimal compared to coefficients used in other benchmarks. After correcting this oversight, we performed a grid search on the validation set to find the optimal lambda value. This adjustment led to significantly improved results, as shown in the following table:

Multi-task performance when merging Flan-T5-base LoRA models on eight GLUE tasks

The updated results demonstrate that our STA+Shuffle algorithm surpasses SMILE [2], achieving higher accuracy while maintaining lower memory requirements. We have incorporated this revised analysis into the manuscript’s “Parameter Efficient Finetuning (PEFT) Model Compression” section.

1. Forward Pass Time Complexity Comparison

Thank you for this important question. As detailed in [GR1], we evaluated our method's runtime overhead when using random seeds to retrieve models with shuffling and superposition applied to all layers. The retrieval process on an Intel Xeon Gold 6448Y CPU averaged 292.70 ms for CLIP-ViT-B/32 and 658.19 ms for CLIP-ViT-L/14 over 10 runs. These overheads remain constant regardless of the number of merged models, as our method introduces no additional memory requirements. GPU optimization could further reduce these runtimes. These measurements have been added to the "Amortizable Memory Overhead" section in our revision, and an asymptotic complexity analysis will be included in the next version of the paper.

2. Missing Baselines from TALL-masks

Thank you for pointing this out. As noted in [GR2], we've added comparisons with TALL-masks [1] algorithms, specifically TALL-masks + TA and Magnitude Masking. We excluded Magnitude Pruning due to its consistently lower performance and missing implementation. While TALL-masks can be enhanced with orthogonal techniques like TIES [2], we excluded this variant for fair comparison. In future version of our paper, we plan to enhance our approach with techniques like Adamerging [3] and include comparisons with TALL-masks+TIES.

3. Extending Experiments to Larger Models and More Tasks

Thank you for your suggestion. We have conducted new experiments merging 8, 14, and 20 ViT-L/14 models as discussed in [GR2]. We will fine-tune and merge even larger models in greater quantities, and include the results in our paper.

4. Discussing Task ID Requirement During Inference for Fair Comparisons

We acknowledge that our method requires the task ID during inference and will emphasize this in our comparison with other methods. However, we believe the comparison remains fair for the following reasons:

1. Other Baselines Also Require Task IDs: For example, when merging image classification models, all baselines require a task ID to select the task-specific classification head for inference. This requirement is made clear in Fisher Merging [7] but not explicitly discussed in many other baselines.
2. Task IDs Enable Strong Performance: Leveraging task IDs allows task-specific retrieval, contributing to our method’s high performance and low memory footprint. This is a good feature and aligns with the goals of model merging and compression.
3. Task Identification is Context-Dependent: In completely task-agnostic environments, it’s possible to infer task ID from the input like WEMoE [5] and SMILE [6]. But in many real-world scenarios where task-specific queries or datasets are naturally tagged with task IDs (e.g., multi-tenant APIs or application-specific deployments), task IDs are easy to come by. The system-specific nature of task identification and its orthogonality to our core contributions have led us to focus on other aspects rather than developing a separate task identification mechanism.

To address your concern comprehensively, we will incorporate a task classifier to our method in the next version of our paper.

5. Providing Detailed Explanations and Categorization of Baseline Methods

Thank you for these thoughtful suggestions. We have enhanced the clarity of our work by:

• Revising the Experimental Setup section to better explain model categorization
• Adding Appendix B with comprehensive details on all baselines, datasets, models, and experimental setups
• Clarifying PSP [8]'s role as an online learning model superposition baseline adapted to demonstrate the effectiveness of our task vector superposition approach in the offline setting

6. Expanding and Enhancing the Related Work Section

We thank the reviewer for this important feedback. We have thoroughly expanded our Related Work section to provide a more comprehensive coverage of model merging and compression literature. The revised manuscript now includes detailed discussions and proper references in these areas. We welcome specific suggestions for any missing critical references to include in the final version.

7. Question: Why is 5.03 the number of Gb attributed to fine-tuned? shouldn’t it be 8x the pre-trained model?

Thank you for this important question. The 5.03x figure for fine-tuned models appears lower than 8x the pre-trained model size because our CLIP models from FusionBench [4] use a frozen text encoder during fine-tuning. Only the vision encoders need to be stored separately for individual fine-tuned models. We have clarified this in the manuscript.

1. Investigating the Impact of Shuffling and Superposition Levels on Performance

We appreciate this valuable suggestion. Following it, we conducted an extensive analysis of shuffling and superposition levels in the newly added Appendix D.2, which has been discussed in [GR3]. Our analysis revealed that performance remains robust at lower skip rates, suggesting opportunities for memory optimization. We discovered a critical cosine similarity threshold that helps explain the effectiveness of different interference reduction methods. We hope these findings provide a clearer picture about optimal interference reduction approaches. Thank you for helping us improve our work.

2. Analyzing Task-Specific Effectiveness of Proposed Strategies

Thank you for this insightful comment about task-specific effectiveness. As shown in [GR3], we observed interesting task-dependent variations - for example, TA+Shift underperformed TA+Shuffle on GTSRB but showed reversed behavior on SUN397. While our current task set is limited for a comprehensive statistical analysis of task characteristics, we have included initial findings about these variations in Appendix D.2. We acknowledge this is an important direction and plan to conduct a more thorough analysis with a larger set of tasks in future work.

3. Clarifying Merged Task Vector Formation and Layer Shuffling Effects

We appreciate you bringing this to our attention. Yes indeed, layer shuffling can mix layer k-1 from model A with layer k+1 from model B, given that they have the same shape. We will enrich these text descriptions and incorporate them to a future version of our paper. Meanwhile, to help readers understand layer shuffling more easily, we have added a visual illustration in Figure 3 (a) of our revision file.

4. Incorporating Relevant PEFT Literature into Related Work

We appreciate the reviewer's suggestion regarding this relevant work. We have added a brief discussion of their paper in the updated Related Work section. Our approach shares similarities in that both methods aim to reduce the memory footprint of delta parameters from fine-tuned models. The fundamental difference is that their method focuses on compressing each individual model's residual separately, whereas we leverage the parameter properties of a set of aligned fine-tuned models to compress them collectively, thereby reducing cross-model redundancy.

We would like to compare our method to theirs. However, we are currently unable to reproduce their results due to the inaccessibility of their codebase, as the provided link is broken. We have contacted the authors to request access to their code and will include a comparison with their methods in a future version of our paper if we are able to obtain it.

5. Minor Formatting Issues

Thank you for your helpful suggestions! We have addressed all the formatting issues, including correcting the LaTeX equation punctuation and ensuring consistent reference formatting throughout the document. Additionally, we have clarified the values in parentheses within the tables, such as the "average (%)" and "Bits (Gb)" columns, by providing detailed explanations in the table captions, ensuring that the meaning is clear without needing to refer back to the text.

[GR2] Benchmarking Against TALL-masks on Larger Models and More Tasks (Reviewer JsQG and mt5T)

Reviewers JsQG and mt5T requested performance comparisons with recent works, particularly the compression baselines from TALL-masks [1]. Reviewer JsQG also emphasized the need for evaluation on a broader range of tasks (e.g., 14 and 20 tasks) and larger models (e.g., ViT-L/14, T5-XXL). In response, we present the performance of our method on the ViT-L/14 benchmark across 8, 14, and 20 tasks, as provided in TALL-masks [1].

The average accuracy and storage cost (in GB) estimates are presented here. These scores differ slightly from those reported in TALL-masks [1] due to corrections we made to dataset split issues in SUN397, EuroSAT, and DTD. Furthermore, the storage cost figures vary from the previously reported values, as the original estimates were inaccurate.

ViT-L/14 Performance Comparison with 8, 14, and 20 Tasks

Key observations:

1. High Performance: Our STA + Shuffle algorithm consistently outperforms alternative methods when merging ViT-L/14 models. Specifically, it achieves performance close to that of individually fine-tuned models, even when merging as many as 14 or 20 ViT-L/14 models.
2. Efficient Storage: While TALL-masks + TA [1] exhibits a gradual increase in storage costs as more models are merged, our STA + Shuffle algorithm maintains a stable storage cost of less than twice the size of a single model, regardless of the number of models merged. This efficiency is achieved because we only need to store a single random seed for each model. Models can then be retrieved with negligible latency—292.70 ms for a single ViT-B/32 and 658.19 ms for a single ViT-L/14—measured over 10 repetitions on an Intel Xeon Gold 6448Y CPU.
3. Comparison to Lightweight Baselines: Although model-merging baselines like Task Arithmetic [2] use 55% the storage of our method, their performance is inferior, resembling pre-trained performance levels as the number of tasks increases.

We have updated our paper to include these comparisons (see Section 5.9 Scalability Analysis), emphasizing our algorithm’s efficiency and effectiveness in merging larger and more numerous models.

[GR3] Dynamics Between Layer Shuffling and Superposition (Reviewer gJhr and jGUu)

Reviewers gJhr and jGUu noted that the interaction between layer shuffling and superposition wasn't thoroughly analyzed, particularly their effectiveness variation across different tasks. Following these suggestions, we conducted a detailed analysis in the newly added Appendix D.2, which revealed several important insights:

1. Higher Skip Rates Increase Interference: We introduced "skip rates" where a skip rate of k means only every k-th target layer within repetitive layer sets is shuffled and superposed. Higher skip rates led to increased cosine similarity and decreased performance.
2. Performance Remains Robust at Lower Skip Rates: Performance maintains stability up to skip rate 2 (Figure 6), suggesting potential for halving memory footprint through selective layer manipulation.
3. Decorrelation Method Impact: We introduced layer shifting as a deterministic alternative to layer shuffling, where layers move one position deeper with wrap-around. Notably, for GTSRB, TA+Shuffle outperformed TA+Shift despite higher cosine similarity, indicating that the method of achieving orthogonality matters beyond the decorrelation level.
4. Task-Dependent Variations: While TA+Shift underperformed TA+Shuffle on GTSRB, this pattern reversed for SUN397. The correlation between cosine similarity and accuracy also varied across tasks (Figure 7), highlighting opportunities for task-specific optimization.
5. Cosine Similarity Threshold Hypothesis: Despite these method and task-specific behaviors, we found that optimal performance consistently occurs at the lowest cosine similarity (Figure 7). This led us to hypothesize a critical cosine similarity threshold below which method selection and task properties become less important. This explains the dominant effectiveness noted by Reviewer jGUu: the STA method achieves near-zero cosine similarity on its own, pushing below this threshold for strong performance. Adding layer shifting (STA+Shift) or shuffling (STA+Shuffle) only marginally reduces the already-low similarity, yielding minimal gains. In contrast, layer shifting/shuffling alone cannot reduce similarity enough to cross this threshold, resulting in inferior performance.

We have highlighted these insights in Appendix D.2 to help develop more effective interference reduction strategies. We plan to explore more scenarios and expand our cosine similarity threshold hypothesis in the next version of our paper.

[1] Wang, K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F., & Frossard, P. (2024). Localizing Task Information for Improved Model Merging and Compression. arXiv preprint arXiv:2405.07813.

[2] Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., & Farhadi, A. (2022). Editing models with task arithmetic. arXiv preprint arXiv:2212.04089.

[3] Tang, A., Shen, L., Luo, Y., Xie, S., Hu, H., Zhang, L., ... & Tao, D. (2024). Smile: Zero-shot sparse mixture of low-rank experts construction from pre-trained foundation models. arXiv preprint arXiv:2408.10174.

[4] Tang, A., Shen, L., Luo, Y., Yin, N., Zhang, L., & Tao, D. (2024). Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. arXiv preprint arXiv:2402.00433.