We thank the reviewer for insightful questions that help refine our work further.

1. Comparison with L2SP, DELTA & FTP.

L2SP, DELTA, and FTP aim to retain pre-trained knowledge by aligning finetuned models with pre-trained ones, reducing distance in weight space, feature space and using projected gradient descent, respectively. They provide no guarantees for generalization error nor study it. But we will of course cite/compare with these interesting works in paper.

Our PACE is very different. We leverage the fact that small gradient norms contribute to flat loss landscapes, which enhances generalization (as per our Th. 2 & 3). PACE introduces a novel consistency regularization on output space over different adapter perturbations, implicitly reducing gradient norms (which improves generalization as per Th. 1) and aligning models.

Thus, Our alignment is very different than the above methods.

PACE offers a key advantage over other alignment methods:

• guaranteed gradient reduction, which is crucial for model convergence,
• robustness and generalization (Th. 1-2).

To validate PACE's effectiveness, we compared it with L2SP, DELTA, and FTP on CIFAR-100 (VTAB) and ImageNet (Domain Adaptation) datasets.

The results, presented in the table below, clearly demonstrate PACE's superior performance.

2. Increased computation.

To achieve computation and memory efficiency on par with compared baselines, we have explored two variants to maintain similar computational and memory requirements:

• 1), PACE$\_{fast}$: We store model outputs for each sample from the previous epoch (${\bf o}\_{e-1}=f_{e-1}({\bf x})$) in the CPU memory (or disk if preferred). During training, we feed these into the consistency regularization w.r.t. the current outputs ($||f\_{e}({\bf x})-{\bf o}_{e-1}||\_2^2$) as two versions used different i.i.d. noises (meeting our theoretical need).

• 2), PACE$_{lazy}^{half}$: At every $N^{th}$ iteration during training, we apply consistency regularization and halve the batch size.

The table below compares the maximum GPU memory usage, total training time, and accuracy for each task, demonstrating that PACE$\_{fast}$ and PACE$\_{lazy}^{half}$ significantly improve over the baseline while maintaining similar computational demands as baselines.

3. How $D^{pace}$ align models.

Recall that $D^{pace}$ aims to reduce $||f({\bf \theta}_0+{\bf z}_1\odot \Delta{\bf \theta}) - f({\bf \theta}_0+{\bf z}_2\odot \Delta{\bf \theta})||_2^2$ where ${\bf \theta}_0, \Delta{\bf \theta}$ are pretrained and adapter weights and $f$ is the network, and ensure it is small for all noise ${\bf z}_1, {\bf z}_2\sim\mathcal{N}({\bf 1}, \sigma^2)$.

• Intuitively, imagine we drew $({\bf z}_1={\bf 0}, {\bf z}_2={\bf 1})$ or $({\bf z}_1={\bf 1}, {\bf z}_2={\bf 0})$. Then it becomes clear we indeed reduce the distance between finetuned and pretrained model $D^{fp}=||f({\bf \theta}_0+\Delta{\bf \theta})-f({\bf \theta}_0)||_2^2$.

To understand general case with any $({\bf z}_1,{\bf z}_2)$, follow these steps:

• Theoretically, with Prop. 1 and Theorem 3, the distance between finetuned and pretrained model can be approximated and upper bounded:

$$D^{fp}=[f({\bf \theta}_0+\Delta{\bf \theta}) - f({\bf \theta}_0)]^2\approx[\Delta{\theta}^T{\bf \nabla}-\frac{1}{2}\Delta{\theta}^T{\bf H}\Delta{\bf \theta}]^2 \leq 2d{\color{red}{||\Delta{\bf \theta}\odot{\bf \nabla}||_2^2}}+d^2{\color{red}{||(\Delta{\bf \theta}\Delta{\bf \theta}^T)\odot{\bf H}||_F^2}}$$

where symbols ${\bf \nabla, H}, d$ are the gradient, hessian matrix and dimension of weights.

• Through Theorem 2, $D^{pace}$ can be approximated as:

$$D^{pace}\approx2\sigma^2{\color{red}||\Delta{\bf \theta}\odot{\bf \nabla}||_2^2}+\sigma^4{\color{red}||(\Delta{\bf \theta}\Delta{\bf \theta}^T)\odot{\bf H}||_F^2}$$

where $\sigma^2$ is the noise variance.

• Since $\sigma$ and $d$ are constants during training, minimizing $D^{pace}$ leads to small ${\color{red}||\Delta{\bf \theta}\odot{\bf \nabla}||_2^2}$ and ${\color{red}||(\Delta{\bf \theta}\Delta{\bf \theta}^T)\odot{\bf H}||_F^2}$. This results in a small upper bound for $D^{fp}$, effectively reducing $D^{fp}$ and ensuring alignment of the finetuned model with the pretrained model.
  
  Thus, PACE effectively aligns the finetuned model with the pretrained model even though PACE does not explicitly do that.
  
  Figures 2(b) & 3(b) illustrate that distance from finetuned model to pretrained model become small after applying PACE, verifying that PACE ensures alignment.

Importantly:

• Beside alignment, $D^{pace}$ ensures gradient reduction, as proven in Theorem 2 & shown empirically in Fig. 2(a) and 3(a). As per our motivation, this reduction is crucial for improving generalization, as established in Theorem 1.

We sincerely appreciate your quick reply and are delighted that our rebuttal has addressed most of your questions. We are grateful for the time and effort you have invested in reviewing our work and providing valuable feedback that has strengthened the background and clarity of our work.

1. Regarding your concern about the additional memory requirements of the introduced variants, we would like to clarify that PACE$\_{lazy}^{half}$ ideally introduces no additional memory requirement. Although it includes a hyperparameter $N$, simply setting $N=2$ without any additional hyperparameter search yields much better results than the baseline.
   
   For PACE$\_{fast}$, the introduced memory to store the output from the model's classification head is marginal compared to the baseline GPU memory requirement. Below table compares the memory required by PACE$\_{fast}$ and the GPU memory required to train the baseline:

As shown, memory required by PACE$_{fast}$ is trivial compared to the Baseline GPU memory requirement.

Despite the variants potentially increasing memory or training time in practice, we demonstrate that by simply reducing the batch size and training epochs, PACE$\_{fast}$ still outperforms the baseline while requiring much less GPU memory and training time.

The table shows that with 1/8 batch size and epochs, PACE$\_{fast}$ still outperforms the baseline by 1.7% while only using ~1/3 GPU memory and ~1/4 training time. This demonstrates the robustness and generalization benefits that PACE brings to models, allowing them to excel under constrained training configurations.

However, we want to emphasize that the most valuable part of our work lies in the theoretical contributions and analysis, especially the link between small gradient norm and better generalization, and the implicit effect of gradient reduction and mode alignment in PACE.

These insights provide a deeper understanding of neural networks and their behavior, which we believe will benefit the research community and inspire more effective and robust methods next.

We appreciate all the concerns raised by the reviewer and the opportunity for discussion. Kindly do let us know if there is anything else we can further clarify or if you have further questions.

We thank the reviewer for insightful questions that help refine our work further.

1. Experiment settings.

ViT/16-B and Swin-B were pre-trained on ImageNet-21K using supervised learning. Our analysis extends to self-supervised pre-trained models as well, since our goal is to retain knowledge from large pre-training datasets, regardless of the specific supervision/pre-training method used.

To demonstrate the effectiveness of our approach across different pre-training methods, we conducted experiments on three VTAB datasets (CIFAR-100, Camelyon and Clevr-Count) using ViT/16-B models pre-trained by Masked Autoencoder (MAE) and DINO, both self-supervised methods trained on ImageNet-1K. The results are shown in the table below.

2. Large domain shift.

Indeed, our experiments on the VTAB benchmark already demonstrate PACE's effectiveness in handling large domain shifts. VTAB consists of 19 diverse tasks drawn from various domains, including several that exhibit significant domain shifts from the ImageNet pre-training data, especially the specialized and structured datasets.

Specifically, we have tested PACE on Camelyon (which is also part of WILDS) and Diabetic Retinopathy. These medical imaging tasks represent a substantial domain shift from natural images in ImageNet.

Our results show consistent improvements over the baseline across these diverse task with large domain shifts. This performance demonstrates PACE's ability to effectively adapt to new domains while leveraging pre-trained knowledge.

3. Vary number of training samples on FGVC.

As per request, we varied the data percent (10% to 100%) of training samples on FGVC. Table below shows that PACE gain more improvement when data size is small, which is motivated by our generalization error analysis of PACE (Theorem 1-3).

4. Experiments on large domain shift and large dataset.

In line with current trends, our focus is on finetuning pre-trained models for downstream tasks, where small/medium size datasets are common. In these scenarios, PACE shows significant improvements. For larger datasets, our improvements are smaller, which aligns with Lemma 1 and Theorem 1 - large datasets already allow models to achieve good generalization, diminishing the need for additional generalization improvement techniques.

However, the theoretical findings from PACE, particularly the implicit gradient reduction and alignment, could motivate future work in areas dealing with large domain shifts in big datasets. While we have not extensively tested PACE in such scenarios, these insights could inspire new approaches for handling significant domain shifts, even when data is abundant.

5. Combine with OFT and BOFT.

We re-implemented OFT and BOFT following the code and experimental settings provided by the authors. For OFT, we found that the constrained version (COFT) yields better results than the unconstrained version.

The table below compares results with/without PACE on CIFAR-100 (VTAB) and ImageNet (Domain Adaptation), demonstrating that incorporating PACE leads to improved performance.

6. PACE$_h$.

PACE$_\text{merge}$ refers to PACE$_h$, where perturbation is applied after merging the adapter feature $\Delta h(\cdot)$ with the pretrained layer feature $h_0(\cdot)$, namely, PACE$_h$ perturbs $h(\cdot)=\Delta h(\cdot)+h_0(\cdot)$. This typo has been corrected.

7. Experiments on language tasks.

Following VeRA (Kopiczko et al., ICLR 2024), we conducted GLUE benchmark experiments using RoBERTa-base. We report Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. The table below compares PACE+LoRA with other methods, demonstrating PACE's effectiveness in language tasks.

Esteem Reviewer,

Thank you for your prompt reply and valuable feedback, which has helped strengthen and clarify our paper.

Regarding the lower CIFAR-100 results on self-supervised MAE/DINO in comparison to our paper, there are two key differences:

• 1. Pretraining stage: Here ViT was pretrained using self-supervised learning on ImageNet 1K, whereas in our paper, it was pretrained with supervised learning on ImageNet 21K.
• 2. Fine-tuning stage: For the VTAB benchmark, we fine-tuned the network using only 1,000 images without data augmentation. This means CIFAR-100 on VTAB has only 10 images per class, and the downstream task was supervised learning.

These differences result in less knowledge overlap between pretraining and fine-tuning compared to our paper's setup.

However, applying data augmentation techniques in the finetuning stage (in agreement with pretraining augmentation types) to increase this overlap improved PACE results:

• MAE increased from 44.8 to 47.2
• DINO improved from 60.8 to 62.9

Despite the improvement, these results remain lower than those in our paper due to the smaller number of images per class in CIFAR-100 VTAB and reduced knowledge overlap with MAE/DINO.

Thank you again for your insightful review and for recognizing the potential impact of our work. We sincerely appreciate your time and expertise. We hope this explanation clarifies the discrepancy in results. If you have any further concerns or questions, kindly do let us know and we will do our best to clarify further details.

We thank the reviewer for helpful comments.

1. Copy the original model for consistency regularization not novel.

We believe there is misunderstanding.

Existing "consistency" models (we will cite/compare these interesting works in paper) align features of fine-tuned model with the pretrained model. They provide no guarantees for generalization error. Our PACE guarantees control of generalization by interplay of noise modulator in network branches. Our "consistency" is thus completely different.

We introduce 3 key innovations:

1) Novel consistency mechanism: Unlike previous methods that apply consistency regularization between finetuned and pretrained models, PACE applies consistency between differently perturbed versions of the same finetuned model while learning adapter parameters. This crucial difference addresses gradient limitations of traditional alignment approaches.

2) Gradient reduction: We prove that naive finetuned-pretrained alignment can not guarantee gradient reduction and even cause gradient explosion (Prop. 1, Figs 4, 7, 8). PACE overcomes this issue, ensuring gradient reduction and better generalization.

3) Theoretical guarantees: We provide rigorous analysis (Thm. 2 & 3) showing how PACE implicitly achieves gradient regularization and model alignment even though PACE does not explicitly do so, a key advancement over previous works. Thm. 1 explains how controlling gradient helps generalization.

4) Superior performance: Our experiments show PACE outperforms both FPA (explicitly align finetuned and pretrained model in output space) and DELTA (Li et al, ICLR 2019) (explicitly align finetuned and pretrained model in feature space with supervised attention) across various datasets (table below).

PACE is a significant step forward in consistency regularization, with a theoretically grounded/empirically superior approach.

2. Efficiency concern.

We have explored two variants with similar computational/memory needs as baseline:

1) PACE$\_{fast}$: We store model outputs for each sample from the previous epoch (${\bf o}\_{e-1}=f_{e-1}({\bf x})$) in CPU memory (or disk). During training, we feed these into the consistency regularization w.r.t. the current outputs ($||f\_{e}({\bf x})-{\bf o}_{e-1}||\_2^2$) as two versions used different i.i.d. noises (our theoretical need).

2), PACE$_{lazy}^{half}$: At every $N^{th}$ iteration during training, we apply consistency regularization and halve the batch size.

Table below compares the max GPU memory usage, total training time, and accuracy for each task, showing that PACE$\_{fast}$ and PACE$\_{lazy}^{half}$ significantly improve over the baseline under similar compute demands.

3. PACE+AdaptFormer & GLoRA.

4. PACE with 100 epochs on VTAB.

We use 300 epochs for VTAB tasks (small gain over 100 ep.) but PACE does not needs more time than others to converge. As the optimizer uses cosine learning rate decay, reducing epochs to 100 has minimal impact.

To ensure fair memory/compute budgets, we tested PACE with 1/2 batch size, 50 epochs where PACE still improves baseline accuracy by 2.1% and outperforms the previous SOTA GLoRA (500 train epochs and 30 for parameter search). Thus, PACE is efficient/effective.

5. Tables 2/3: results not that good.

1) In Table 2, PACE shows gains on all shot settings, especially in low-data scenarios crucial for fine-tuning large models on downstream tasks with limited data, and consistently with our motivation to improve the generalization error (Thm. 1-3).

Table below shows average results across datasets and shots. PACE consistently outperforms baseline by 0.9 to 2.9.

2) In Table 3, we observe a 0.7 gain (expected on large data size). When reducing the data size, we see more gains.

Below is averaged accuracy over different data percentages on FGVC tasks (average gain: 1.8 points).

6. Typos.

Duly noted. We will fix.

Esteem Reviewer,

We thank you for your prompt response. We greatly appreciate your time, effort, and the concerns you have shared with us, which significantly improve the completeness, strength, and clarity of our paper.

1. We will be sure to discuss in detail the similarities and differences with the multi-modal fine-tuning approaches maintaining consistency.

Previous works such as VioLET, PromptSRC, and CoPrompt identified the forgetting problem during multi-modal fine-tuning. They propose aligning the fine-tuned model with the pre-trained model, and maintain consistency to prevent catastrophic forgetting.

Specifically:

• VioLET prevents forgetting through collaborative multi-modal gradients
• PromptSRC uses L1 distances between pre-trained and fine-tuned outputs
• CoPrompt employs cosine similarity between pre-trained and fine-tuned outputs.

PACE establishes the link between smaller gradient norm and better generalization, identifying the gradient issues in typical alignment, and proposes a new consistency regularization between fine-tuned models with different noise perturbations, rather than between fine-tuned and pre-trained models.

PACE aligns models and regularizes gradients, thus preventing catastrophic forgetting and improving generalization.

2. Regarding novel ways to avoid higher computational costs.

We value the reviewer's suggestions and will explore this further.

While seemingly simple, these variants are firmly grounded in our theoretical analysis of gradient behavior during fine-tuning. They serve as practical demonstrations of our core insights while avoid higher computational costs.

The simplicity of these variants allows for easy extension and adaptation to various scenarios and model architectures. We want to emphasize that the most valuable contributions of our work are establishing an important link between small gradient norms and better generalization and introducing a new type of consistency regularization.

We hope our ideas will benefit a deeper understanding of fine-tuning neural networks and their behavior, and will motivate further works.

We truly thank the reviewer again for the constructive suggestions that help us enhance the completeness and clarity or our work. If there is anything we can clarify/improve further, kindly do let us know.

We thank the reviewer for insightful questions that help refine our work further.

1. Theoretical analysis done for functions from $R^d\rightarrow R$.

Thank you. In practice, we use the squared L2 distance for multi-dimensional outputs for $D^{fp}$ and $D^{pace}$, which allows our one-dimensional analysis to naturally generalize to multiple dimensions. For example, for a vector-valued function in the naive alignment, $f({\bf \theta}) = [f_1({\bf \theta}), ..., f_m({\bf \theta})]$, where $m$ is the output dimension, we have:

$||f({\bf \theta}_0) - f({\bf \theta}_0 + \Delta {\bf \theta})||\_2^2 = \sum\_{i=1}^m [f_i({\bf \theta}_0) - f_i({\bf \theta}_0 + \Delta {\bf \theta})]^2.$

This equality shows that the squared L2 distance in multiple dimensions is simply the sum of non-negative squared differences in each dimension. Consequently, this additive nature enables our one-dimensional analysis to extend seamlessly to multiple dimensions in practice, aligning with our empirical observations.

2. Increased computation and memory requirements.

Thank you. Motivated by your questions, we have explored two variants to maintain similar computational and memory requirements as the baseline:

• 1), PACE$\_{fast}$: We store model outputs for each sample from the previous epoch (${\bf o}\_{e-1}=f_{e-1}({\bf x})$) in the CPU memory (or disk if preferred). During training, we feed these into the consistency regularization w.r.t. the current outputs ($||f\_{e}({\bf x})-{\bf o}_{e-1}||\_2^2$) as two versions used different i.i.d. noises (meeting our theoretical need).

• 2), PACE$_{lazy}^{half}$: At every $N^{th}$ iteration during training, we apply consistency regularization and halve the batch size.

The table below compares the maximum GPU memory usage, total training time, and accuracy for each task, demonstrating that PACE$\_{fast}$ and PACE$\_{lazy}^{half}$ significantly improve over the baseline while maintaining similar computational demands as baselines. Moreover, techniques such as gradient accumulation can be applied for smaller GPU memory if needed.

3. Experiments on language tasks.

Following VeRA (Kopiczko et al., ICLR 2024), we conducted GLUE benchmark experiments using RoBERTa-base. We report Matthew's correlation for COLA, Pearson correlation for STSB, and accuracy for other tasks. The table below compares PACE+LoRA with other methods, demonstrating PACE's effectiveness in language tasks.

4. Results of 100 epochs on VTAB.

We use 300 epochs for VTAB tasks as we observed slight extra improvements over 100 epochs. However, it does not mean PACE needs more training time to converge. Since the optimizer uses cosine learning rate decay, reducing the number of training epochs to 100 has minimal impact on performance. For FGVC tasks, we maintained 100 epochs as their larger datasets (1k-20k samples vs. VTAB's 1k) ensure convergence with fewer epochs.

The table below shows results for different training epochs on VTAB.

To ensure fair memory and computational budgets, we tested PACE with half batch size and 50 epochs. Under these conditions, PACE still improves baseline accuracy by 2.10% and outperforms the previous SOTA GLoRA, which uses 500 epochs for training and 30 for parameter search. This demonstrates PACE's efficiency and effectiveness across various training configurations.

5. Why smaller weight changes lead to significant divergence in the output space.

Smaller weight changes can lead to significant divergence in output space, particularly when a network is sensitive to weight changes or has entered a sharp local minimum. In such cases, even minor weight adjustments can cause substantial output changes or even incorrect predictions. This behavior contrasts with robust networks or those in flatter minima. Consequently, constraining changes in weight space alone is suboptimal, as it disregards the loss landscape, which is also influenced by gradients. For further details on this phenomenon, kindly refer to the papers listed below.

1. Wu et al, Adversarial Weight Perturbation Helps Robust Generalization, NeurIPS 2020.
2. Foret et al, Sharpness-aware minimization for efficiently improving generalization, ICLR 2021.
3. He et al, Defending and Harnessing the Bit-Flip based Adversarial Weight Attack, CVPR 2020.

6. How is the plot shown for the gradient norm of parameters in multiple layers?

We plotted the sum of gradient norms from all layers.

Thank you for the detiled response and for clarifying my questions. I have read the authors' response as well as other reviews. Overall, the method is promising as an add on to standard finetuning methods, since it has demonstrated applicability to LoRA, OFT and BOFT. Furthermore, it shows theoretical links between gradient norm and generalization which is valuable to the community. I would encourage the authors to add the case of $R^d \rightarrow R^d$ this to the manuscript for completeness. I also thank the authors for providing references in point 5 of the rebuttal. However, I will maintain my original score. Following is a summary of my reasoning:

• The newly proposed efficient variants do seem effective, but the come at a cost of increased CPU memory/disk space. In case where an offload to the disk is required, it increases the read/write time (which can be very slow). Thus, the method will have additional overheads no matter the variant.

• Due to the additional overhead, I would hesitate to call it a PEFT method, although it leads to a small improvement in accuracy.

• While the authors provided experiments on GLUE, the method is still not tested on generative tasks, which is an important application area.

Esteem Reviewer,

Thank you for your prompt and detailed reply.

We are humbled that our rebuttal has addressed most of your concerns and that you recognize our method is promising and has theoretical value, which we believe will motivate more effective PEFT methods in the future.

We want to thank again for your constructive suggestions, which have truly strengthened our work.

1. Regarding the memory cost. We appreciate the concern about efficiency. Our method is designed for fine-tuning models on downstream tasks, which typically involve limited data. We want to clarify that PACE$_{fast}$ only needs to store the output of the model's classification head for the training samples. This means that the additional memory required by PACE$\_{fast}$ is trivial (and can be stored even on the GPU).
   
   Below we compare the memory required by PACE$\_{fast}$ and the GPU memory required to train the baseline.

   Dataset	Mem. PACE$\_{fast}$	Baseline GPU Mem	ratio
   CIFAR-100 VTAB (ViT/16-B)	390KB	8.9GB	0.0042%
   Camelyon VTAB (Swin-B)	7.81KB	15.7GB	0.000047%
   ImageNet DomAda (ViT/16-B)	61MB	8.9GB	0.67%

   As shown, memory required by PACE$_{fast}$ is trivial compared to the Baseline GPU memory requirement and can even stored into the GPUs.
   
   Even in the rare scenario of fine-tuning on the full ImageNet 1K dataset (1.2 million samples), PACE$_{fast}$ requires only 4.8G of additional memory for temp. storage of the output of the model's classification head. This can be easily done on GPU alone. This is significantly smaller than the dataset itself (>100G) and can be easily accommodated in CPU/GPU memory without needing disk storage.
   
   We should have not mentioned disk storage as it is not needed. Even for datasets with 10 million images or more, with typical multi node processing, we cannot see any need for a disk usage. We just mean extreme situations, e.g., a user with a single 3090GPU and 4GB RAM trying to use on full ImageNet. Even then, half-precision floats would fit output into 2.4GB.
   
   While PACE$\_{fast}$ requires trivial small amount of memory, PACE$\_{lazy}^{half}$ does not require additional memory and also enjoy competitive performance.

2. We appreciate your question regarding parameter efficiency. Our understanding of parameter efficiency is that when adapting to downstream tasks, especially during inference, the required additional parameters should be minimal. We can confidently state that PACE adheres to this principle:

   • PACE introduces no additional learnable parameters for model adaptation. The small set of model classification head outputs we keep for PACE$_{fast}$ are not learnable and will be discarded after training.
   • During inference, all PACE variants require no extra parameters beyond the baseline PEFT method.
     
     Therefore, PACE maintains the parameter efficiency of the baseline PEFT method while enhancing performance.
     
     We will add all detailed clarifications into the paper and we are keen to see reviewer's further suggestions that we will accommodate accordingly.

3. Regarding generative tasks, we kindly ask for reviewer's understanding that PACE is built upon the generalization theory for discriminative tasks.
   
   Currently, the generalization theory for generative models remains underdeveloped and requires collective effort from the entire research community.
   
   Nonetheless, we are looking at this problem and we are trying to figure out the potential implementation details. We will try and see if we are able to get something implemented and processed within remaining discussion time but we feel this is perhaps outside of the scope of the paper given the theoretical framework.
   
   
   Meantime, kindly do let us know if you have any other questions that we can answer.

Thank you for your thoughtful feedback.

We are very glad that our response has resolved your concerns.

We appreciate your active involvement and advice, which has provided valuable opportunities to improve our work and make it clearer for the readers.

We will incorporate all your suggestions, along with those from other reviewers, into the final version.

Below are the changes (we will doublecheck if we did not miss any other details/requests) we will implement in the final version of our paper based on your suggestions:

1. Include $\mathbb{R}^d \rightarrow \mathbb{R}^m$ before Prop. 1.
2. Introduce two efficient variants in Sec 3.5. Add Sec. 4.4 to report experiments on these variants and include overhead statistics in the supplementary material.
3. Add experiments on language tasks and 100-epoch results on VTAB in Sec. 4.1.
4. Include citations and explain why smaller weight changes lead to divergence in Sec. 3.3.
5. Clarify how to calculate the gradient norm in Sec. 4.2.
6. Clarify that PACE requires no additional parameters beyond those required by baseline PEFT methods in Sec. 3.5.
7. Clarify our context focuses on discriminative tasks in Sec 3.3 and 3.5.

Additionally, we will implement other changes based on suggestions from other reviewers.

1. Compare PACE with VioLET, PromptSRC, CoPrompt, L2-SP, DELTA, and FTP in Sec 2, with experimental comparisons in Sec. 4.3.
2. Add experimental comparisons with AdaptFormer, GLoRA, OFT, and BOFT in Sec. 4.3.
3. Include MAE and Dino experiments in Sec 4.1
4. Refine explanation for Theorem 3 to clarify why $D^{pace}$ aligns models despite not using explicitly alignment like exisiting methods.
5. Present experiments with varying training sample sizes on FGVC and provide average improvements in Tables 2 and 3.
6. Address reviewer-identified typos and conduct a comprehensive proofreading of the entire paper.

We truly appreciate your time and effort.

If there is anything important we missed/oberlooked, kindly do let us know.

Dear Reviewers and Authors,

Thank you very much for all the informative discussions. We appreciate you for the big efforts.

To reviewers: if you need any further clarifications from authors, please make sure they can be addressed by text-only response without additional experimental results.

To authors: If you have any clarification and summary you would like to share with reviewers, or request their comments on some of your rebuttal and/or discussion points, please add those and/or kindly remind the reviewer(s).

Thank you.