Differentially Private Bias-Term Fine-tuning of Foundation Models

We thank the reviewer for the comment. Here is a point-to-point response.

1. Contrary to the claims in the paragraph "novelty", it seems to me there is not a huge creative leap in putting together BitFit and DP-SGD as is done here. What is the "substantial algorithmic innovation"? What is the naive way of combining these ideas that your algorithm improves upon?

Response We agree that BiTFiT and DP-SGD are both existing methods. However, our main contributions also include the complexity analysis and engineering implementation. 1) Complexity analysis of DP parameter-efficient fine-tuning (PEFT): DP-BiTFiT in main text, DP-LoRA and so on in appendix C. This is a missing piece in previous DP PEFT literature and significantly helpful in determining the benefit of applying different PEFT methods. E.g. we rigorously show the complexity saving is 50%, and the benefit of BiTFiT on long-context language tasks or high-resolution image tasks, i.e. the computation overhead of DP-BiTFiT in the last column of Table 2 is independent of T. This analysis is also missing in the non-DP BiTFiT. Without this analysis, our understanding of efficient fine-tuning methods will be at most empirical, and certain special properties of DP-BiTFiT (like activation-free optimization and feature-dimension-free overhead) will not be revealed. 2) Engineering effort: at the time of writing this paper, none of existing codebases including GhostClip and Opacus remove the forward hooks, because no analysis has established that only BiTFiT, not LoRA/Adapter/Compactor or full fine-tuning, can be activation-free. This is a simple yet effective trick to save memory in large model training. Our code will be open source to benefit the whole DP community.

In summary, the algorithmic innovation that improves the naive way of combining is visualized in Figure 2: we significantly simplify the computation graph from left and upper to the DP-BiTFiT (bottom right), with complexity guarantee.

2. Why is Ci given as an input to Algorithm 1? I wonder if you mean the clipping function C.

Response We meant Ci is initialized as 1, so that if one ignores all red lines in Algorithms 1, one can recover the non-DP SGD (which still uses Ci in line 8 because we want to write the DP/non-DP SGD in a unified way).

3. Is this any better from a clipping perspective? Do you have to materialize the per-example bias gradients in order to compute their norms? You could combine this with ghost clipping, right?

Response No. We discussed this above Algorithm 1, in the "Novelty" paragraph that ghost clipping only works for the weight, not the bias. We have to materialize the per-sample bias gradients but this is extremely cheap. There are about 0.1% parameters in biases and materializing per-sample bias gradients only incurs $O(Bp)$ time/space complexity, in contrast to $O(BTpd)\approx O(BTp^2)$ of per-sample weight gradients. This is empirically confirmed in Figure 4, including on GPT2-large.

We hope the reviewer can reconsider the score given that the complexity analysis is an important component of this work. Even though the methods are not new, a theoretical analysis could be insightful and novel, like those convergence analysis to the SGD algorithm.

Thank you for extending the discussion. We would like to keep the current complexity analysis without $O$ notation because 1) The complexity is computed correctly up to leading terms, e.g. if the actual complexity is 3BTp+5Bp, the leading term is cubic and leaves 3BTp. 2) The big $O$ notation is an asymptotic symbol hence not precise. In full fine-tuning, standard non-DP optimization takes 6BTpd time complexity and GhostClip takes 10BTpd, the significant slowdown is only depicted without writing both as $O(BTpd)$. 3) We agree that the time of B operations is not B times, due to parallelism of GPU; however, it still holds that the theoretical complexity increases by B times.

We have rewritten Algorithm 1 to include your suggestion.

Regarding ghost clipping, it is a technique that computes the per-sample weight gradient norm without materializing the per-sample weight gradient. It specifically leverages activation and output gradient (see Equation 3 of https://arxiv.org/abs/2110.05679) via an algebraic trick, but the bias gradients do not need the activation (see our Equation 4; this activation-free property is key to BiTFiT). Therefore, 1) GhostClip only applies on the weights, not the bias; 2) GhostClip has time/space complexity $O(BT^2)$ making it inefficient on large $T$ tasks such as long context and high-resolution image (see Section 3.1 of https://arxiv.org/pdf/2210.00038.pdf), but per-sample bias gradients are computed with complexity independent of $T$. These two distinct differences make the computation of weight/bias gradients "orthogonal".

We are happy to discuss further if needed.

We would like to clarify that it is norm to measure time complexity in float-point operation (FLOP), so that either a multiplication or an addition is one operation, despite two operations may cost different wall-clock time. Within this measure, our analysis can give precise calculation of complexity, without using big O. One particular benefit of using our notation (instead of using big O) is that we can infer the wall-clock time: e.g. standard SGD takes 6BTpd time complexity, GhostClip takes 10BTpd, standard BiTFiT takes 4BTpd, and our DP-BiTFiT takes 4BTpd+Bp. This translates to very accurate estimate of running time: GhostClip takes 1.7X more time than SGD, BiTFiT (DP and standard) takes 66% time of SGD.

We can also write DP-BiTFiT complexity in different operations, e.g. (only for illustration purpose, not the actual breakdown) $1.8BTpd\*O_{multiplication}+2.2BTpd\*O_{addition}+Bp\*O_{other}$. Notice that our main conclusions will hold in either notation: for instance, 1) DP-BiTFiT is almost as fast as BiTFiT, 33% faster than full fine-tuning; 2) the overhead ($Bp\*O_{other}$) is independent of $T$, hence DP-BiTFiT is uniquely suitable for high dimension tasks.

I made a typo in my previous response. Let me correct it by stating "the bias gradients do not need the activation", not the output gradient. I believe there is a misalignment of output gradient here. It is not the gradient of loss w.r.t. the last layer's output. In fact, it is the gradient of loss w.r.t. each layer's output. It would be clear to say: 1) the k-th layer's weight gradients need k-th layer's activation and k-th layer's output gradient. 2) the k-th layer's weight gradients need only k-th layer's output gradient. Therefore, GhostClip does not apply to weight gradients because it explicitly works with the outer product of activation.

We thank the reviewer for the comment and the positive feedback. Here is a point-to-point response.

1. The paper can provide more intuitions about why fine-tuning the bias term can be effective enough with pre-trained models. As a comparison, LoRA is supported by a strong intuition that fine-tuning updates can be considered low-ranks.......is there intuition to support the proposed method?

Response We agree more understanding of why BiTFiT works is desirable. This has been difficult even for the original paper in the non-DP regime. Looking backwards from the effectiveness of BiTFiT, we may have a intuition that the fine-tuning updates are generally in low-dimension, as is also the case for LoRA (despite the rank, the number of trainable parameters is small). We may test this hypothesis by randomly selecting a small number of parameters to update at each layer. However, even if any small number of parameters would work, BiTFiT has its unique advantage of being activation-free.

2. While remark 4.2 mentioned that the method may be less effective for models with convolutional layers without bias terms, it is not clear whether all modules' bias terms (attention, fully connected, convolutional, etc.) have the same effect or some of them are more important than others. Besides, as some recent LLM architecture may not include biases in some layers (PaLM [0]) to increase training stability, it is unclear how general the proposed method can be in practice.......Are the bias terms in different modules (attention, fully connected, convolutional, etc.) have the same effects?

Response We haven't looked into a finer-grained analysis of bias terms. This could be an interesting future work had more people become interested in our method. We agree that not all models are suitable for DP-BiTFiT, but a long list of models in our Table 1 should be. Additionally, we study DP-BiTFiT as a standalone method for fair comparison to other standalone methods. In practice, one can always combine two parameter-efficient methods, such as DP-BiTFiT and last-layer training in Section 4.3. Specifically, turning on DP-BiTFiT only needs one line of code: [param.requires_grad_(False) for name, param in model.named_parameters() if 'bias' in name].

3. Is there any potential limitations of the proposed algorithm in terms of the fine-tuning tasks? Can we expect similar improvement if we fine-tune for more complicated tasks, for example fine-tune LLaMA for GSM8k or for MMLU tasks?

Response I am experimenting with LLaMA and do observe similar behavior.

4. Is there any requirement or conclusion about how well-pre-trained a model should be or how complex a model should be so that the proposed method can be effective? An inappropriate extreme example might be that one should not expect tuning the bias in a linear model to fit arbitrarily shifted distribution.

Response Our experiments in Table 3,4,5 show that DP-BiTFiT tends to be more comparable to full fine-tuning when model sizes are larger. This also implies that a better pre-trained model can benefit from DP-BiTFiT more: tuning only bias terms from scratch is not going to work.

We thank the reviewer for the comment. We would like to clarify a misunderstanding in the "Weakness". We do not require "the additional design for two phases." This was an alternative method presented in older version of this paper, hence it should not be treated as a main part of this work on which the review is based.

1. Regarding the novelty, we claim at least 2 types of new contributions: 1) Complexity analysis of DP parameter-efficient fine-tuning (PEFT): DP-BiTFiT in main text, DP-LoRA and so on in appendix C. This is a missing piece in previous DP PEFT literature and significantly helpful in determining the benefit of applying different PEFT methods. E.g. we rigorously show the complexity saving is 50%, and the benefit of BiTFiT on long-context language tasks or high-resolution image tasks. This analysis is also missing in the non-DP BiTFiT. 2) Engineering effort: at the time of writing this paper, none of existing codebases including GhostClip and Opacus remove the forward hooks, because no analysis has encouraged to do so. Note our code will be open source to benefit the whole DP community.

2. Regarding Table 4, we believe the situation is natural: the perplexity is only one of many performance measures other than BLEU. In fact, it is not unusual sometimes the cross-entropy loss (i.e. perplexity) is at odds with the performance measure, e.g. in https://arxiv.org/pdf/2106.07830.pdf Figure 8. We are not sure why DP-BiTFiT seems to outperform DP-full on GPT2-large, though we do consistently observe that the gap is smaller as we scale up the model sizes. We have tuned the hyperparameters of DP-full carefully so our hypothesis is the dimensional issue.

3. We haven't compared to DP-LoRA in all experiments. We are happy to do so in the camera-ready revision if the reviewer can consider raising the score.

Dear reviewer,

We have addressed your comments point-by-point, trying our best to clarify any confusion. We have changed all \cite to \citep in this revision and corrected the table captions.

Specifically, we do not need two-phase training in this paper (it's merely an alternative in the appendix, hence should not be counted towards Weakness). We sincerely hope you would reconsider your score if the score is based on the mis-understanding.

We would love to elaborate the significance of our complexity analysis and in the engineering efforts: removing the forward hooks is never implemented in previous codebases because only DP-BiTFiT is activation-free, not DP-LoRA or any other methods. Being activation-free is an important characteristics that make DP-BiTFiT uniquely suitable for high dimension (e.g. long context or high resolution image) tasks, which was not discovered in existing DP parameter-efficient fine-tuning literature which lacks the analysis on the computation graph and complexity.

Your evaluation is important to us and we look forward to further discussion if needed.

We thank the reviewer for the comment and appreciating our work. Although in Figure 1 we didn't precisely split the memory between storing the model, materializing activations, etc., this information can be inferred (approximately in the middle sub-plot): given that DP compactor/LoRA/Adapter have similar (though larger) number of trainable parameters to DP-BiTFiT, we claim most of the memory difference is indeed in the activations.

We would like to clarify Figure 4. The x-label and legend are both correct. The DP-BiTFiT is serving as a reference line diagonally. Any method such as "non-DP full" can be comparable to DP-BiTFiT by looking at the slope (less steeper means less scalable). Each dot represent the actual throughput or memory (from left to right is large to small models). The throughput means number of examples processed per second by maximizing the batch size, as a measure of training speed. We will clarify it in the camera-ready revision.

We are grateful that the reviewer spot this typo in Table 5: the vit-large result is Ours, obtained through full fine-tuning. The horizontal line should be one row higher.

We thank the reviewer for the comment. Here is a point-to-point response.

1. Removing the forward hook is quite natural as one does not have to compute the gradient on weights. This is a natural choice when combining BiTFiT and differential privacy.

Response: We agree it is natural to remove the forward hooks with the benefit of hindsight. But being natural does not conflict the significance of our choice. At the time of writing this paper, none of existing codebases including GhostClip and Opacus remove the forward hooks, because no analysis has encouraged to do so. We are the first to point this rigorously with complexity analysis and computation graph in Figure 2. We also point out that even the original non-DP BiTFiT paper does not investigate these aspects, e.g. the complexity saving is 50%, and the benefit of BiTFiT on long-context language tasks or high-resolution image tasks. Besides, we complement the missing piece of complexity analysis in the existing literature of DP parameter-efficient fine-tuning (PEFT), e.g. DP-LoRA, making the comparison among methods more solid in theory. We believe this is good practice that can benefit the community and provide this in Appendix C.

2. Moreover, the empirical evaluation shows performance drop of DP-BiTFiT compared with other parameter-efficient fine-tuning techniques.

Response: We agree that there is sometimes a minor drop of DP-BiTFiT, but overall DP-BiTFiT is comparable when applied alone. For fair comparison, we study DP-BiTFiT as a standalone method to compare with other PEFT. A promising application is actually to apply DP-BiTFiT together with LoRA or last-layer tuning, as we have shown in Section 4.3. Also, a small performance drop is acceptable in the cases when we emphasize the memory efficiency (since DP-BiTFiT is activation-free) and implementability (only need one line of code [param.requires\_grad\_(False) for name, param in model.named\_parameters() if 'bias' in name]}).

3. In the Contribution 4, "DP-BiTFiT is a unique algorithm in that the computation overhead is independent of the feature dimension T" where T is the sequence length. The author should be more specific for this claim.

Response: We have explained the computation overhead of DP-BiTFiT in the last column of Table 2, where both the time and the space complexity is O(Bp), independent of T. This is also empirically verified in Section 3.3: we quote "DP-BiTFiT is amazingly scalable since its computational overhead is negligible and independent of T (though the total complexity, mainly due to forward and output gradient, is still linear in T)."

We hope the reviewer can raise the score if satisfied.