Large Convolutional Model Tuning via Filter Subspace

We sincerely thank Reviewer gvwM for the constructive comments. We have incorporated these suggestions into the revised manuscript and will continue refining our paper.

Q1: Provide computational cost and time delay associated with decomposing convolutional atoms and atom coefficients.

A1: Computational time: The decomposition process using the ISTA algorithm for atoms and atom coefficients takes about 1 second for each layer and 20 seconds for the whole model, with the code implemented on a GPU. This time is negligible compared to the training duration, which is approximately 60 minutes.

Additionally, we only need to perform sparse coding once for each pre-trained model. The decomposed coefficients can then be reused across all fine-tuning tasks, further reducing the computational cost.

Computational cost: We estimate the computation cost in terms of FLOPs for solving the sparse coding problem: $\min \frac{1}{2} ||W - \alpha D||_2^2 + \lambda ||\alpha||_1$, where we aim to obtain atom coefficients $\alpha$ and atoms $D$ from the pre-trained weights $W$. Here $\alpha \in \mathbb{R}^{c'c/k^2 \times m}$, $D \in \mathbb{R}^{m \times k^2}$, $W \in \mathbb{R}^{c' \times c}$, $c'$ and $c$ are the numbers of input and output channels, $k$ is the kernel size, $m$ is the number of filter atoms. Suppose ISTA requires $K$ iterations, the FLOPs required for this algorithm are $K(4c'cm+c'c+6mk^2)$.

In comparison, given the input data $x \in \mathbb{R}^{B \times c'}$ with batch size $B$, the FLOPs required for one linear layer $z=Wx+b$, where $W \in \mathbb{R}^{c' \times c}$ is $6Bc'c+4Bc+c'c+c$ which includes $2Bc'c+2Bc$ (forward pass), $4Bc'c+Bbc$ (backward pass) and $c'c+c$ (update parameters).

Suppose we have $c'=c=512$, $k=4$, $B=64$, $m=9$, with one iteration the computational cost of the decomposition is approximately $9.7$ MFLOPs, while the computational cost of one linear layer is $101$ MFLOPs.

Q2: Provide comparison with recent methods, such as SSF, FacT, Adapter, Compactor, BinaryAdapter, etc.

A2: Additional results are presented in the table below. Compared to SSF, FacT, and Adapter, our method achieves higher average accuracy while keeping the number of tuned parameters minimal.

Thank you for the detailed responds.

My concerns about the additional cost and GPU overhead have been well. addressed. As for the comparison on VTAB and Dreambooth, I have seen the proposed method achieves the best performance.

However, I also noticed the experimental results are not consistent to the original paper. For instance, in the FacT paper, the reported VATB acc is 75.6 with 0.069M #param. But in the rebuttal, the corresponding results are 73.23 with 0.26M #param. So I wonder whether the experimental setup is different here? I still suggest to use the official implementation of previous works to ensure the sota performance of the proposed methods.

In the Dreambooth task, since the OFT paper use the DINO, CLIP-I, CLIP-T and LPIPS as the metrics, what is the performance of the proposed method on these metrics?

We sincerely thank Reviewer GeoC for the constructive feedback. We have incorporated most of these suggestions into the revised manuscript and will continue to refine it to further clarify these points.

Q1: What is the difference when using group convolution and point-wise convolution as filter atoms and coefficients?

A1: The main difference between our approach and group convolution or point-wise convolution lies in its design for parameter-efficient fine-tuning, enabling our method to reconstruct the parameters of the pre-trained model. For instance, in a convolutional layer with $c'$ input channels and $c$ output channels, our method uses filter atoms and atom coefficients to represent the weight update $\Delta F$ as $\alpha \times \Delta D$. In contrast, group convolution and point-wise convolution are unable to represent such weight updates.

In our paper, we further demonstrate that our formulation can be extended to linear weights, a capability that cannot be achieved by group convolution or point-wise convolution.

Q2: Dicuss the memory usage and computation of the proposed method. How to obtain the total parameters of fine-tuning across different networks?

A2: Memory usage. The GPU memory requirements for various generative methods are shown in the table below, with a training batch size of 1. Our method uses less GPU memory than most other approaches, mainly because it fine-tunes fewer parameters. While intermediate features are similar across methods for the same batch size, fewer parameters result in reduced GPU memory usage for storing backward gradients.

Computational cost. The FLOPs for our method is about $4Bc'c/k_c+4Bk_cc'+4Bc+mk_c^2$, where $B$ is the batch size, $c'$ and $c$ are number of input and output channels, $k_c$ is the size of atoms, $m$ is the number of filter atoms. Suppose we have $c'=c=640$, $k_c=4$, $m=9$, $B=1$, our method requires only about $0.4$ MFLOPs.

Number of parameters. Let's consider two types of layers as examples: convolutional layers with dimensions $(c', c, k, k)$, and attention layers with parameters $W_q$, $W_k$, $W_v$, $W_o$, which have dimensions $(d,d)$. The table below lists the PEFT fine-tuning methods along with their corresponding parameter counts. Suppose $c'=c=d=640$, $k=3$, the hyper-parameter for other approach is $r=8$, the hyper-parameters for our method are $k_c=4,m=9, m_1=3$.

In the table, "Ours ($D$ or $D_c$)" refers to our method with tuning filter atoms $D$ and atoms in the linear layer $D_c$, while "Ours (+$\beta$)" indicates that, in addition to tuning filter atoms, we also incorporate overcomplete filter atoms and their coefficients $\beta$.

Compared to other approaches, our method requires the least number of parameters. To determine the parameter counts reported in the paper, we enumerate all the model parameters and sum those that require gradients.

Q4. There are multiple important hyper-parameters. How to set these hyper-parameters?

A4: We have conducted an ablation study on these hyper-parameters in Table 1.

We typically set $m=k^2$ to ensure that the reconstructed convolutional kernels from the filter atoms and atom coefficients are full rank, where $k$ represents the kernel size. For instance, for a convolutional layer of size $(c', c, k, k)$, when $k=3$, we set $m=9$.

In Section 4.2, we observe that increasing the hyperparameters $m_1$ and $k_c$ gradually improves accuracy, but also results in more parameters to tune. In our experiments, we find that $m_1=3$ performs well in most cases. For $k_c$, a value of $4$ works effectively for discriminative tasks, while $k_c=16$ is better suited for generative tasks.

We sincerely thank Reviewer MS4b for the supportive feedback. We have addressed the clarification in the revised manuscript and will continue to refine our paper.

Q1: While the proposed decomposition and fine-tuning method is different, this method adjusts parameters in the big model. Comparatively, LoRA serves as a plug-in, which reduces the chance to hurt the capacity of pre-train models.

A1: Our method is also used as a plug-in. As shown in Figure 2, our method maintains the parameters $F$ in the large models fixed. We only tune $\Delta F=\alpha \times \Delta D$, which is composed of a fixed coefficient $\alpha$ and tunable filter atom $\Delta D$.

Furthermore, we demonstrate in our paper that our method more effectively preserves the capacity of pre-trained models. For instance, in Table 2, compared to the pre-trained model that generates the most diverse images with the highest T2I alignment, our method maintains high diversity and T2I alignment. In contrast, LoRA overfits to the fine-tuned concept, resulting in significantly lower diversity and T2I alignment.

Q2: Parameter fine-tuning often involves one large pre-trained model and many small tasks. Multiple LoRA's can be plug-in to one model even though there could be conflicts, to solve that scenario. How could this method achieve that?

A2: As our method functions as a plug-in, it allows for separate $\Delta D$ for multiple small tasks. Consequently, our approach can exhibit behavior similar to LoRA in handling multiple subtasks.