Group and Shuffle: Efficient Structured Orthogonal Parametrization

Training times and double GSOFT. Thank you for raising these questions. We have fixed these issues in the top-level comment.

CLIP scores. Thank you for pointing out this observation. Typically, the more parameters are trained the more easily the model overfits. This results in higher image similarity and lower text similarity. In addition, if a model with a large number of training parameters is trained for a long time, the image similarity can often start to decrease as the model starts to collapse and artefacts start to appear.

Models with fewer trainable parameters overfit less, but require longer training to capture the concept carefully, and usually have an upper limit on the maximum image similarity: at some point the increase in image similarity becomes small, while text similarity starts to decrease dramatically. Therefore, the very common result of a usual fine-tuning is either an overfitting with poor context preservation or an undertraining with poor concept fidelity.

The orthogonal fine-tuning shows a different behaviour. The model with less trainable parameters can be trained longer (OFT, BOFT) and capture the concept more carefully without artefacts. At the same time, it overfits less and shows higher text similarity.

Indeed, this needs to be discussed and we will add this to the paper, thank you.

GPT series methods. While we certainly agree with your point, RoBERTa experiment took us multiple weeks of compute time. Conducting experiments with multi-billion models would require the amount of computational resources that we unfortunately do not currently have. Nevertheless, we believe that our experiments are of interest, as they show the comparison with the closest competitor BOFT on a different domain apart from diffusion experiments.

W1. Thank you for your comments. We added compute times and comparison to the standard GSOFT for subject-driven generation (see the top-level comment). In the NLP tasks, we are also faster or on par with the BOFT approach. Compared to LoRa we are several times slower, but observe a slight quality boost on the scale we consider. The BOFT paper shows more quality gain for larger models like Llama. Unfortunately, such experiments require much more computational resources.

W2 and W3. Thank you for pointing out these papers. We will be sure to make a more detailed survey in the revised version. To be more specific, [1]-[3] work in the setting of dense matrices and try to parameterize essentially the whole manifold of orthogonal matrices. For $n \times n$ matrices this requires computing inverses or exponential maps of $n \times n$ matrices at every optimization step. This takes $O(n^3)$ time, which can be computationally challenging for larger architectures. Moreover, such a parametrization utilizes $\mathcal{O}(n^2)$ trainable parameters, which makes it inapplicable for PEFT (see the OFT paper [Qiu et. al, NeurIPS '23] for more details). Our method is different in a sense that it provides a trade-off between expressivity (describing only a subset of the orthogonal manifold) and efficiency. For example, for a dense $n \times n$ matrix, when setting a number of blocks equal to $\sqrt{n}$, our method uses $O(n \sqrt{n})$ parameters and does $O(n^2 \sqrt{n})$ computations at optimization steps.

Work [4] applies orthogonal parametrization to a reshaped convolution tensor. Although it is indeed a widely used heuristics, it does not formally lead to an orthogonal convolution mapping and certified robustness. The work [5] was outperformed by the SOC method that we utilize (there is a comparison in the SOC paper). Finally, in [6], the authors utilize periodicity of convolution and their padding is not equivalent to a widely-used zero-padded convolution. What is more, the survey [7] suggests that [6] and other 1-Lipschitz architectures yield worse robustness metrics than SOC, which is why it serves as a baseline in our paper.

[7] B. Prach, et al. "1-Lipschitz Layers Compared: Memory Speed and Certifiable Robustness." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Small experiment scale. We do understand your concerns regarding the scale of experiments.

For diffusion experiments, the few-shot setting is one of the most challenging tasks for fine-tuning. On the one hand, the model should carefully capture the target concept, but on the other hand, it should preserve it's ability to follow the target prompt. This setting is one of the main motivations for orthogonal fine-tuning (OFT), as it additionally helps to avoid overfitting and was used in the pioneering work with the OFT paradigm.

Despite the few shot setting of a task, we report results accumulated over 30 different concepts and 25 contextual text prompts. In total, there are 750 unique concept-prompt pairs and a total of 7500 images for each baseline for robust evaluation. And our evaluation includes different hyperparameters for each baseline method.

Notice that BOFT only presents a visual comparison on $11$ different concepts with one fixed set of hyperparameters and does not provide a qualitative evaluation. Thus, our evaluation in this task is more comprehensive and gives more insight into how different models perform with different parameters.

For 1-Lipschitz architectures, we took a standard neural network that was specifically developed for smaller datasets. We are not aware of any other architecture baselines that guarantee exact 1-Lipschitzness and are developed for larger datasets such as ImageNet. The works we aware of do not go beyond smaller datasets in their experiments, see e.g. [1, 2, 3]. One of the factors for this absence could be the high computational complexity associated with 1-Lipschtiz layers.

Finally, regarding the NLP experiment, the main concern for us was the computational resources required for running larger models. RoBERTa experiment took us multiple weeks of compute time. Conducting experiments with multi-billion models would require the amount of computational resources that we unfortunately do not have. Nevertheless, we believe that our experiments are of interest, as they show the comparison with the closest competitor BOFT on a different domain apart from diffusion experiments.

Missing evaluations. Thank you for pointing this out. We added training times for the Table 2 (see the top-level comment). Both GSOFT and Double GSOFT are faster (and better in terms of metrics) than BOFT.

Missing comparisons. Thank you for this comment. We did an additional experiment with DoRA for subject-driven generation. We found that DoRA gives slightly better results compared to LoRA, but still shows significantly lower text similarity compared to GSOFT and DoubleGSOFT. See Table 1 and Figure 1 from attached PDF file. We also plan to add DoRA comparison for NLP experiment in the revisited version.

SVD during optimization steps. We actually did not use SVD at any step of the process. The SVD steps we describe when projecting to the class of GS matrices can be a useful tool for future applications. For example, for different initiazation strategies of GS-matrices.

[1] Singla et al. "Skew orthogonal convolutions." International Conference on Machine Learning. 2021

[2] M. Laurent, et al. "A dynamical system perspective for lipschitz neural networks." International Conference on Machine Learning. 2022

[3] X. Xiaojun, et al. "Lot: Layer-wise orthogonal training on improving l2 certified robustness." Advances in Neural Information Processing Systems. 2022

Thank you for your comments and suggestions. First of all, we add a source code for our experiments (see the top-level comment). Regarding the Monarch matrices, the key drawback in the Monarch representation for us is that if one block-diagonal factor has many small blocks, then the other one has to have several large dense blocks. As a result, we are not able to obtain as few parameters as we want, which was critical for our experiments. For example, in the natural language understanding experiment, obtaining number of parameters similar to LoRa with rank 8 is impossible with Monarch Matrices, which would require at least 4x parameters with any block structure. On the other hand, additional flexibility introduced in GS matrices allows for matrices with small number of blocks in each of the matrices, making it possible to use this method with any parameter budget. At the same time, the order-p generalization of Monarch matrices from the follow-up paper, is only capable of working with very specific matrix sizes (see also Appx. C). What is more, the used permutation matrices do not allow for optimally filling the matrix with non-zeroes using a given budget of parameters. Following your suggestions, we will emphasize these points in the revised version and add more discussion concerning the Monarch matrix class.