Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models

[W.1] Qualitative Evaluation

We appreciate the reviewer's suggestion. we have included additional qualitative samples in Fig. r2 of the Rebuttal-PDF. These samples further demonstrate Terra's ability to handle morphing under various scenarios.

[W.2] Comparative Analysis

We thank the reviewer for suggesting additional comparisons. In response, we have conducted experiments with CoVi and PMTrans, and the results are presented in Tab. r4 of the Rebuttal-PDF. Notably, Terra consistently improves performance in all tasks with those UDA methods, further verifying the effectiveness of our method.

[W.3] Clarity of Technical Details

Thank you for your comment. In the DG experiments, we first use contrastive learning loss to train a network $g(\cdot)$ to predict sample-level $t$ for Terra. This approach allows us to better capture inter-domain and intra-domain styles differences. After training, randomly sampling the value of $t$ can generate more diverse samples between domains. Specifically, we use two-dimensional $t$, with each dimension sampled from -2 to 2 at intervals of 0.1 to generate diverse samples. We will include these details in the revision. Moreover, we present the distribution of the learned time variable of the target domain in Fig. r4 of the Rebuttal-PDF. As illustrated, the random sampling of $t$ effectively covers the target domain, offering a clearer understanding of the rationale behind our approach that the generated samples may bring useful information for the target domain under the DG setting.

[Q.1] Scalability to Multiple Domains

We appreciate the reviewer's interest in the scalability of Terra. Our theoretical analysis indeed extends to multiple domains or tasks.

Theorem 1 can be generalized to multiple matrices using higher-order generalized singular value decomposition (HO GSVD) [r1]. Specifically, given K low-rank adapters $\Delta W_k \in \mathbb{R}^{m \times n}$ for K domains/tasks, we can concatenate the matrices horizontally and vertically to form $H$ and $V$, and let $k = \max\\{rank(H), rank(V)\\}$. Then, similar to Eqs. (14) and (15), based on HO GSVD, each matrix can be exactly factored as $\Delta W_i = Y K_i X$, where $Y \in \mathbb{R}^{m \times k}$, $K_i \in \mathbb{R}^{k \times k}$, and $X \in \mathbb{R}^{k \times n}$. We can then construct $\mathcal{K}(t)$ to satisfy $\mathcal{K}(t_i) = K_i$ for fixed $t_i$. In the case of multiple domains or tasks, we propose using a vector $\symbfit{t}_i$ with a small increase in parameters. The matrix $K_i$ can be constructed using interpolation methods, such as polynomial interpolation or spline interpolation, or non-linear time-varying matrices as those in Table 6.

In summary, this approach allows Terra to use one adapter structure to represent multiple LoRAs with fewer parameters. When the domains share knowledge—implying $k$ is small—the required parameters are further reduced. Additionally, for unknown tasks, we can determine the matrices $\mathcal{K}(t_i)$ using least squares or other optimization algorithms [r2], enabling a meta-learning approach.

[r1] Ponnapalli SP, Saunders MA, Van Loan CF, Alter O. A Higher-Order Generalized Singular Value Decomposition for Comparison of Global mRNA Expression from Multiple Organisms. PLOS ONE, 2011.

[r2] Friedland S, Torokhti A. Generalized rank-constrained matrix approximations. SIAM Journal on Matrix Analysis and Applications, 2007.

[Q.2] Comparison with Other LoRA Variants

For cross-domain learning based on MoLE, it can be seen as respectively training LoRAs on different domains first and then training a gating function to combine the trained LoRAs. While both MoLE and Terra are designed for customization of diffusion models, they differ in several key aspects:

• Objective: MoLE focuses on combining multiple pre-trained LoRAs to achieve multi-concept customization, whereas Terra aims to learn a single adapter structure that can capture multiple domains and construct a domain flow for generation.
• Training: MoLE only optimizes the gating function to preserve the characteristics of trained LoRAs on different domains, whereas Terra participates in the diffusion fine-tuning stage and aims to learn domain-general knowledge and domain-specific knowledge, allowing for control over different domains through a time variable.
• Expressiveness: MoLE uses a separate gating function for each LoRA layer, which requires entropy-based balancing to resolve conflicts when combining multiple LoRAs. In contrast, Terra achieves domain adaptation through a single time variable $t$, making it more stable. For two-domain interpolation, Terra and MoLE have similar expressiveness. Considering two domains with time variables $t_1$ and $t_2$, we have

$$$$

$

\Delta W(\alpha t_1 + (1-\alpha)t_2) = B\mathcal{K}(\alpha t_1 + (1-\alpha)t_2)A =(\alpha t_1 + (1-\alpha)t_2)BWA+ BA = \alpha \Delta W(t_1) + (1-\alpha) \Delta W(t_2).

$

$$This is equivalent to the linear arithmetic composition in MoLE. As shown in the response to [Q.1], this conclusion can be extended to interpolation with three or more LoRAs. Finally, the relation between MoLE and Terra is similar to that between **Gaussian Mixture Model (GMM)** and **Gaussian Process (GP)**. GMM composes a complex distribution by multiple Gaussian distributions, and GP is a distribution over functions within a continuous domain (such as time). Analogously, MoLE excels at the composition capabilities, while Terra excels at constructing a manifold.$$

[W.1] I feel the term "time" and "t" contradicts to the widely used time step t in diffusion model. Although I do see the authors used another symbol for diffusion models' timestep, it would be much better to use another term for the proposed "time" term to avoid confusion.

We appreciate the reviewer's concern. As you mentioned, we have used the symbol $\tau$ to differentiate our notation from the timestep in diffusion models. We intend to retain the term "time" because Terra draws inspiration from multiple research fields, including Fluid Dynamics and Control Theory, where "time" and "t" are also commonly used notations.

• In the context of fluid dynamics, for each time variable $t$, the matrix updates of Terra is $\Delta W(t)$, enabling a time-dependent velocity field $\frac{d}{d t} \Delta W(t)$. Terra constructs a "LoRA flow" in the parameter space based on the Lagrangian and Eulerian descriptions [r1]. Therefore, after the cross-domain diffusion fine-tuning, Terra demonstrates the ability to generate domain flow.

• From a control theory perspective, Terra can be viewed as solutions to time-varying systems. For instance, consider a linear system $\dot x(t) = A x(t)$, where the "state vector" $x(t)$ can be seen as the continuous image feature across different domains, and a closed-form solution is $x(t) = e^{A(t)} x_0$ [r2]. Here is the "exponential" form Terra with the time-varying matrix function.

To clarify, we will provide a more detailed explanation of $\tau$ used in the diffusion model. Specifically, "since $t$ in this paper refers to the time variable in Terra, we use $\tau$ here to represent the timestep $t$ in the diffusion model".

[r1] Villani C. Optimal transport: old and new. Berlin: springer, 2009, Pages 26.

[r2] Williams R L, Lawrence D A. Linear state-space control systems. John Wiley & Sons, 2007, Pages 52-55.

[W.2a] I appreciate the provided possible forms of Terra in Tab 6. Yet, I am wondering if the authors can provide more insights/principles in terms of how to choose the function form.

We appreciate the reviewer's interest in the possible forms of Terra presented in Table 6. To provide more insights, we elaborate on the guiding principles behind the choice of the three forms:

• Linear: The $tW+I$ is the simplest form, related to a straight and steady flow, which is sufficient for two domains according to Theorem 1 and 2. Its constant velocity of weight changes ensures smooth morphing and is suitable for simple interpolating between two domains under the UDA setting.

• Cosine-Sine: This form is adopted because of the bounded range and non-linearity of trigonometric functions, preventing image collapse during generation and enabling a complex parameter manifold to capture relationships between multiple domains. We recommend using this form in complex scenarios, such as interpolating multiple domains in DG.

• Exponential: $e^{tW} = I + \sum_{k=1}^{\infty} \frac{t^k}{k!} W^k$, implemented using torch.matrix_exp, also defines a smooth curve in a high-dimensional manifold. This form is more expressive and suitable for handling multiple domains, as it enables feature transformations as mentioned in our response to [W.1]. Notably, it related to three types of transformations: scalings, rotations, and shears [r3].

[r3] Ronald N. Goldman, VII.3 - Transformations As Exponentials, Graphics Gems II, 1991, Pages 332-337.

[W.2b] Besides, there seems missing an ablation on the different functions for Terra, to justify why "Linear" is used for generative interpolation/UDA and "cosine-sine" is used for DG?

We appreciate the reviewer's comment and would like to provide clarification on this point.

• For tasks involving only two domains, such as generative interpolation and UDA, the expressive abilities of the "Linear" and "cosine-sine" functions are equivalent. Specifically, when $t=0$, both terms of the middle matrix reduce to the identity matrix $I$, and when $t=1$, the learnable matrices are constrained equally.
• However, for tasks involving more domains, such as DG, we have conducted an empirical ablation study to investigate the effectiveness of the "cosine-sine" form compared to the "linear" form. The results, presented in Table 7 of the manuscript and reproduced below, demonstrate the superiority of the "cosine-sine" form on the PACS dataset in DG scenarios. \begin{array}{lcccccl} \hline Method & \text{A} & \text{C} & \text{P} & \text{S} & \text{Average} \newline \hline \text{Linear} & 87.47 & 80.17 & 97.85 & 77.16 & 85.66 \newline \text{Cosine-sine (dim=1)} & 88.29 & 82.36 & 97.53 & 73.31 & 85.37 \newline \text{Cosine-sine (dim=2)} & 89.51 & 79.66 & 98.20 & 78.64 & 86.50 \newline \hline \end{array}

[W.1] The presentation can be improved. I suggest elaborate and provide more details on sec. 3.3 and to move Fig. 7 in appendix to the main paper. The implementation details of morphing between style and subject should be explained here.

We will follow your suggestion by (a) relocating Fig. 7 to Sec. 3.3 and (b) enhancing the organization of Sec. 3.3 to better illustrate how to construct evolving visual domains.

• Stage 1: Fine-tune the parameters of Terra (i.e., $\Delta_\theta=W_{up}\cup W_{mid}\cup W_{down}$) using the loss function defined in Eq. (5), where the first part with $t=0$ uses source samples $\mathcal{D}_S$ and the second part with $t=1$ uses target samples $\mathcal{D}_T$.
• Stage 2: Generate an intermediate domain by (a) uniformly sampling $t$ from [0,1] and (b) inputting the text prompt and a random noise into the fine-tuned diffusion model corresponding to domain $t$ (i.e., $\theta_0 + \Delta W(t)$ where $\theta_0$ is the pre-trained diffusion model) for the backward process.

Regarding the implementation details of generative interpolation tasks, including those involving morphing between image pairs, styles and objects:

• We will follow the reviewer's suggestion to introduce a new section on Generative Interpolation via Terra between Secs. 3.3 and 3.4. This section will list generative interpolation as one of the three concurrent applications of the proposed domain flow generation framework of Terra. It will present the details originally provided in Line 241-256 and 604-614.
• For convenience, we also provide the details here.
  • Morphing in image pairs: In Stage 1, we instantiate the loss function in Eq. (5) by setting (a) $\mathcal{D}_S$ to include one image of the pair, (b) $\mathcal{D}_T$ to include the other image, and (c) the text prompt to describe the images. In Stage 2, we generate intermediate images by uniformly transiting $t$ from 0 to 1 using the same text prompt as in Stage 1; each value of $t$ results in an interpolated image.
  • Morphing in styles/objects: This differs from morphing in image pairs only in $\mathcal{D}_S$ and $\mathcal{D}_T$ used in Stage 1, which are a group of images in one style/object and a group of images in another style/object, respectively.

[W.2a] The authors should provide qualitative comparisons.

We provide qualitative comparisons of our method with all other baselines in Fig. r1 of the Rebuttal-PDF, which shows that Terra could generate intermediate images that are visually smooth and natural.

[W.2b] Terra does not show performance improvement over DiffMorpher or LoRA Interp. in PPL. Comparisons should include off-the-shelf UDA/DG+other similar morphing works for a fair comparison.

We'd like to humbly clarify the potential misunderstandings.

First, Terra is a general framework that constructs a continuous parameter manifold and thus generates a domain flow. To illustrate, in this paper, we show the three representative applications of Terra, including image morphing, UDA, and DG.

Second, image morphing, UDA, and DG are three parallel applications of Terra, so that the roles of Terra in them differ.

• In image morphing, similar to DiffMorpher or LoRA Interpolation, Terra generates a continuously interpolative sequence of images.
• In UDA, as shown in Fig. 2(a), Terra bridges the gap between the source and target domains by generating more target-like samples from source samples and random noise. Thus,
  • in UDA, image morphing, whose output is an interpolative sequence of images, cannot be directly applied into off-the-shelf UDA methods.
  • for a fair comparison with prior arts in Terra's effectiveness in generating more target-like samples, we add a comparison against off-the-shelf UDA + direct generation of target style samples using prompts on Office-Home:
    • SDXL (target): We use the prompt A [CLASS] in the style of [TARGET] to generate samples, where [CLASS] and [TARGET] denote the placeholders for the label name and target domain name, respectively.
    • SDXL (target styles): We prompt GPT-4 with Generate 50 prompts describing [TARGET] style for image generation to obtain more detailed style descriptions for synthesis.
    The results in Tab. r3 of the Rebuttal-PDF verify Terra's effectiveness in generating more target-like samples.
• In DG, as shown in Fig. 2(b), Terra expands the source domains by (a) learning a $t$ predictor that maps each source domain to [-1,1] and (b) randomly sampling values of $t$ to generate more domains.
  • Image morphing can be adapted to off-the-shelf DG methods by including interpolative sequences of all pairs of images from different source domains, despite being computationally expensive.
  • For a fair comparison in Terra's effectiveness in expanding source domains that generalize better, we include the comparison against off-the-shelf DG + morphing works on Office-Home. That is, we train a LoRA for each domain and adopt LoRA Interp./DiffMorpher to interpolate. The results in Tab. r2 of the Rebuttal-PDF verify Terra's effectiveness, since Terra interpolates between domains instead of images and thus better models distributions in two domains.

Third, even on the image morphing, Terra outperforms the DiffMorpher, which specifically designed for morphing by customized techniques such as attention interpolation, adaptive normalization, and a new sampling schedule.

• Terra supports morphing of styles and objects, which DiffMorpher seems incapable of.
• Terra enjoys better rationality and fidelity (measured by FID) than DiffMorpher.
• Terra is competitive in smoothness and consistency (measured by PPL) with DiffMorpher, despite being simple and general without any specific design. Moreover, equipped with those customized techniques used in DiffMorpher, Terra is even better than DiffMorhper (see Tab. r1 of the Rebuttal-PDF).

[W.1] The method essentially adopts the LoRA approach and constructs a low-rank parameter manifold through F(W,t)=tW+I. This can be seen as an interpolation version of LoRA with fewer parameters.

Terra constructs a "LoRA flow" in the parameter space, and is NOT an interpolation version of LoRA with fewer parameters. Here are three key differences:

• Formulation: $tW+I$ is just one instance of Terra. Other possible forms are listed in Table 6 of the Appendix, i.e., $\exp(t{W})$ and $\cos(t{W})$.

• Training: Unlike LoRA interpolation, which needs training separate LoRAs for different domains, our method only needs to train one adapter for multiple domains. In our training, the middle time-varying matrix $\mathcal{K}(t)$ is domain-specific, while the matrices $W_{up}$ and $W_{down}$ are shared across domains, enabling domain-general knowledge learning.

• Application: The interpolation between two domain is just one application of our Terra. More importantly, based on Terra, we can design effective frameworks for UDA and DG:

  • For UDA, our framework can learn the domain-general subjects and domain-specific styles due to the Terra structure, enabling the generation of target-like samples from both source samples and random noise, thereby reducing the domain gap.
  • For DG, Terra constructs a domain manifold in the parameter space, facilitating random interpolation to generate diverse samples, which enhances the model's generalization ability.

[W.2] There is a lack of more direct comparative experiments, as mentioned in the Questions.

Please refer to our following response to [Q.1a].

[W.3] As a method that fine-tunes SD, the paper would benefit from directly evaluating the quality of generated images using different fine-tuning methods (such as those mentioned in lines 58-61).

• The methods mentioned involve customized image generation tasks, such as image editing and multi-concept generation, focusing primarily on single-domain image generation. In contrast, our approach enhances the LoRA structure to create a continuous parameter manifold, allowing for image generation across a domain flow.

• A related work is Diffmorpher, which trains two LoRAs on image pairs and introduces techniques for generating a continuously interpolative sequence of images, referred to as image morphing. We compare Diffmorpher with our method in the manuscript.

• To demonstrate that the paper's potential, we conduct an additional experiment by replacing the two LoRAs with our proposed Terra. The combined method "Terra + Diffmorpher" yields improved FID and PPL scores. The qualitative and quantitative results are presented in Fig. r1 and Tab. r1 of the Rebuttal-PDF.

[W.4] Lack of discussion of limitations and failure cases.

• Limitations: We have thoroughly discussed the limitations in Appendix D. Additionally, while we have adapted to downstream domains through fine-tuning, our model may still be influenced by the prior of the foundation model to some extent.

• Failure cases: We acknowledge that a small number of generated images may exhibit poor quality due to the conflict between SD prior knowledge and the knowledge required for downstream tasks. We showcase some failure cases in Fig. r3 of the Rebuttal-PDF. However, the number of those poor-quality images is small, and it does not affect the overall performance of the model.

[Q.1a] I recommend adding a comparison with data augmentation with SD's prior knowledge to demonstrate Terra's design advantages. Alternatively, I encourage the authors to illustrate the respective contributions of Terra and the SD prior to the performance boost through other reasonable means.

We appreciate the reviewer's suggestion to explore the prior of foundation models. To address this concern, we design several methods to synthesize data based on the SDXL model and evaluate their effectiveness on UDA tasks:

• (1) SDXL (random): We use the prompt A [CLASS] to generate samples for each class, where [CLASS] denotes the placeholder for the label.
• (2) SDXL (styles): We first use the prompt Generate 50 prompts describing diverse styles for image generation to ask GPT-4, and then use the prompt A [CLASS], an everyday object in office and home, in the style of [STYLE] to generate samples, where [STYLE] denotes the placeholder for style prompts generated by GPT-4 (e.g. "Classic", "Modern").
• (3) SDXL (target): Based on (2), we use the name of target domain (e.g. "Clipart") to replace the [STYLE] as the new placeholder for exploring the SD prior on the target domain.
• (4) SDXL (target styles): We use the prompt Generate 50 prompts describing [TARGET] style for image generation to ask GPT-4 and obtain more detailed style prompts for synthesis.
• (5) SDXL (selected): Inspired by [2], we use a confidence-based activate learning method to filter out poor-qulity and misclassified samples generated in (4) and select valid samples.

The comparison results on Office-Home for UDA are shown in Tab. r3 of the Rebuttal-PDF. Terra outperforms the comparison methods, indicating that despite the boost in accuracy from target style design and active learning, the prior knowledge is insufficient to align with the downstream tasks. This issue can be furthur mitigated through finetuning with Terra, which demonstrates Terra's design advantages.

[Q.1b]: I noticed a minor issue in Table 1: DGP is based on a GAN model, while Terra uses SD XL. Therefore, a direct numerical comparison is unfair. I suggest the authors include the base model for clarity.

Thank you for pointing out this issue. Here, we compare with GAN-based model DGP and the stable diffusion-based methods DDIM, LoRA Interpolation, and DiffMorpher. In our revision, we will include the base models for clarity. You can find the updated Tab. r1 and Fig. r1 of the Rebuttal-PDF.