DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models

We sincerely thank the reviewer for providing such thorough and constructive feedback. We greatly appreciate the time and effort invested in reviewing our work, and we acknowledge that the raised concerns are valid and important for strengthening our paper.

1. Insufficient Justification for Flexibility Enhancement

Response: We thank the reviewer and the flexibility claims will be supported with additional justification in the final draft.

Theoretical Foundation: We would like to clarify that the DEFT's design is motivated by the below limitations of existing methods: (i) PaRa operates on the column subspace of the original pre-trained model, aiming to modify weights while preserving original information, but it cannot simultaneously add new concepts or styles, limiting its flexibility for novel concept adaptation. (ii) Conversely, LoRA can effectively learn new concepts but may lose some capabilities of the original model, which is particularly problematic in text-to-image models where maintaining original capabilities (prompt following, object combination, image editing) while personalizing new concepts is crucial.

DEFT address both these limitations by operating on two complementary subspaces. For a given original weight W₀ with SVD: W₀ = UΣV^T:

• LoRA constraint: Updates AB^T can interfere with existing singular structure
• DEFT advantage:
  • (I - PP^T) operates in col(P)⊥ (orthogonal complement)
  • PR operates in col(P)

This dual control enables: (1) Selective modification through (I - PP^T), allowing preservation or removal of specific existing directions, and (2) Addition of entirely new orthogonal directions through PR.

Empirical Evidence: The flexibility of DEFT design can also also observed in the quantitative results presented in our paper as copied below:

Positional Flexibility Analysis: We provide quantitative results demonstrating DEFT's 3D positional awareness capabilities:

Visual Demonstrations:

• Figure 1 - Scene Personalization: Office scene control showing incremental additions (desk+computer → +books → +coffee → +plant → +papers → +clock), and Church Rock positioning across diverse environments (pool, city, mountain, beach, forest)
• Figure 5 - Novel Composition: Training set adaptation showing diverse combinations under varying testing conditions beyond original training data
• Supplementary Figure 4: Multi-task versatility across realistic photographs, abstract interpretations, segmentation, depth mapping, and edge detection

As suggested, we will included, additional quantitative analysis to further strengthen our flexibility argument.

2. FID Performance Analysis

Response: We appreciate the reviewer's attention to this important metric. We acknowledge that our FID performance requires honest explanation.

The FID versus SSIM performance pattern reflects a fundamental design trade-off in DEFT's architecture. DEFT employs a conservative adaptation strategy: W = (I - PP^T)W₀ + PR, where (I - PP^T)W₀ actively preserves pre-trained knowledge while PR enables controlled adaptation.

This design intentionally prevents aggressive style deviations while learning new concepts, resulting in more structurally consistent generations that maintain better SSIM scores but may appear less visually striking to FID's perceptual metrics. DEFT explicitly trades raw generation quality (FID) for structural consistency (SSIM), making it particularly suitable for applications where consistency and controllability are prioritized over pure visual novelty.

We believe this trade-off is valuable for many practical applications, though we acknowledge it may not be optimal for all use cases.

3. Emergent Properties Claims

Response: We sincerely acknowledge this valid concern and agree that our current title may overstate our claims relative to the experimental evidence provided.

While Section 4.2.4 demonstrates what we consider emergent behavior through Figure 5 (multi-concept spatial composition from single-concept training) and VisualCloze results (single adapter handling 8+ distinct tasks), we recognize this evidence requires more rigorous quantitative analysis to support such prominent claims.

In response to this valuable feedback, we propose revising the title to focus on our core technical contribution: "Dual-decomposition Column Space Extension for Fine-tuning of Text-to-Image Models"

This revision better reflects our mathematical foundation while avoiding overstated claims about emergent properties.

4. User Study and Perceptual Improvements

Response: We appreciate the suggestion for more rigorous evaluation. We conducted both human and automated evaluations:

Human User Study Results:

Automated Evaluation (Claude Sonnet Judge):

We acknowledge that a larger-scale user study would further strengthen these claims.

5. DreamBooth PaRa Comparison

Response: We thank the reviewer for this important suggestion. We have provided comprehensive comparisons with PaRa in our supplementary results using the DreamBooth dataset with SDXL baselines:

6. Typography Corrections

Response: We sincerely apologize for these oversights and will carefully address all typography issues:

Corrections to be made:

• L176: "dreamboot" → "DreamBooth"
• L173, L231: "DreamBench plus" → "DreamBench Plus"
• L262, Fig 6: "eqn" → "Eq."
• Fig 5: "omnigen" → "OmniGen"
• Standardize all instances to "VisualCloze"
• L248: "omnigem" → "OmniGen"
• Fig 6: "with out" → "without", "W_0" → proper LaTeX formatting

We deeply appreciate the reviewer's attention to detail and will ensure careful proofreading in our revision.

Thank you for providing your valuable feedback!

Ablation Studies: Component-wise Analysis

Response: We provide comprehensive ablation analysis using our actual experimental results:

Individual Component Performance (DreamBooth Dataset, CLIP-Image scores):

Core Difference: Coordinated vs Independent Adaptation

• PaRa only: W' = W₀ - QQ^T W₀
• LoRA only: W' = W₀ + AB^T
• Composition: W' = W₀ - QQ^T W₀ + AB^T (independent Q, A, B)
• DEFT: W' = W₀ - PP^T W₀ + PR (shared P matrix)

Parameter Count Comparison:

Key Insight: DEFT achieves the benefits of composition with 40% fewer parameters than naive combination (4.7M vs 7.8M parameters).

Performance Efficiency Analysis:

Based on our authentic individual results, we can analyze the efficiency gains:

Results:

DEFT Training Advantage: 50% faster than expected naive composition while using 40% fewer parameters.

Architectural Coordination Benefits:

Composition Problems:

W' = (I - QQ^T)W₀ + AB^T
     ↳ Q removes information from W₀
     ↳ A,B add unrelated information
     ↳ Potential interference between operations

DEFT Coordination:

W' = (I - PP^T)W₀ + PR
     ↳ P defines removal space
     ↳ R operates within P's coordinate system
     ↳ Guaranteed alignment between operations

Decomposition Results:

Our actual decomposition ablation demonstrates DEFT's flexibility:

Summary:

1. Parameter Efficiency: DEFT uses 40% fewer parameters than naive PaRa+LoRA composition
2. Performance Superiority: DEFT outperforms both individual methods and expected composition performance
3. Training Efficiency: 50% faster training than expected composition approach
4. Architectural Advantage: Coordinated subspace adaptation prevents interference between removal and addition operations

Note: We acknowledge that direct experimental comparison with naive composition would strengthen this analysis. The theoretical analysis based on our authentic individual results demonstrates DEFT's fundamental optimization advantages over the simple method combination.

Thank you for providing your valuable and constructive feedback!

1. Title and Central Claims

Response: We acknowledge this valid concern and agree the current title overstates our claims relative to the experimental evidence provided. While Section 4.2.4 demonstrates genuine emergent behavior such as multi-concept spatial composition from single-concept training (Figure 5) and single adapter handling 8+ distinct tasks through unified training (VisualCloze results), we recognize this evidence requires more rigorous quantitative analysis to support such prominent title claims.

We propose revising the title to focus on our core technical contribution: "Dual-decomposition Column Space Extension for Fine-tuning of Text-to-Image Models". This better reflects our mathematical foundation where column space extension col(W_total) = col(W_reduce) + col(QR) enables the demonstrated capabilities without overstating the emergent properties aspect.

2. Decomposition Methods Analysis

Response: We provide a comprehensive quantitative evaluation demonstrating distinct performance characteristics across decomposition methods:

Relexing P achieves the highest CLIP-I and DINO-V1 scores, while all methods maintain consistent performance levels, demonstrating the robustness and flexibility of our framework. QR maintains quality, strikes a balance between quality and personalization. The results show NMF excels in image-text alignment and feature representation, while QR provides the best aesthetic quality (0.0166). This validates our claim that different decomposition methods offer distinct advantages for specific tasks.

3. Inference Efficiency Analysis

Response: We acknowledge that comprehensive efficiency claims should include detailed training and inference analysis. We conducted extensive benchmarks comparing DEFT against LoRA and PaRa across different configurations:

Training Efficiency Analysis:

Rank 64 Configuration (Batch size = 2):

Key Performance Insights:

1. Training Speed Hierarchy: DEFT shows moderate overhead (23% slower than LoRA at rank 8, 8% slower at rank 64)

2. Memory Efficiency:

   • At rank 8: DEFT uses 7.71GB vs LoRA's 7.72GB (marginal improvement)
   • At rank 64: DEFT uses 7.95GB vs LoRA's 8.03GB (1% improvement)

3. Parameter Efficiency Trade-offs:

   • DEFT achieves competitive parameter efficiency without sacrificing performance

Scalability Analysis: The performance gap between DEFT and LoRA decreases at higher ranks (23% → 8% slowdown), suggesting DEFT's overhead becomes relatively smaller for larger adaptations. This indicates favorable scaling properties for complex fine-tuning scenarios.

The efficiency analysis reveals DEFT occupies a sweet spot: trading minimal speed for substantial gains in memory efficiency and output quality, making it optimal for deployment scenarios where resource constraints and quality requirements are balanced priorities.

Furthermore, we conducted extensive benchmarks (928 configurations) comparing DEFT against LoRA and other baselines:

Inference Efficiency Benchmark (Omnigen 3.8B model 1 layer):

Decomposition Method Performance vs Speed Analysis

Key Findings: DEFT achieves parameter efficiency of 5.18% while maintaining inference speed within 20% of LoRA. With precomputed caching of (I - QQ^T)W during deployment, DEFT's inference cost reduces to within 10-20% of LoRA while preserving superior task performance.

4. Missing Literature Comparisons

Response: Thank you for this important suggestion. We have extended our comparisons to include OFT and acknowledge DoRA's relevance:

Extended Comparison with OFT:

OFT requires additional orthogonality constraints during training (R^T R = RR^T = I, ||R - I|| ≤ ε) which increases computational overhead, whereas DEFT requires no additional constraints. Regarding DoRA, while it decomposes weight updates into magnitude and direction components (W' = W + m * (V * B^T / ||V * B^T||)), it requires tuning magnitude scaling and direction learning rates, unlike DEFT's parameter-free approach. We will incorporate these methods in the expanded Section 2.

5. R Matrix Motivation

Response: DEFT's innovation lies in its dual learning approach. The projection term (I - PP^T)W₀ strategically modifies the pre-trained model's representation space, while the R matrix actively learns new task-specific knowledge to populate that modified space.

Intuitive Explanation: Consider the original model with col(W₀) = span{v₁, v₂, v₃}, but a new task requires direction v₄.

Without R: W = (I - PP^T)W₀ → col(W) ⊆ span{v₁, v₂, v₃} (cannot represent v₄)

With R: If P learns such that col(P) contains v₄, then W_DEFT = (I - PP^T)W₀ + PR enables col(W_DEFT) ⊆ span{v₁, v₂, v₃} + span{v₄}, where R learns to map inputs to the v₄ direction.

This transforms adaptation from mere 'knowledge rearrangement' to active 'knowledge extension'.

6. OmniGen Comparison Fairness

Response: Thank you for seeking this clarification. We used the same base models as VisualCloze for evaluation. OmniGen was not fine-tuned on the full VisualCloze dataset as it leverages in-context learning for generalization. For fair comparison, we fine-tuned both DEFT and LoRA on the depth estimation task using 200k image-instruction pairs:

This controlled comparison demonstrates DEFT's consistent advantages even in fair evaluation settings.

7. FID Performance Trade-off Explanation

Response: The FID vs SSIM performance pattern reflects a fundamental design trade-off in DEFT's architecture. DEFT employs a conservative adaptation strategy: W = (I - PP^T)W₀ + PR, where (I - PP^T)W₀ actively preserves pre-trained knowledge while PR enables controlled adaptation.

This design prevents aggressive style deviations while learning new concepts, resulting in more structurally consistent generations that maintain better SSIM scores but may appear less visually striking to FID's perceptual metrics. DEFT explicitly trades raw generation quality (FID) for structural consistency (SSIM), making it particularly suitable for applications where consistency and controllability are prioritized over pure visual novelty.

8. Limitations and Societal Impact

Response: We acknowledge this significant omission and will include a dedicated limitations section addressing:

Societal Impact: Potential misuse for deepfake generation (standard concern for generative models) Limitations: Limited instruction dataset coverage affecting novel combinations

We will have a separate section for limitations and societal Impact in the revised manuscript.

9. Formatting Issues

Response: We acknowledge the formatting concern regarding supplemental content in the main paper. We will relocate the mathematical proofs for PaRa and DEFT to the supplementary material and ensure strict adherence to formatting guidelines in the revision.

We appreciate the reviewer's thorough analysis and will address all raised concerns through title revision, expanded efficiency analysis, comprehensive literature comparison, dedicated limitations section, and proper formatting in our revision.

Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment.

1. Literature Review on Training-Free Strategies

Response: Thank you for this valuable suggestion. We acknowledge that training-free approaches represent an important category in model adaptation. Our current manuscript includes several training-free methods such as Textual Inversion variants, ELITE, and InstantBooth that control generation through text-encoder modifications.

However, these training-free methods have significant limitations compared to DEFT. They struggle with extending to new concepts, especially out-of-distribution images, and rely on conditional components that are incompatible with unified models where image and text tokens are processed through unified self-attention. Additionally, many require per-image optimization, resulting in higher inference times. DEFT achieves superior subject fidelity with CLIP-I scores of 83.59 vs 74.21 compared to Textual Inversion variants on the DreamBooth Plus dataset (please see table 1 in the supplementary material).

2. Quantitative Ablation Study on Decomposition Effects

Response: We provide a comprehensive quantitative evaluation of different decomposition methods with finetuning omnigen on sks dog from dreambooth dataset for 200 epochs each:

Relexing P achieves the highest CLIP-I and DINO-V1 scores, while all methods maintain consistent performance levels, demonstrating the robustness and flexibility of our framework. QR maintains quality, strikes a balance between quality and personalization.

3. Visual Comparison and User Study

Response: We agree that visual quality appears similar in Figure 3, but DEFT demonstrates superior instruction-following capabilities. We conducted a comprehensive user study with 16 participants comparing PARA, LoRA, and DEFT across different scenarios:

Additionally, we conducted a Claude Sonnet evaluation across multiple metrics:

DEFT consistently outperforms both methods across all evaluation criteria.

4. Convergence Speed Analysis

Response: Thank you for highlighting this important aspect. We conducted a convergence analysis comparing DEFT and LoRA at different training steps on the full Dreambooth (30 concepts) on omnigen:

DEFT shows consistent improvement across all metrics and achieves better convergence characteristics. Our method modifies the column subspace of the pretrained model, providing advantages for convergence with both large and small image datasets.

Convergence Characteristics: Because the baseline model is very large (around 3.8 B), it is difficult to prove the convergence in a small dataset. We found that training loss is very fluctuating for all the cases. Here are some insights:

DEFT achieves the lowest training loss (0.0240) significantly faster, at epoch 4, compared to LoRA, which requires 27 epochs, and PaRa, which requires 12 epochs. Note that faster convergence does not guarantee the quality of image generation, and the performance is also not comparable, so scores are calculated after 200 epochs only. The faster convergence stems from our method's ability to modify the column subspace of the pretrained model, providing inherent advantages for both large and small image datasets.

5. VisualCloze Fine-tuning Details

Response: Thank you for this clarification request. We used the same base models as VisualCloze for evaluation. OmniGen was not fine-tuned on the full VisualCloze dataset as it leverages in-context learning for generalization. For fair comparison, we fine-tuned both DEFT and LoRA on the depth estimation task using 200k image-instruction pairs:

DEFT consistently outperforms LoRA across all metrics in this controlled comparison.

6. Emergent Properties Evidence

Response: We appreciate this feedback and provide clarification of our emergent properties definition. By emergent properties, we refer to capabilities that arise from DEFT's dual decomposition structure without explicit training for those specific combinations.

Our existing evidence demonstrates multi-concept composition from single-concept training data, as shown in Figure 5 where individual concepts (single dog images, separate human images) combine to create complex spatial reasoning scenarios. The VisualCloze results further demonstrate that a single DEFT adapter handles over 8 distinct tasks with unified capability and emergent transfer between modalities.

We acknowledge this could be presented more comprehensively and will strengthen this section with additional generation results showing completely new task combinations.

7. Failure Cases and Limitations

Response: We appreciate this important question. Our primary limitation occurs with complex scene compositions involving multiple novel elements. For example, prompts like "generate Einstein + new features + dog + car + tree + house in a kitchen setting" can produce mixed or blended features when the original model and DEFT training dataset haven't encountered such complex combinations.

The quality bottleneck often stems from limited instruction dataset coverage. When attempting to compose highly complicated combinations of objects, spatial relationships, and contexts, the method may struggle with feature disambiguation. We will expand this analysis in the main paper and provide a more comprehensive discussion of limitations and societal impacts.