KIND: Knowledge Integration and Diversion for Training Decomposable Models

Dear Reviewer Dcjo,

We sincerely appreciate your insightful comments and your recognition of both the novelty and soundness of our methods, as well as the contribution of our benchmark to the community. Below, we provide a detailed response.

Q1: Discussions on the limitations of the proposed method.

We have briefly discussed the limitations in Section 7 of the original manuscript. Here, we further elaborate on these limitations and outline potential directions for future improvements:

• Limitations in Class-Conditional Generation. To illustrate the transferability of class-agnostic knowledge obtained via knowledge diversion, we focus on class-conditional generation tasks, where variations induced by different class labels naturally introduce downstream tasks with substantial domain shifts.

  In contrast, text-conditional generation, which controls image content via prompts, is also widely adopted. However, large-scale text-to-image diffusion models (e.g., Stable Diffusion 3) are pre-trained on a broad spectrum of internet images, making it challenging to define tasks with significant domain shifts. Moreover, extending KIND to text-conditional generation requires transitioning class gate from discrete, countable class labels to an open-ended, uncountable prompt space, where a binary class gate is insufficient.

  Future work will explore Mixture-of-Experts (MoE) techniques for knowledge diversion, enabling dynamic allocation of a limited set of tailors based on prompts. We have conduct preliminary experiments using a PixArt-based model trained for 50,000 steps, see Tabel below. KIND may underperform compared to parameter-efficient fine-tuning (PEFT) methods such as LoRA, as pre-trained text-to-image diffusion models inherently bridge moderate domain differences.

  Dataset: MRI	CLIP Score↑	LPIPS↓
  PixArt-Lora	33.88	0.4252
  PixArt-KIND	33.20	0.4213

  Dataset: Pokemon	CLIP Score↑	LPIPS↓
  PixArt-Lora	32.48	0.4279
  PixArt-KIND	32.55	0.4288

• Limitations in Structural Expansion. KIND has been applied exclusively to Transformer-based architectures, focusing on knowledge diversion in Multi-Head Self-Attention ($W_q$, $W_k$, $W_v$, $W_o$) and Pointwise Feedforward layers ($W_{in}$, $W_{out}$). While DiT is becoming the dominant architecture for diffusion models, many classic diffusion models still rely on convolution-based UNets.

  Extending KIND to architectures dominated by convolutional layers presents a key challenge. Although convolutional weights can be represented as three-dimensional tensors and prior work has explored SVD-based decomposition for convolutional layers, the strong inductive biases of convolutional kernels pose unique difficulties. Developing effective knowledge diversion strategies for convolutional networks remains an important direction for future research.

• Limitations in Other Tasks (e.g., Image Classification). KIND has been primarily evaluated on image generation tasks, yet its knowledge diversion mechanism—encapsulating class-agnostic and class-specific knowledge into distinct network components—suggests broader applicability.

  A particularly promising direction is cross-domain few-shot learning, where models must generalize across domains with limited data. Traditional methods often struggle under large distribution shifts due to their reliance on prior knowledge from the source domain. KIND offers a key advantage: learngenes serve as a transferable backbone for stable adaptation, while tailors enable task-specific fine-tuning with minimal data, improving generalization.

  However, unlike image generation, image classification lacks access to class labels during inference, requiring each image to traverse all tailors to extract features, leading to increased computational overhead. As the number of classes grows, classification complexity further escalates. Preliminary experiments applying KIND to ViT for cross-domain few-shot learning (see Tabel below) demonstrate significant improvements over baselines, though a performance gap remains compared to state-of-the-art methods. Thus, developing an efficient learngene-tailer framework for classification remains an open research direction.

  	ChestX	ISIC	EuroSAT	CropDisease	Average
  Vanilla ViT (Baseline)	26.3	46.1	88.6	94.6	63.9
  P>M>F	27.3	50.1	86.0	93.0	64.1
  StyleAdv	27.0	51.2	90.1	96.0	66.1
  KIND	26.5	47.9	88.8	95.0	64.6

We will provide a detailed discussion of these limitations and future research directions in the revised Appendix to inspire further advancements in KIND and broaden its application scope.

Dear Reviewer Y7yt,

We sincerely appreciate your insightful feedback and your recognition of the innovation and practicality of our work. Below, we provide our detailed response.

Q1: Lack of comparison of similar methods (e.g., FacT and BOFT).

FacT and BOFT leverage matrix decomposition techniques relevant to this work but are fundamentally Parameter-Efficient Fine-Tuning (PEFT) methods. They focus on decomposing pre-trained weight matrices to identify compact parameter subspaces, that can be efficiently fine-tuned for adapting pre-trained models to novel tasks. However, these methods heavily rely on traditional pre-trained models, which are typically fixed in size, structurally inflexible, and risk negative transfer.

In contrast, KIND introduces a novel pre-training paradigm that explicitly decomposes knowledge into class-agnostic and class-specific components, encapsulated in learngenes and tailors, respectively. This approach results in decomposable pre-trained models, where the modular design enhances transferability while enabling task-specific adaptation, effectively addressing the limitations of traditional pre-training.

Table 2 in the manuscript highlights the superior transferability of learngenes by comparing KIND with PEFT-based methods, including SVD-based approaches (e.g., SVDiff, PiSSA) related to FacT, and OFT-based methods related to BOFT. Per your suggestion, additional comparisons with FacT and BOFT (see Table below) further demonstrate that KIND consistently outperforms these PEFT methods, underscoring its ability to transfer only class-agnostic knowledge while avoiding the deployment challenges and the redundant, biased, or harmful transfer often associated with traditional pre-trained models on which PEFT approaches typically rely.

Q2: Application in the limited resources scenario is not highlighted in experiments.

The decomposable model pre-trained by KIND can be flexibly restructured to accommodate computational constraints, enabling efficient deployment on mobile and edge devices. Although direct evaluation on mobile hardware is beyond our current resources, we approximate its feasibility by analyzing FLOPs, memory footprint, and extrapolating inference latency on modern mobile chip.

We construct a mobile-compatible DiT using KIND and evaluate its efficiency across three state-of-the-art mobile chip: Apple A18, Kirin 9020, and Snapdragon 8 Gen3, which sustain 1907, 1720 and 2774 GFLOPS, respectively.

As shown in Table below, KIND-Mobile achieves a 3.3$\times$ reduction in FLOPs and a 1.8$\times$ reduction in memory usage compared to traditional pre-training, while maintaining strong generative performance (FID=21.14). Notably, inference latency remains under 4 seconds across all tested mobile chip, demonstrating KIND’s adaptability in resource-constrained environments.

Q3: Supplement the comparison of speed in Table 2.

To further emphasize the computational efficiency of KIND on novel tasks, we report the GPU time of different methods in the Table below, following your suggestion.

KIND achieves the best performance with the most efficient training compared with state-of-the-art PEFT methods. This is attributed to its encapsulation of class-agnostic knowledge into learngenes through knowledge diversion, thus enhancing structural flexibility.Notably, under large domain shifts, transferring only learngenes improves the adaptability of the pre-trained model while significantly enhancing transfer efficiency by reducing model parameters through the elimination of redundant class-specific knowledge encapsulated in tailors.

Q4: Introduce a README part in the code.

Thank you for your suggestion. We will include a comprehensive README file in the future open-source release to facilitate the reproduction of KIND.

Dear Reviewer QbMr,

We sincerely appreciate your valuable comments and recognition of our work’s innovation and performance. Below is our detailed response.

Q1: Where does the KIND apply, pretraining or post-training? What do you want to compare with, pretraining approaches or post-training approaches?

KIND is a novel pre-training method for constructing decomposable models. Unlike traditional pre-training approaches, KIND explicitly partitions knowledge into class-agnostic and class-specific components, encapsulating them in learngenes and tailors, respectively. This decomposition transforms fixed-size models into modular structures, where learngenes enhance transferability, and tailors adapt to specific tasks.

KIND operates during pre-training, producing learngenes and tailors for flexible downstream deployment. Accordingly, we first compare it with pre-training methods such as traditional PT and Laptop-Diff, demonstrating that training a decomposable model incurs no extra computational cost while improving structural flexibility. To highlight the strong transferability of class-agnostic knowledge, we also compare it with parameter-efficient fine-tuning methods (i.e., post-training approaches) such as LoRA and PiSSA, particularly in tasks with large domain shifts.

In summary, KIND is a pre-training framework that enhances post-training adaptability by enabling adaptive model scaling based on task demands and computational resources.

Q2: Did other post-training methods in Table 1,2,3 use the KIND pretrained DiTs or normally pretrained DiTs? Did normally pretrained DiTs use the target label information?

As noted in Q1, KIND is a pre-training framework for decomposable models that enhances post-training flexibility. Table 1 compares different pre-training strategies, including traditional PT and knowledge distillation (e.g., Laptop-Diff), in constructing models of various sizes. In contrast, Tables 2 and 3 evaluate downstream performance, comparing KIND with parameter-efficient fine-tuning methods applied to normally pretrained DiTs.

This setup ensures a fair comparison, as normally pretrained DiTs also incorporate target label information. Normally pretrained DiTs generate images based on class-conditioned information and inherently depend on class labels during pre-training. To further validate fairness, we compare the training performance of KIND-pretrained models with normally pretrained models (see Table below). The results confirm that normally pretrained models perform comparably to KIND, demonstrating that improvements in Tables 2 and 3 solely stem from the class-agnostic knowledge encapsulated in learngenes.

Q3: Fairness of experimental results.

We have addressed your concerns in detail in Q1 and Q2, which we believe sufficiently clarify this issue.

Q4: Algorithm 1 line 1 says the initialization of both $W$ and SVD matrices $U$, $\Sigma$, $V$. Did you use both matrices, or one? How do you constrain $U^\top U=I$?

We apologize for any confusion caused by the imprecise wording in Algorithm 1, line 1. As you noted, during knowledge diversion, gradient updates are applied only to $U$, $\Sigma$ and $V$, while $W$ is indirectly updated via Eq. (5) as $W=U\Sigma V^\top$.

Regarding the constraint $U^\top U=I$, we initially explored enforcing orthogonality using Cayley parameterization (details can be found in official PyTorch documentation), a transformation that maps a skew-symmetric matrix to an orthogonal matrix. Specifically, we can construct $U$ as $U=(I+Q)(I-Q)^{-1}$, where $Q$ is a skew-symmetric matrix satisfying $Q=-Q^\top$. While this guarantees orthogonality, it incurs substantial computational overhead ($\sim7\times$, see table below) due to the matrix inversion, without notable empirical benefits.

Given these trade-offs, we do not enforce explicit orthogonality constraints. Instead, we employ class gates to associate distinct feature representations with corresponding singular vectors, thereby naturally mitigating correlations among the singular vectors. This enables $U$ and $V$ to approximate orthogonality through the learning process.

As shown in Figure, the process of knowledge diversion through class gate allows the learned matrices to preserve orthogonality without requiring explicit constraints.

Q5: Other suggestions.

Thank you for your valuable suggestion. We will move Algorithm 1 to the main text and complete the missing details in Figure 2 in revision.

Dear Reviewer abHL,

We sincerely appreciate your recognition of our practicality and efficiency. Due to length constraints, experimental tables and figures, are provided via anonymous links (permitted by ICML25).

Q1:Class Gate and Sparse Gradients

Parameter updates remain sufficient without excessive sparsity (see Table). The learngene, shared across all classes, receives updates from all samples, while tailors benefit from batch training, ensuring broad class coverage (batch size=256, total classes=150).

This setup also reduces gradient conflicts across classes, allowing KIND to achieve competitive performance with similar overhead while maintaining a flexible, decomposable structure.

Q2:Task-agnostic Knowledge in Learngenes

Figure 7 shows that images generated solely from learngenes lack class-specific semantics (visually similar regardless of input labels under the same seed).

To quantify this, Table (i.e., Table 5) analyzes the classification distributions of InceptionNet across raw ImageNet images, pre-trained model outputs, learngene-generated images, and pure noise. Entropy, variance, and kurtosis are used to measure distribution uniformity, discreteness, and sharpness.

Results show that learngene-generated images exhibit low correlation with all ImageNet classes and align statistically closer to noise, confirming their class-agnostic nature and lack of semantics.

Q3:Specific Combinations of Training Class

This concern arises only with extremely limited classes, e.g., shared features like fur may dominate between cats and dogs, but introducing a distinct class, such as turtles, reduces this effect.

Tabel shows that beyond 100 classes, additional classes offer minimal benefit. As long as the class set is sufficiently diverse, its specific composition has little impact.

Q4:Nature of Transferred Knowledge

BLIP-based VQA verification in Tabel confirms that learngene-generated images are natural images (only 5.1% are classified as 'noisy images'). This suggests that learngenes encode a general noise-to-image mapping, while tailors inject class semantics.

Q5:Cross-class Validation

As shown in Figure, the 'panda' tailor fails to generate images for other classes, confirming its class-specific nature.

Q6: Theoretical Assumptions

The core assumption holds under large domain shifts. Frobenius norm in Tabel shows minimal weight perturbations ($||E^{[t]}||\ll||W^*||$) when transferring learngenes across domains.

Q7:Larger Number of Classes

A larger number of training classes is unnecessary, as 100 classes is sufficient for capturing class-agnostic knowledge (see Q3). Once the class-agnostic knowledge has encapsulated in learngenes, additional classes can be integrated by training only corresponding tailors.

Q8:Missing Details for Table 5

See Q2.

Q9:Related Works in NAS

Both KIND and NAS support variable-sized models, but only KIND extracts class-agnostic knowledge for transferability, while NAS focuses solely on network structure.

Tabel shows that NAS struggles with domain shifts.

Q10:Tailor Initialization and Model Size

In new tasks, the tailor is typically randomly initialized, which is simple and effective (see Q14).

The number of tailors and parameters varies with class count, with Table 2 reporting the average parameters across tasks.

Model size is also influenced by task complexity, and adjusting each tailor's rank balances performance and efficiency (see Tabel).

Q11:Relationship between Parameters and FLOPs

Training parameters and FLOPs are related but not strictly proportional, as FLOPs depend on both parameter count and computational patterns like matrix multiplications.

Q12:Performance Gains over FT

We further compare full-parameter fine-tuning in Tabel, confirming that KIND improves performance with lower computational cost.

Q13:Multi-class Generation

Multi-class generation can be achieved by setting the class gate to 1 for desired classes (see Figure).

Q14:Selection Criterion for Tailors

As noted in Q10, randomly initializing tailors is simple and effective. Future work may explore fine-tuning from similar-class tailors (e.g., Koala from Monkey in Tabel) or adaptively integrating multiple tailors via MoE.