Exploring Diffusion Transformer Designs via Grafting

We thank the AC and the reviewer for their valuable time and feedback. In what follows, (1) we clarify the scope/goal of this work and address (2) requests for grafting experiments beyond image generation, (3) suggestions to include additional metrics beyond GenEval for T2I evaluation, and (4) clarifications on experimental details.

$~$

W1: Since obtaining a model with better performance is one of the most important goals in architecture design, this method lacks such a variant to illustrate the superiority of the method in this aspect. So far, it seems only be helpful in inference acceleration at a cost of performance.

We thank the reviewer for the thoughtful question. We agree that model quality is one of the important goals in architecture design. However, the goal of this work is not to outperform the base model, but to investigate whether diffusion transformer architectures can be edited post-training under small compute budgets, and to characterize the resulting quality–efficiency tradeoffs. Prior to this work, it was unknown whether such post-training architecture editing could preserve generation quality at all. Our results provide the first systematic study of post-training architectural edits on diffusion transformers (Tables 4, 5, and 6), mapping the quality–efficiency frontier in this setting.

While some quality drop is expected when replacing expensive operators with cheaper alternatives, we quantify how much can be traded off, and demonstrate that architectural editing can preserve or closely match baseline generation quality.

To summarize, the key outcomes of our work:

• Table 4 & Fig. 3 (Right): We obtain post-training hybrid architectures that are within 0.5 FID of the DiT-XL/2 baseline, while grafting 50% of MHA operators (Hyena-SE/X/Y, SWA) and 100% of MLP operators (MLP r=3) with cheaper alternatives.

• Table 5: On PixArt-Sigma, we obtain a 1.43× faster hybrid variant via operator replacement, with less than 2% GenEval score drop at 2048×2048 resolution.

• Table 6: We restructure DiT-XL/2 by trading depth for width, producing a 2× shallower model (2× faster with fused implementation, bs=1) that achieves better FID (2.77) and outperforms pruning baselines and shallow-from-scratch trained models at depth=14. To our knowledge, this is the first attempt to convert sequential computation into parallel in DiTs post-training, enabling computation to be restructured.

$~$

W2: Generalization beyond image generation

We note that Generalization was a recurring theme raised by two other reviewers.

• Reviewer r7Mh suggested experiments beyond “diffusion-based image generation” and “architectures beyond DiT”.
• Reviewer VANM suggested “more open-source models with different architectures” with video generation as one possible experiment.

To address these concerns collectively, we evaluate grafting in the language modeling setting—a task, architecture, and modality distinct from diffusion-based image generation with DiTs.

Setup:

• Task: Diffusion-based image generation → Autoregressive language modeling (next-token prediction)
• Architecture & Scale: DiT-XL/2 (0.7B) → Qwen3 (4B)
• Operator: MHA → Sliding Window Attention (window=256)
• Dataset: Alpaca-cleaned (50k instruction-following examples)
• Evaluation: PiQA, ARC-e, ARC-c, HellaSwag, Winogrande, MMLU following [F]
• Experiment details: We apply our exact two-stage grafting procedure—activation distillation followed by lightweight fine-tuning. Stage 1 uses context length 1024; Stage 2 uses 8192.

Results:

Our grafted Qwen3-4B model achieves a 1.4x decode throughput (at 8K context length with KV cache, Nvidia H100) with less than 1% average performance drop, demonstrating that grafting generalizes to new tasks, architectures, and modalities. We leave video generation to future work.

$~$

W3: For text-to-image generation, only using GenEval as the metric may not be sufficient.

We thank the reviewer for this suggestion. While the reviewer did not specify a preferred metric, we note that GenEval is a widely adopted objective metric for evaluating prompt–image alignment in text-to-image generation [A,B]. Other common metrics such as IS and FID are less appropriate for high-resolution text-to-image tasks, as they rely on low-resolution ImageNet-1K features. To follow best practices, our paper already includes GenEval scores and qualitative samples (see Supplementary). To further strengthen the evaluation, we now include a small-scale human study, following recent work [C, D]. We randomly sample 350 text–image pairs and ask three expert annotators to rate each image along two axes:

• Prompt–image alignment: Does the image accurately reflect the text?
• Realism: Does the image appear visually realistic?

These results show that the grafted variant (Hyena-X) maintains good quality across both axes. Together, we provide both objective (GenEval) and human (alignment/realism) evaluations of generation quality.

$~$

Q1: Why MHA and MLP prefer different regression loss?

This difference arises from benign overfitting. MLPs have 2× more parameters than MHA (10.6M vs. 5.3M), which makes them robust to outliers [E]. This behavior is also reflected in our validation loss curves (Supplementary D.3). For MHA, L1 regression consistently yields lower validation loss than L2, while the opposite holds for MLPs. We therefore adopt L1 for MHA and L2 for MLP. We will make this clear in the final version.

$~$

Q2: Actually, I may not agree with the author's opinion: 'We trade depth for width via grafting, achieving better quality than models of comparable depth.' Though the depth may be similar, e.g., DiT-B, TinyDiT-D14, and Grafting, but the number of parameters is not. Is there a possibility that the better performance of Grafting is attributed to more model parameters?

The reviewer is correct that the depth -> width restructured model has 6% more parameters (712M vs. 675M for DiT-XL/2) due to linear projections. However, the goal of this case study is not to reduce parameter count, but to investigate whether deep, sequential transformer blocks can be restructured post-training into shallower, wider configurations that improve inference time while preserving quality.

In our restructuring, every two sequential DiT blocks are rewired post-training to run in parallel—reducing the model’s depth from 28 to 14 and achieving a 2× speedup (fused implementation, bs=1). Despite the slightly higher parameter count, the model achieves better quality (FID = 2.77) while benefiting from increased parallelism—demonstrating a favorable depth–width tradeoff. This experiment is not focused on parameter efficiency, but rather on post-training architectural restructuring—demonstrating, to our knowledge, the first instance of block-level rewiring of pretrained DiTs. For completeness, we include Table 6 below with speedup and FID metrics.

We hope our response addresses the reviewer's questions, and we’re happy to provide further details if helpful.

====

[A] Yao, Yuan, et al. "Diffusion Transformer-to-Mamba Distillation for High-Resolution Image Generation." arXiv (2025)

[B] Tang, Bingda, et al. "Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis." CVPR 2025

[C] Cai, Shengqu, et al. "Diffusion self-distillation for zero-shot customized image generation." CVPR. 2025.

[D] Chen, Junsong, et al. "Pixart-Sigma" ECCV, 2024.

[E] Bartlett, Peter L., et al. "Benign overfitting in linear regression." PNAS 2020.

[F] Zhang, Michael, et al. "LoLCATs: On Low-Rank Linearizing of Large Language Models." ICLR 2025

We thank the AC and the reviewer for their helpful feedback. The reviewer appreciated our operator replacement and depth to width restructuring studies. The reviewer’s questions focus on: (1) whether lightweight fine-tuning remains modest in compute for deep edits and how this compares to small from-scratch retrains; (2) role of distillation and the feasibility of pretraining hybrids; (3) experimental details (replacement strategy, hyperparameters); and (4) presentation suggestions. Below, we address each point in detail.

$~$

W1: While finetuning is lightweight, recovery of good performance may still require significant data/compute for certain tasks or very deep editing. Whether it always outperforms a fresh, small retrain is not clear for all cases.

We agree that very deep edits can require more updates. Empirically, our depth→width restructuring reached FID = 2.77 after 230k iterations (new result obtained during rebuttal), which is still tiny compared to the ~7M iterations used to train the base model—i.e., even for deep edits we expect compute to remain at least an order of magnitude lower than pretraining. On “fresh small retrain”: the reviewer is correct to raise this. We note that pretraining is out of scope in our work as we focus on post-training architectural editing to study the resulting quality–efficiency trade-offs. Studying the relationship to from-scratch training is left to future work due to compute demands. Even pretraining a 64×64 ImageNet-1K DiT takes >12 days on 32×A100s [A] which is not feasible in academic compute budgets. We have noted this in our limitation section (Supp Sec. I).

$~$

W2 (1): Grafting is proposed as a method for exploring new architectures with limited compute. However, the authors have not trained their final proposed model from scratch for comparison against the grafted counterpart. As a result, it is unclear how much of the performance gain can be attributed to the distillation process itself.

The reviewer is correct—distillation is important for grafting, and isolating its contribution would require training the hybrid architectures from scratch for a head-to-head comparison. We note that pretraining is beyond the scope of this work, as we focus on post-training architectural editing of diffusion transformers. While an interesting direction, even training a 64x64 DiT takes more than 12 days on 32×A100s [A], making pretraining infeasible on an academic budget. That said, we agree this is an intellectually important question. To address it, we highlight independent evidence from language modeling showing that hybrid architectures mixing attention with Hyena-style operators (including Hyena-SE) can be trained from scratch and remain competitive.

Evidence: StripedHyena2: 7B-scale language models trained with Hyena-SE [B]: StripedHyena2 [B] trains large-scale hybrid models from scratch using setups which mix attention and Hyena operators, including Hyena-SE. At the 7B scale, these models achieve lower perplexity than transformer baselines (2.83 vs. 3.09) after 400B tokens of pretraining—demonstrating that hybrid architectures remain expressive at scale.

$~$

W2 (2) Additionally, unless I have misunderstood (and please feel free to clarify), it appears that the authors have not yet identified an architecture that outperforms their base model.

We thank the reviewer for the thoughtful question. For clarity, the goal of this work is not to outperform the base model but to investigate whether pretrained DiT architectures can be edited post-training under small compute budgets and to characterize the resulting quality–efficiency trade-offs.

Prior to this work, it was unknown whether such post-training architecture editing could preserve generation quality at all. Our results provide the first systematic study of post-training architectural edits on diffusion transformers (Tables 4, 5, and 6), mapping the quality–efficiency frontier in this setting.

To summarize, the key outcomes of our work:

• Table 4 & Fig. 3 (Right): We obtain post-training hybrid architectures that are within 0.5 FID of the DiT-XL/2 baseline, while grafting 50% of MHA operators (Hyena-SE/X/Y, SWA) and 100% of MLP operators (MLP r=3) with cheaper alternatives (<2% pretraining compute).
• Table 5: On PixArt-Sigma, we obtain a 1.43× faster hybrid variant via operator replacement, with less than 2% GenEval score drop at 2048×2048 resolution.
• Table 6: We restructure DiT-XL/2 by trading depth for width, producing a 2× shallower model (2× faster with fused implementation, bs=1) that achieves better FID (2.77) and outperforms pruning baselines and shallow-from-scratch trained models at depth=14.

Future work will explore pretraining and scaling of hybrid DiT architectures to improve model quality.

$~$

Q1: In scenarios where multiple types of operators are to be replaced, is there a recommended strategy? Should they be replaced all at once, or sequentially? Additionally, does the effectiveness of Stage 1 distillation degrade when replacing several components?

Replace at once is recommended; we observed only minor improvements over sequential replacements (we tried this—FID improvements were < 0.07 for full MHA self-grafting). Reviewer is correct, replacing more components can propagate error because stage 1 regression loss is not zero for every operator, but Stage-2 (lightweight fine-tuning) handles this. For example, under full self-grafting of MHA and MLP we achieve near-baseline performance (Table 2) after stage 2.

$~$

Q2: Are the hyperparameters from the original (base) model typically retained, or is it generally necessary to re-tune them after grafting?

Yes, base hyperparameters are reused. The only changes are (1) a simple linear warmup to the original base learning rate during Stage 2 (e.g., warming up to 1e‑4), and (2) a weight decay of 1e‑5 in stage 2. After warmup, the learning rate remains constant, following the original DiT recipe. Full hyperparameters for both Stage 1 and Stage 2 are provided in Supplement B.1.

$~$

Suggestion: For Table 4, I suggest that the authors consider adding the original FID values to plot (b), perhaps using dotted lines for visual reference. This would help contextualise the deltas and make the plot more informative.

Thank you for the suggestion. We will include this in the final version.

We hope our response addresses the reviewer's questions, and we’re happy to provide further details if helpful.

====

[A] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." NeurIPS 2022

[B] Ku, Jerome, et al. "Systems and algorithms for convolutional multi-hybrid language models at scale." arXiv preprint arXiv:2503.01868 (2025).

We thank the AC and reviewer for their valuable time and feedback. The reviewer raised several concerns: (1) generation quality gap and the practical relevance of grafting, (2) lack of comparison to caching methods, (3) limited generalization beyond DiT architectures and image generation, and (4) suggestions regarding latency reporting and improved results in Table 6. Below, we address all points in detail.

$~$

W1: Generation quality gap and practical relevance

The reviewer is correct, we agree that generation quality is essential for practical utility. However, the goal of this work is not to outperform the base model, but to investigate whether pretrained diffusion transformer architectures can be edited post-training under small compute budgets, and to characterize the resulting quality–efficiency tradeoffs. Prior to this work, it was unclear whether such post-training architecture editing could retain generation quality at all. Our results show that they can: across all experiments, the grafted architectures demonstrate good generation quality, despite significant architectural editing. For example:

• Table 4 & Fig. 3 (Right): We obtain hybrid architectures that are within 0.5 FID of the DiT-XL/2 baseline, while grafting 50% of MHA layers (Hyena-SE/X/Y, SWA) and 100% of MLPs (MLP r=3).

• Table 5: On PixArt-Σ, we obtain a 1.43× faster variant at 2048×2048 resolution with <2% GenEval score drop.

• Table 6: We restructure DiT-XL/2 by trading depth for width, producing a 2× shallower model (2× faster with fused implementation, bs=1) that achieves better FID (2.77) and outperforms pruning baselines and shallow-from-scratch trained models at depth=14. While some quality drop is expected when replacing expensive operators with cheaper ones, our study quantifies these tradeoffs and demonstrates that high-quality generation is achievable when editing DiT architectures post-training.

$~$

W2: Comparison to caching methods

Thank you for raising this point. Caching methods like ToCa, TeaCache, PAB, and TaylorSeer accelerate inference by reusing intermediate activations at runtime. These techniques alter execution but leave the model’s architecture unchanged. In contrast, grafting edits the model’s architecture itself—replacing expensive operators/restructuring the computational graph post-training. The two approaches are orthogonal and complementary. We will include a discussion of these works in the final version. Our submission also compares grafting with SOTA pruning methods in Section 6 (Table 6).

$~$

W3: Generalization beyond image generation

We note that Generalization was a recurring theme raised by two other reviewers.

• Reviewer r7Mh suggested experiments beyond “diffusion-based image generation” and “architectures beyond DiT”.

• Reviewer JnND mentioned “experiments only in image generation may be insufficient” with video generation as one possible experiment.

To address these concerns collectively, we evaluate grafting in the language modeling setting—a task, architecture, and modality distinct from diffusion-based image generation with DiTs.

Setup:

• Task: Diffusion-based image generation → Autoregressive language modeling (next-token prediction)
• Architecture & Scale: DiT-XL/2 (0.7B) → Qwen3 (4B)
• Operator: MHA → Sliding Window Attention (window=256)
• Dataset: Alpaca-cleaned (50k instruction-following examples)
• Evaluation: PiQA, ARC-e, ARC-c, HellaSwag, Winogrande, MMLU following [A]
• Experiment details: We apply our exact two-stage grafting procedure—activation distillation followed by lightweight fine-tuning. Stage 1 uses context length 1024; Stage 2 uses 8192.

Results:

Our grafted Qwen3-4B model achieves a 1.4x decode throughput (at 8K context length with KV cache, Nvidia H100) with less than 1% average performance drop, demonstrating that grafting generalizes to new tasks, architectures, and modalities.

$~$

Q1: Table 6 latency and improved FID results.

We thank the reviewer for this suggestion. In the final version, we will add latency/ speedup relative to the baseline. As per reviewer’s suggestion, we have trained an improved variant that achieves a better FID (2.77) using 230k iterations. Below we include the updated table for your reference (Speedup measured using a fused implementation, bs=1).

We hope our response addresses the reviewer's questions, and we’re happy to provide further details if helpful. Thank you for the thoughtful feedback.

===

[A] Zhang, Michael, et al. "LoLCATs: On Low-Rank Linearizing of Large Language Models." ICLR 2025

Dear Reviewer VANM,

Thank you for your follow-up.

To address your question on how grafting affects efficiency in autoregressive LLMs, we report decode throughput (tokens/s) for the Qwen3-4B baseline and our grafted variant with 50% of MHA operators replaced by SWA (window=256). We observe:

• 1.4× increase in decode throughput over the baseline
• <1% drop in average performance (68.6 vs. 69.0)

Evaluated on a single NVIDIA H100 (KV cache, 8K context length, batch size ≈ 32).

These results show that grafting improves generation latency while preserving quality.

We hope this addresses your concern.

Looking forward to your reply.

Thanks!

We thank the AC and the reviewer for their constructive feedback. The reviewer highlighted the simplicity and efficiency of our approach and our experiments. The reviewer’s questions center on: (1) generalization of grafting beyond diffusion-based image generation using DiTs, (2) how operators interact with surrounding layers, (3) pretraining hybrid architectures, (4) potential artifacts from synthetic data, and (5) clarifications on experimental results. We respond in detail below:

$~$

W1+Q3: Generalization beyond diffusion-based image generation using the DiT architecture.

Thank you for your feedback. To address reviewer concerns, we evaluate grafting in an autoregressive language modeling setting—a task, architecture, and modality distinct from diffusion-based image generation with DiTs.

Setup:

• Task: Diffusion-based image generation → Autoregressive language modeling (next-token prediction)
• Architecture & Scale: DiT-XL/2 (0.7B) → Qwen3 (4B)
• Operator: MHA → Sliding Window Attention (window=256)
• Dataset: Alpaca-cleaned (50k instruction-following examples)
• Evaluation: PiQA, ARC-e, ARC-c, HellaSwag, Winogrande, MMLU following [E]
• Experiment details: We apply our exact two-stage grafting procedure—activation distillation followed by lightweight fine-tuning. Stage 1 uses context length 1024; Stage 2 uses 8192.

Results:

Our grafted Qwen3-4B model achieves a 1.4x decode throughput (at 8K context length with KV cache, Nvidia H100) with less than 1% average performance drop, demonstrating that grafting generalizes to new tasks, architectures, and modalities.

$~$

W2 (1) However, it's unclear whether different operator might rely on their surrounding operators—since grafting requires freezing the context parameters.

The reviewer is correct: operator initialization relies on the surrounding operators. We freeze surrounding parameters only in Stage 1 (activation distillation). In Stage 2 (lightweight fine-tuning) we update all parameters end-to-end (see Fig. 1, Tables 4 and 6). For PixArt-Σ, we perform LoRA finetuning for memory efficiency (long sequence length = 16,384). We apply LoRA to self-attention (q/k/v/o), cross-attention (q, kv), and MLP (fc1/fc2), and update 1D causal convolutions in Hyena-X directly.

$~$

W2 (2) + Q4: It also remains uncertain whether architectures that perform well under the grafting setting would yield similarly results when trained from scratch.

Thank you for the thoughtful question. We note that pretraining is beyond the scope of this work, as we focus on post-training architectural editing of diffusion transformers. While an interesting direction, even training a 64x64 DiT takes more than 12 days on 32×A100s [A], making pretraining infeasible on an academic budget. That said, we agree this is an intellectually interesting question. To address it, we highlight independent evidence from language modeling showing that hybrid architectures mixing attention with Hyena-style operators (including Hyena-SE) can be trained from scratch and remain competitive.

Evidence: StripedHyena2: 7B-scale language models trained with Hyena-SE [B]: StripedHyena2 [B] trains large-scale hybrid models from scratch using setups which mix attention and Hyena operators, including Hyena-SE. At the 7B scale, these models achieve lower perplexity than transformer baselines (2.83 vs. 3.09) after 400B tokens of pretraining—demonstrating that hybrid architectures are competitive at scale.

$~$

Q1: In Table 4(c), the interleaved strategy performs best, even outperforming the top-local strategy. Is there any explanation for this result? Does it suggest that the effectiveness of grafting is sensitive to the contextual operators surrounding the integrated operator?

Thank you for the thoughtful question. We believe the strong performance of the interleaved strategy stems from its ability to balance local and global receptive fields. In contrast, stacking multiple local operators (top-local) may restrict the model’s effective receptive field. Alternating with global operators helps retain broader context. Similar interleaving strategies have recently been shown effective in the pretraining of language models [B] and biological foundation models [C].

$~$

Q2: The paper uses synthetic data for two-stage training and observes some localized artifacts. Would such artifacts also appear when training on real data? Are there methods to solve this problem?

Thank you for the question. We do not expect such artifacts to appear when using real data. In Supplementary Fig. D.4 and D.4, we show that some low-quality synthetic samples can introduce artifacts during grafting. A practical way to mitigate this is to curate or filter synthetic data before use. Recent works [D] have explored automated data curation pipelines using VLMs for improving data quality.

We hope our response addresses the reviewer's questions, and we’re happy to provide further details if helpful. Thank you for the thoughtful feedback.

[A] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." NeurIPS 2022

[B] Ku, Jerome, et al. "Systems and algorithms for convolutional multi-hybrid language models at scale." arXiv preprint arXiv:2503.01868 (2025).

[C] Brixi, Garyk, et al. ‘Genome Modeling and Design across All Domains of Life with Evo 2’. bioRxiv, Cold Spring Harbor Laboratory, 2025.

[D] Cai, Shengqu, et al. "Diffusion self-distillation for zero-shot customized image generation." CVPR 2025.

[E] Zhang, Michael, et al. "LoLCATs: On Low-Rank Linearizing of Large Language Models." ICLR 2025

We thank the AC and the reviewer for their valuable time and feedback. The reviewer appreciated our activation behavior and locality analysis, testbed and experiments. The reviewer raised concerns about training hybrid architectures from scratch, general questions about the experiment setup, and applications of grafting. Below, we address all comments.

$~$

W1: Training from scratch with operators from grafting.

Thank you for the thoughtful question. As noted by the reviewer, pretraining is beyond the scope of this work, as we focus on post-training architectural editing of diffusion transformers. While an interesting direction, even training a 64x64 DiT takes more than 12 days on 32×A100s [A], making pretraining infeasible on an academic budget. That said, we agree this is an intellectually interesting question. To address it, we highlight independent evidence from language modeling showing that hybrid architectures mixing attention with Hyena-style operators can be trained from scratch and remain competitive.

Evidence: StripedHyena2: 7B-scale language models trained with Hyena-SE [B]: StripedHyena2 [B] trains large-scale hybrid models from scratch using setups which mix attention and Hyena operators, including Hyena-SE. At the 7B scale, these models achieve lower perplexity than transformer baselines (2.83 vs. 3.09) after 400B tokens of pretraining—demonstrating that hybrid architectures remain expressive at scale.

$~$

Q1: Why was LoRA used for training PixArt-? What is exactly being done here? Why do you need LoRA when you are finetuning all the replaced operators in the 2nd stage?

We used LoRA in our high-resolution PixArt-Σ experiments to reduce memory and compute overhead. As noted in lines 254–255, these experiments operate over long sequences (e.g., shape [B,16384,1152]), which makes standard finetuning expensive. In Stage 2 (lightweight finetuning), we apply LoRA to self-attention (q/k/v/o), cross-attention (q, kv), and MLP (fc1/fc2), and we update the 1D causal convolutions in Hyena-X directly (added to the optimizer; no LoRA, since these are 1D convs with a small parameter count). We will clarify this in the final version.

$~$

Q2 (1) Difference between Hyena-SE, Hyena-X and Hyena-Y operators

We summarize the differences below. The key distinctions lie in the use of convolutional featurizers and the presence or absence of the inner filter. As suggested by the reviewer, we will add Hyena-SE to Fig. 3 in the final version. We have also provided the operator equations in Supp. E.

$~$

Q2 (2) Question reg. causality

While our testbed (Table 3) already includes bidirectional operators (e.g., self-grafting, sliding window attention), we introduced causal Hyena operators to study causal inductive biases. Notably, causal variants perform competitively—Hyena-Y achieves a FID of 2.61, compared to 2.62 for Sliding Window Attention (SWA). If the reviewer prefers, we can include a bidirectional version of Hyena-X or Hyena-Y in the final version. This requires only a simple change: removing left-padding in the inner convolution and instead padding the input on the right.

$~$

Q3 (2) Memory footprint of hybrid architectures

The reviewer is correct in noting that efficient operators can reduce memory usage of the full model in addition to speed, depending on their hardware-aware implementation. We provide a reference: At a sequence length of 2048, peak memory is ~2.70 GB for a GQA Transformer++ baseline, versus ~2.55 GB for the corresponding Hyena-Y hybrid architecture (Δ −150 MB, ≈ −5.6%).

$~$

Q3 (1) Edge applications

The reviewer is correct—while this work studies grafting as a general-purpose architectural editing approach, replacing expensive components with cheaper alternatives naturally leads to faster inference/ lower memory usage, which are particularly valuable in edge scenarios.

To further demonstrate broader applicability, we also applied grafting to large autoregressive language models. These experiments were conducted to address grafting’s generalization to other domains (different task/architectures) suggested by Reviewers r7Mh, VANM and JnND. We believe the results may be of interest here, especially in the context of real-time or edge deployment. By grafting a 36-layer Qwen3 4 billion parameter language model with Sliding Window Attention operators (w=256, grafting ratio=50%), we obtain 1.4x decode throughput (at 8K context length, Nvidia H100 with KV cache) with less than 1% overall performance drop.

We hope our response addresses the reviewer's questions, and we’re happy to provide further details if helpful. Thank you for the thoughtful feedback.

===

[A] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." NeurIPS 2022

[B] Ku, Jerome, et al. "Systems and algorithms for convolutional multi-hybrid language models at scale." arXiv preprint arXiv:2503.01868 (2025).