REPA Works Until It Doesn’t: Early-Stopped, Holistic Alignment Supercharges Diffusion Training

Thanks for your comments. We discuss the questions as follows.

Q1: Missing explanations in several figures.

Thank you for your comment. We acknowledge that the illustrations of Figures 5,6 can be clearer. Therefore, we provide detailed explanations of them.

• Figure 5: The time interval [0, 0.25] stands for the diffusion timesteps where the input to pre-trained encoders is replaced with low-frequecy part of the image. For image filtering, we convert the image to frequency domain using the Fourier transform, and filter out the high-frequency components, then invertly convert it back to an image.
• Figure 6: We compute the feature and attention alignment progress mainly to explain the complementary effect of both methods. We will add baselines to (b) and (c) in the latest version.

Q2: Will allocating more training computation to larger timesteps with higher noise levels serve as a better baseline?

Thanks for your insightful question. We provide comparision of REPA, HASTE, and log-normal timestep sampling on SiT-B/2.

While log-normal has been proved effective on models like EDM, we find acceleration methods based on external representations still achieve better performance.

Q3: The use of gradient-angle trigger.

Thanks for your important question. In the following, we first explain the role of gradient signals in our method, and then provide a plain example of automatic gradient-angle trigger.

• Dual termination strategy clarification: Our method employs a hybrid approach where fixed iterations serve as the primary and practical termination strategy, while the gradient-angle rule provides theoretical insight and robustness across different datasets. Specifically:
  • In practice, we find setting a fixed termination iteration $\tau$ for HASTE actually performs reliably well in ImageNet and MS-COCO experiments. Gradient-angle threshold serves as a supplemental alternative, adding adaptability to different scenerios.
  • The gradient analysis helps us to locate the termination point. As we state in Lines 283-287, we choose to stop around 250K iteration when we observe limited alignment benefits and relatively low gradient cosine similarities.
• Plain implementation of automatic gradient-angle trigger: We train SiT-B/2 on ImageNet 256 $\times$ 256 following original settings of HASTE using a gradient-angle trigger.

  • Implementation:

    • We start evaluating the gradient cosine similarity between holistic alignment and denoising objectives every 10K iterations from 50K iteration, when the training process has been relatively stable.
    • In each round of evaluation, we compute the similarity separately on selected diffusion timesteps $t$ at different noise levels with 2048 images randomly sampled from ImageNet, and take the average value. When the average similarity falls below a threshold $\delta$, we terminate the holistic alignment.
    • In the plain example, we perform gradient evaluation on the 5th SiT block (counting from 1). We select $t=0.02$, $t=0.1$, $t=0.5$ and $t=0.8$, and set $\delta=-0.05$.

  • Results:

    • Gradient cosine similarity:

    timestep\iteration	50K	60K	70K	80K
    0.02	-0.0174	0.0418	0.0872	-0.0032
    0.1	0.0126	-0.0767	-0.0148	-0.1155
    0.5	0.0134	0.0106	-0.0848	-0.1272
    0.8	0.0034	0.0406	0.1025	-0.0212
    average	0.0030	0.0041	0.0225	-0.0668(terminate)

    • Performance:

    method	iteration	termination	FID$\downarrow$	sFID$\downarrow$	IS$\uparrow$
    SiT	100K	-	63.46	7.31	20.6
    SiT+HASTE	100K	-	39.86	7.16	35.8
    SiT+HASTE	100K	80K	38.69	6.88	36.4

  • Findings: Our automatic trigger terminates the holistic alignment at 80K iteration, and achieves better performance at 100K iteration.

  • Limitations: Though we validate the automatic gradient-angle trigger on SiT-B/2, we find it challenging to design such trigger for precise termination, because of:

    • Additional overhead: Gradient evaluation introduces extra computational cost, limiting the practical evaluation frequency.
    • Hyperparameter setting: The automatic trigger is sensitive to hyperparameters, such as block choice, timestep selection, and the threshold.

  • Conclusion: Given that monitoring model performance during training is a standard practice in modern deep learning workflows, we think it's more practical to integrate the periodic evaluation of both metrics and gradient cosine similarity to determine the optimal termination point.

Q4: Inconsistent performance of SiT-XL/2 at 400K and 500K iteration.

Thanks for your question. As we mentioned in Line 277-281, we attribute the inconsistency to over-regularization introduced by continuously using holistic alignment. Actually, our termination strategy at 250K iteration helps to alleviate such a trend and demonstrates better performance at 500K iteration.

Q5: The use of external pre-trained models introduces extra training cost.

Thank you for the question. We evaluate the extra training overhead of HASTE on SiT-XL/2 introduced by external vision model DINOv2-B:

• Extra memory overhead: The peak CUDA memory with local batchsize 32:

  HASTE	REPA
  30666.84 MB	27818.01 MB

• DINOv2-B forward overhead: During training, we compute DINOv2‑B representations online following REPA. The forward pass of DINOv2‑B incurs approximately 23.19 GFLOPs for the base‑size model at 224×224 input resolution without XFormers.

We will add these evaluations in our latest version.

Formatting Concern: "The paper would benefit from a figure that illustrates the REPA and ATTA regularization mechanisms."

Thanks for your advice. We will add the illustrations in the latest version.

Thanks for the insightful and motivational comments from reviewer qGU6. We discuss the questions as follows.

Q1: Validating gradient-angle-based automatic trigger.

A1: Thanks for the question. To address this concern, we provide a plain implementation as an example for an automatic gradient-angle trigger as follows.

• Setting: We train SiT-B/2 on ImageNet 256 $\times$ 256 following the original settings of HASTE using a gradient-angle trigger.

• Implementation:

  • We start evaluating the gradient cosine similarity between holistic alignment and denoising objectives every 10K iterations from 50K iterations, when the training process has been relatively stable.
  • In each round of evaluation, we compute the similarity separately on selected diffusion timesteps $t$ at different noise levels with 2048 images randomly sampled from ImageNet, and take the average value. When the average similarity falls below a threshold $\delta$, we terminate the holistic alignment.
  • In the plain example, we perform gradient evaluation on the 5th SiT block (counting from 1). We select $t=0.02$, $t=0.1$, $t=0.5$ and $t=0.8$, and set $\delta=-0.05$.

• Results:

  • Gradient cosine similarity:

    timestep\iteration	50K	60K	70K	80K
    0.02	-0.0174	0.0418	0.0872	-0.0032
    0.1	0.0126	-0.0767	-0.0148	-0.1155
    0.5	0.0134	0.0106	-0.0848	-0.1272
    0.8	0.0034	0.0406	0.1025	-0.0212
    average	0.0030	0.0041	0.0225	-0.0668(terminate)

  • Performance:

    method	iteration	termination	FID$\downarrow$	sFID$\downarrow$	IS$\uparrow$
    SiT	100K	-	63.46	7.31	20.6
    SiT+HASTE	100K	-	39.86	7.16	35.8
    SiT+HASTE	100K	80K	38.69	6.88	36.4

• Findings: Our automatic trigger terminates the holistic alignment at 80K iterations, and achieves better performance at 100K iterations.

• Limitations: Though we validate the automatic gradient-angle trigger on SiT-B/2, we find it challenging to design such a trigger for precise termination, because of:

  • Additional overhead: Gradient evaluation introduces extra computational cost, limiting the practical evaluation frequency.
  • Hyperparameter setting: The automatic trigger is sensitive to hyperparameters, such as block choice, timestep selection, and the threshold.

• Conclusion: Given that monitoring model performance during training is a standard practice in modern deep learning workflows, we think it's more practical to integrate the periodic evaluation of both metrics and gradient cosine similarity to determine the optimal termination point.

Q2: Validation of termination strategy in HASTE on MS-COCO.

A2: To address this concern, we conduct the following text-to-image (T2I) generation experiment on MS-COCO.

• Setting: Following the original settings of HASTE and REPA on MS-COCO, we continue to train MMDiT with both methods from 150K to 250K iterations, and compare the performance of MMDiT+HASTE with and without termination $\tau$=200K. As training progresses, we raise the classifier-free guidance (cfg) [1] scale to 3.5 for all evaluations at 250K iterations to steer the sampling process within the data manifold. We consistently use the SDE Euler-Maruyama sampler with NFEs=250.
• Results:

  method	iteration	termination	FID$\downarrow$
  MMDiT+REPA	250K	-	4.28
  MMDiT+HASTE	250K	-	4.10
  MMDiT+HASTE	250K	200K	4.06

• Conclusion: As expected, while our holistic alignment outperforms REPA for MMDiT, the termination strategy also leads to better performance in the T2I task.

Q3: Guidelines for τ selection.

A3: Thanks for the question. We first explain our selection strategy in practice, and then give guidelines for τ selection.

• "Exhaustive" tuning is not necessary:
  • Gradient analysis helps: In practice, we don't really exhaustively tune $\tau$, because the gradient analysis helps us to locate the termination point. As we state in Lines 283-287, we choose to stop around 250K iterations when we observe limited performance improvement and relatively low gradient cosine similarities.
  • Tolerance of $\tau:$ Additionally, we find HASTE is relatively robust to $\tau$. In Figure 7b, we show that varying $\tau$ from 250K to 290K doesn't lead to a significant difference in the FID of SiT-XL/2 at 400K iteration.
• Guidelines for τ selection: As we have mentioned above, the periodic evaluation of metrics and gradient cosine similarity helps to locate the approximate termination period. Additionally, termination within such a period works reliably well for HASTE and is more practical than locating the precise point.

Q4: Pseudocode for HASTE training loop.

Thanks for the advice. We will add it to our latest version.

Q5: Using ATTA alone in data-scarce or computationally constrained scenarios.

A5: Thank you for the insightful advice. In the following, we provide experiments on MS-COCO to explore the performance of ATTA, HASTE, and REPA in the given scenario.

• Setting: To simulate data-scarce or computationally constrained conditions, we sample 20K images ($<\frac{1}{4}$ original size) from MS-COCO training dataset, and set the batchsize of 64 ($\frac{1}{4}$ default batchsize), and train MMDiT for only 50K iterations. We do not apply termination for HASTE.
• Results:

  method	iteration	FID
  MMDiT	50K	49.03
  MMDiT+REPA	50K	32.59
  MMDiT+ATTA	50K	43.17
  MMDiT+HASTE	50K	31.39

• Findings: Under the specific setting, all three methods provide considerable acceleration to the training process. HASTE works best and REPA performs better than ATTA.
• Analysis: As we mentioned in Lines 167-169, ATTA nails routing but lacks supervision over condition processing, which is even more evident in MMDiT architecture: we only perform attention alignment with the $QK^T$ matrix generated from the input image to avoid affecting textual processing. In contrast, REPA performs alignment over the hidden states of MMDiT, acting on both vision and textual processing. Therefore, combining feature and attention alignment provides more holistic guidance for training.
• Conclusion: In data-scarce or computationally constrained scenarios, ATTA can accelerate the training process, but not as effectively as REPA and HASTE.

Q6: Discussion of risks from faster generative model training.

A6: Thanks for the concern. We acknowledge that faster generative model training could potentially lower barriers for harmful content generation and accelerate deepfake production.

While we recognize these concerns, we believe HASTE's risk profile is relatively contained for several reasons:

• Research-focused contribution: Our method primarily benefits legitimate research by reducing computational barriers, enabling smaller labs and researchers to participate in generative modeling research.
• Safeguards applicable: Standard safety measures, such as content filtering, ethical guidelines, and responsible disclosure, remain compatible with HASTE-trained models.

Limitations and future work: As a technical contribution focused on training efficiency, our work doesn't directly address broader AI safety challenges, which we acknowledge as important future work. We are going to include safety concerns in Section 5 in the revision.

Thanks reviewer fiaM for adoring our work. We are delighted to discuss the questions below.

W1&Q1: Relationship between "capacity mismatch" and "gradient divergence". Thanks for your insightful question. We first provide a further detailed clarification of "capacity mismatch", and then explain how it connects with "gradient divergence".

• Capacity Mismatch: We refer "capacity mismatch" to two fundamental differences between vision representation models (e.g., DINOv2) and diffusion models.
  • Task mismatch: Diffusion models are designed to generate high-quality samples through denoising, while representation models are aimed at capturing semantic information and constructing good representations. Therefore, the two kinds of models possess fundamentally different objectives.
  • Distribution mismatch: Diffusion models learn complex, noisy distributions across different noise levels. In contrast, representation models learn only through the clean marginal data distribution. In the training process, external representations help diffusion models to get a coarse and fast grasp of the generative task but fail to further improve the fine-grained generation capacity, especially at diffusion timesteps with lower noise levels.
• Relationship:
  • Statement concern: While we acknowledge that gradient interference is a more direct description, we would like to clarify that capacity mismatch serves as the cause and provides deeper insight, which helps to explain the source of training constraint. We also follow that insight to design the low-frequency filter experiment in Figure 5, Section 2.2.
  • Experimental evidence: In Section 2.2, we leverage gradient analysis for diagnosis and observe the Ignition-Plateau-Conflict trends, which are consistent with the shift of REPA from accelerator to brake. Additionally, our low-frequency filter experiment further verifies the existence of capacity mismatch.
  • Mismatch and divergence: When diffusion models seek high-frequency detail to further improve the generation quality, the capacity mismatch leads to the constraint of improvement, which is obvious at diffusion timesteps at lower noise levels (e.g., 0-0.1). At these timesteps, we observe evident gradient divergence.

W2: Mismatch between gradient direction figures and cosine similarity plots.

Thanks for your careful observation. We understand the concern about potential inconsistency between the main text figures and the appendix data. Please allow us to clarify this important point.

• Timestep-dependent gradient evolution:

  As reported in Appendix A.1, we observe that gradient cosine similarities across different diffusion timesteps exhibit similar overall trends—starting relatively high, gradually decreasing, and eventually becoming negative. However, the rate of this evolution varies significantly across timesteps:

  • For t ∈ (0, 0.1] (fine detail generation), we observe negative cosine within 500K iterations.
  • For larger timesteps (coarser features), similar behavior appears much later.
  • Figures 3 and 4 are schematic illustrations intended to demonstrate this general trend, with line 98 explicitly noting that Figure 3 uses t = 0.1 as an example.

• Balancing alignment benefits and conflicts

  Our method fundamentally seeks to balance two competing effects:

  • Benefits: Alignment provides valuable guidance for low-frequency, structural features
  • Costs: Alignment eventually hinders the learning of high-frequency details

  The key is to terminate before the harm to fine-grained details outweighs the benefits to coarse-grained structure. Notably, our actual termination point (250K iterations for HASTE) is chosen well before gradient similarities become fully negative across all timesteps. This reflects a principled trade-off rather than waiting for complete gradient opposition. Additionally, our experimental results with HASTE strongly validate this termination strategy.

Q2: Using register tokens to remove attention sinks in CLIP.

Thanks for your advice. First, we would like to clarify that our original submission doesn't include experiments of ATTA with CLIP attention. Nevertheless, we can provide the experimental results of HASTE with CLIP-ViT-B/16.

• Setting: We use SiT-B/2 for fast validation following the original settings of HASTE. Since SiT-B/2 and CLIP-ViT-B/16 possess different patch numbers (i.e., 16x16 and 14x14), we perform additional spatial interpolations to align the dimensions.

• Results (w/o cfg):

  • Comparing HASTE with REPA at 100K iteration:

    method	iteration	encoder	FID↓	sFID↓	IS↑
    SiT	100K	-	63.46	7.31	20.6
    SiT+REPA	100K	DINOv2-B	49.50	7.00	27.5
    SiT+REPA	100K	CLIP-ViT-B/16	54.92	7.63	24.7
    SiT+HASTE	100K	DINOv2-B	39.86	7.16	35.8
    SiT+HASTE	100K	CLIP-ViT-B/16	48.74	7.92	28.7

  • Validating the termination strategy of HASTE:

    method	iteration	encoder	termination	FID↓	sFID↓	IS↑
    SiT	150K	-	-	52.71	7.06	26.2
    SiT+HASTE	150K	CLIP-ViT-B/16	-	38.36	7.23	38.2
    SiT+HASTE	150K	CLIP-ViT-B/16	100K	36.13	6.95	40.5

• Register tokens: We sincerely appreciate the insightful paper you referred to. We think it's a promising direction to leverage the register tokens without additional training to improve the attention quality of pre-trained encoders. We will discuss the possibility introduced by the proposed method in Limitations and future work, Section 5.

Thanks reviewer KpE1 for constructive comments. We are delighted to discuss the questions below and sincerely hope these answers can address the concerns.

Q1: Implementation of gradient-angle threshold.

A1: Thanks for your question. Here’s a clarification and an updated revision for transparency.

• Dual termination strategy: We mainly use fixed iteration $\tau$ due to its simplicity and effectiveness (e.g., ImageNet/MS-COCO). The gradient-angle threshold adds theoretical benefits. Gradient analysis (Lines 283–287) shows alignment gains drop around 250K iterations.

• Gradient-angle trigger (plain implementation): We train SiT-B/2 on ImageNet 256 $\times$ 256 following the original settings of HASTE using a gradient-angle trigger.

  • Process:

    • After 50K iterations, every 10K, we check the gradient cosine similarity between alignment and denoising objectives.

    • At each check, we use selected timesteps $t$ at various noise levels on 2048 random ImageNet images; average the resulting similarities. If average $<$ threshold $\delta$, we stop alignment.

    • Example uses 5th SiT block, timesteps $t=0.02$, $0.1$, $0.5$, $0.8$, and $\delta=-0.05$.

  • Results:

    • Cosine similarity:

    timestep\iteration	50K	60K	70K	80K
    0.02	-0.0174	0.0418	0.0872	-0.0032
    0.1	0.0126	-0.0767	-0.0148	-0.1155
    0.5	0.0134	0.0106	-0.0848	-0.1272
    0.8	0.0034	0.0406	0.1025	-0.0212
    avg	0.0030	0.0041	0.0225	-0.0668(terminate)

    • Performance:

    method	iter	terminate	FID$\downarrow$	sFID$\downarrow$	IS$\uparrow$
    SiT	100K	-	63.46	7.31	20.6
    SiT+HASTE	100K	-	39.86	7.16	35.8
    SiT+HASTE	100K	80K	38.69	6.88	36.4

  • Key point: The automatic trigger ends holistic alignment at 80K and improves results at 100K.

  • Limitations:

    • Overhead: Gradient checks add computation, limiting check frequency.
    • Hyperparameters: Trigger is sensitive to hyperparameters (block, timestep, threshold).

  • Conclusion: Since monitoring model performance during training is a standard practice in modern deep learning workflows, it's more practical to integrate the periodic evaluation of both metrics and gradient cosine similarity to determine the optimal termination point.

Q2&Q5: Exploration of alignment layer depth and theoretical framework for mid-layer attention distillation.

A2&A5: Thanks for your question. Below are concise explorations of alignment depth and our choice of mid-layer for distillation.

• Alignment depth:

  • Setup: Train SiT-B/2 on ImageNet 256$\times$256 with HASTE for 100K iters. Vary alignment depth; no termination.

  • Results:

    method	DINO	SiT	Depth	FID$\downarrow$	sFID$\downarrow$	IS$\uparrow$
    SiT	[7,9,11]	[2,3,4]	5	39.86	7.16	35.8
    SiT+HASTE	[7,9,11]	[4,5,6]	7	40.59	7.27	35.6
    SiT+HASTE	[7,9,11]	[6,7,8]	9	43.37	7.75	33.5

  • Conclusion: Depth 5 yields best performance.

• Mid-layer distill choice:

  • Alignment depth set per REPA (e.g., 8 for SiT-L/XL), to observe ATTA for HASTE.

  • As in Lines 171-183, mid-layer choice is due to:

    • Shallow mismatch: Early layers should handle noisy input. (SiT-L/2: layer 4-7 > layer 2-7 in Table 6)
    • Deep freedom: Deeper layers focus on denoising. Aligning too deep hurts results (REPA: depth >8 reduces performance). So, use layer 4-7 for alignment.

• Limits & future work: While empirical layer choice works, theory may explain more. We plan to study attention in diffusion transformers further.

Q3: Validation of termination strategy in HASTE on MS-COCO.

A3: To address your concern, we ran a T2I test on MS-COCO.

• Setup: Using HASTE/REPA settings, we trained MMDiT with both from 150K-250K iterations, comparing MMDiT+HASTE with/without termination at $\tau$=200K. At 250K, cfg set to 3.5 for eval; Sampler: SDE Euler-Maruyama, NFEs=250.

• Results:

  method	iter	term	FID$\downarrow$
  MMDiT+REPA	250K	-	4.28
  MMDiT+HASTE	250K	-	4.10
  MMDiT+HASTE	250K	200K	4.06

• Conclusion: Holistic alignment beats REPA for MMDiT; termination strategy further improves T2I.

Q4: Quantification of additional training costs introduced by holistic alignment.

A4:

Thank you for the question. We evaluate the extra overhead of HASTE on SiT-XL/2 introduced by external vision model DINOv2-B:

• Extra memory overhead: The peak CUDA memory with a local batchsize 32:

  HASTE	REPA
  30666.84 MB	27818.01 MB

• DINOv2-B forward overhead: During training, we compute DINOv2‑B representations online following REPA. The forward pass of DINOv2‑B incurs approximately 23.19 GFLOPs for the base‑size model at 224×224 input resolution without XFormers.

Q6: Comparison with TREAD and FasterDiT.

A6: Thanks for your comment. Here’s a detailed comparison with TREAD[1] and FasterDiT[2].

• ImageNet 256$\times$256 Since TREAD does not report SiT+TREAD+cfg, we use their best DiT+TREAD result.

  • Results:

    • w/o cfg

      method	epoch	FID$\downarrow$
      FasterDiT	400	7.91
      SiT+HASTE	80	7.31
      SiT+TREAD	80	4.89

    • w/ cfg

      method	epoch	FID$\downarrow$
      FasterDiT	400	2.03
      DiT+TREAD	740	1.69
      SiT+HASTE	400	1.44

  • Analysis: TREAD gets lower FID w/o cfg, but HASTE is better w/ cfg. TREAD notes, “Restricting info in training can hurt CFG performance.” Our HASTE gains a lot with cfg.

    • HASTE at 600 epochs even outperforms REPA’s best FID:

      method	epoch	FID$\downarrow$	sFID$\downarrow$	IS$\uparrow$
      SiT+REPA	800	1.42	4.70	305.7
      SiT+HASTE	600	1.41	4.51	296.2

  • Conclusion: As CFG is key for most gen tasks, HASTE is more practical for acceleration.

• MS-COCO 256$\times$256 We did not compare with FasterDiT or TREAD in T2I because:

  • FasterDiT lacks T2I results/public code.

  • Our model/setup differs from TREAD:

    • TREAD uses smaller DiT-B, HASTE uses larger MMDiT.

    • TREAD uses FID-10K, HASTE uses FID-50K.

Q7: Exploration of different pre-trained encoders.

A7: Thanks for your advice. To address this, we further explore using CLIP(ViT-B/16) as the pre-trained encoder:

• Setting: We use SiT-B/2 for fast val as in HASTE's original setup. Since SiT-B/2 and CLIP-ViT-B/16 have different patch numbers (16x16 vs. 14x14), we interpolate spatial dims for alignment:

  • Feature map interp: SiT hidden states [bs,16x16,768] are reshaped to [bs,16,16,768], then downsampled with bilinear interp from [16,16] to [14,14].

  • Attn map interp: SiT attention maps [bs,H,16x16,16x16] are reshaped to [bs,H,16,16,16,16] and interpolated on key/query spaces.

• Results (w/o cfg):

  • HASTE vs REPA at 100K:

    method	iter	encoder	FID↓	sFID↓	IS↑
    SiT	100K	-	63.46	7.31	20.6
    SiT+REPA	100K	DINOv2-B	49.50	7.00	27.5
    SiT+REPA	100K	CLIP-ViT-B/16	54.92	7.63	24.7
    SiT+HASTE	100K	DINOv2-B	39.86	7.16	35.8
    SiT+HASTE	100K	CLIP-ViT-B/16	48.74	7.92	28.7

  • HASTE termination at 150K:

    method	iter	encoder	term	FID↓	sFID↓	IS↑
    SiT	150K	-	-	52.71	7.06	26.2
    SiT+HASTE	150K	CLIP-ViT-B/16	-	38.36	7.23	38.2
    SiT+HASTE	150K	CLIP-ViT-B/16	100K	36.13	6.95	40.5

• Analysis: At 100K, both REPA and HASTE with CLIP-ViT-B/16 speed up SiT-B/2 training. DINOv2-B gives better gains, matching [REPA] that semantically meaningful pre-trained encoders enhance gen performance. At 150K, earlier termination with HASTE further lowers FID.

• Conclusion: HASTE with CLIP-ViT-B/16 shows its generalizability across different encoders. We handle patch mismatch by interpolation, not just via resizing.

In sum, Swin Transformer[3]'s window attn/patch merging means its features differ from those in diffusion transformers, so we use ViT-based encoders (e.g., DINOv2) for easier analysis and implementation; experiments focus on these.

Q8: While FID improvements are highlighted, the paper does not address scenarios where HASTE might underperform, such as when generating highly complex or rare visual concepts.

A8: Thanks for your comment. First, we clarify that our work mainly focuses on:

• relation between vision representative models & diffusion models,

• and acceleration Methods for generative diffusion training,

not on solutions for spec. hard cases. So, we test mainly on widely-used gen. datasets (e.g., ImageNet), and note HASTE may underperform in some conditions.

We also evaluate HASTE in more complex cases:

• Text-to-image generation: In our paper, we test HASTE on MS-COCO 256x256, where MMDiT captures complex semantics with ~80K train imgs.

• High-resolution image generation: We also test HASTE on ImageNet 512x512. We set $\tau=300K$, cfg scale 1.25.

  method	iters	term.	FID$\downarrow$	sFID$\downarrow$	IS$\uparrow$
  SiT+REPA	400K	-	2.55	4.16	241.2
  SiT+REPA	500K	-	2.36	4.16	254.2
  SiT+HASTE	400K	300K	2.49	4.20	231.4
  SiT+HASTE	500K	300K	2.34	4.23	253.4

Thanks for your insightful question, we will explore rare visual concept generation in future work.

[1] TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training. In arXiv.

[2] FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification. In NeurIPS 2024.

[3] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In ICCV 2021.