Official Response to Reviewer 2PrL (Part 1/1)

We are glad the reviewer finds our method to be versatile and generally applicable. This is exactly what we are trying to convey in this paper.
Q1: More comprehensive evaluation metrics like sFID, precision, Recall

A1: We have conducted additional evaluation, and now report their full evaluation metrics. These results are consistent with our main findings and further support the effectiveness of GFT.

For fine-tuning experiments, GFT consistently achieves better or comparable performance across all metrics, as shown below:

DiT-XL/2

LlamaGen-3B

For pretraining experiments, GFT also shows consistent improvement, especially in FID, sFID, and IS:

DiT-B/2

LlamaGen-L

More detailed numbers & visualizations at: https://anonymous.4open.science/r/Additional-Results-4CDD/README.md

---

Below, we borrow some space to further respond to Reviewer 5kGx due to space limit. Thank you for understanding!

---

Official Response to Reviewer 5kGx (Part 2/2)

Additional Input of beta

Q7: GFT saves much compute ..... however it comes at a cost. The guidance hyper-parameter becomes part of the model, which complicates the training of the model.

A7: We want to offer the reviewer a new perspective for incorporating $\beta$ as the model's input.

1. $\beta$ does not complicate training, because it is merely a sampling hyperparameter. In training, $\beta$ is always sampled from the uniform distribution [0,1] and does not need to be "tuned" (Algorithm 1 in paper). Thus $\beta$ is very like diffusion time-step $t$, which also serves as a scalar input of the model, but no one should think input $t$ is a "cost" or "training complication".

2. $\beta$ needs to be adjusted for GFT during sampling. However, guidance scale $s$ also needs to be tuned for CFG. So, we can conclude GFT is no more complicated than CFG during inference. Plus, GFT is 2x more efficient.

Q8: The hyper-parameter beta now becomes part of the model, ..., However, this makes the optimization hard as one needs to sample different values of beta.

A8: We are glad the reviewer mentions this concern, because our experiments exactly prove "sampling different values of beta" does NOT complicate optimization.

The most clear evidence is Figure 6 (convergence plot) in the paper. In large-scale from-scratch training, the losses for GFT converge as fast as the classic CFG training w/o a $\beta$ input.

Also in Table 4, performance numbers across 3 distinctive models clearly show that GFT w/ $\beta$ as model input slightly outperforms CFG baselines.

Q9: I checked the proof yet was not fully convinced. ... beta is part of the target model and can be adjusted .... Why is beta set to be 1 in the proof?

A9: The reviewer is referring to the proof for the unconditional model convergence point in Line 730-736 in Appendix B, where we select $\beta = 1$ to derive the conclusion.

As a matter of fact, for other $\beta < 1$, the convergence point is the same. We have already proved this point in the following lines 737-748. At line 749, we summarized: "... this does not change the convergence point of loss. The optimal unconditional solution remains the same".

We thank the reviewer for the comment and have updated our proof for better clarity [1].

Formatting and clarity

Q10: Some notations are used without any proper definitions, such as $p^s$.

A10:

We had explicitly defined $p^s$ in the background section $2.2$, Eq. 5, and referred to this definition in the Method section $3.1$, Line 141.

In short, $p^s$ corresponds to our target sampling distribution. $p^s_\theta$ is our sampling model.

$p^\text{s}(x|c) \propto p(x|c) \left[\frac{p(x|c)}{p(x)}\right]^s.$

Q11: The loss forms in Table 1 are not reasonable.... Suggest revising them and providing more clarification.

A11: We thank the reviewer for the suggestion. Previously, we abbreviated some terms in Table 1 due to space constraints. We have now updated Table 1 to focus more on equation clarity and linked it to the formal equation in the paper [1].

Official Response to Reviewer DEzb

Q1: GFT introduces an additional hyperparameter $\beta$ that requires tuning in training and still in inference.

A1: We believe there is some misunderstanding over how we tune and inject $\beta$.

1. $\beta$ is not a training hyperparameter and it does not require tuning during training.

(Algorithm 1): GFT is essentially learning various sampling models under different $\beta$ at the same time.

$\beta$ is always randomly sampled from the uniform distribution [0,1] for each data point.

$\beta$ is very like the diffusion time-step $t$, it is only a scalar model input instead of a hyperparameter to be tuned.

2. $\beta$ indeed needs to be adjusted in sampling. However, guidance $s$ also needs to be tuned for CFG. So, GFT requires no more tuning than CFG.

Q2: Providing more intuition for how to select values for both training and inference.

A2: Following A1, we do not "select" $\beta$ in inference.

In sampling, according to Theorem 1 and Line 192 in paper, there exists a one-to-one correspondence between CFG $s$ and GFT $\beta$.

$\beta = \frac{1}{1+s}$.

Suppose we already know optimal CFG $s$ is 0.4, then the optimal GFT $\beta$ should be $\frac{1}{1.4}$. In practice, this value is usually not accurate due to training bias, but it provides a good starting point.

CFG:

GFT:

Q3: $\beta$ may introduce various issues in how to inject this condition. However there lacks ablation study on this.

A3: We are happy the reviewer is interested in this.

Actually, how to inject a scalar condition into a diffusion model is well-explored, because diffusion-time $t$ is similar stuff. We simply borrowed their design and found it works well enough.

We indeed tried several ablation choices initially.

1. Instead of posing $\beta$ as model input, we leverage $s = \frac{1}{\beta} -1$ in Theorem 1, and model

$\epsilon\_\theta(x\_t|c, \beta) := \epsilon\_\theta^1(x\_t|c) + (\frac{1}{\beta} - 1) \epsilon\_\theta^2(x\_t|c),$

where $\epsilon\_\theta^1$ is the pretrained network, and $\epsilon\_\theta^2$ the pretrained network with a new MLP head. Our hope is to avoid making $\beta$ as network input. However, this design performs poorly:

We suspect the main reason is that '$\beta$ as model input' allows better leveraging the full potential of pretrained model parameters.

2. We ablated how the $\beta$ input MLP encoder size on DiT-XL. (Turns out to be insensitive)

Q4: Limited exploration of how the approach extends to text-to-image generative models.

A4: We feel the reviewer might miss Table 3, Figure 4, Figure 8, and Figure 12 in our paper, where we have already conducted experiments on T2I generative models using Stable Diffusion as base model, and LAION 5+ as the dataset.

In short, GFT significantly increases guidance-free FID from 22.55 to 8.10, CLIP score from 0.252 to 0.313, achieving same level of CFG performance.

Q5: When GFT might not be advantageous compared to CFG?

A5:

1. If training from scratch, GFT still requires 20% more computation than CFG, but is 2x more efficient in inference.

2. When it comes to advanced guidance methods, such as dynamically adjusting the guidance scale during decoding [1]. CFG only requires modifying the sampling code. However, GFT requires redesigning the training code (though only 1-3 lines).

[1] Applying guidance in a limited interval improves sample quality in diffusion models.

Q6: How does CFG/GFT training computation trade-off change with model scale?

A6: In short, the influence of model size is almost negligible.

VAR as an example:

Q7: CFG doesn't work well with semantic latent space. Does GFT present any advantage over CFG with such semantic tokenizers?

A7: Unfortunately, due to the similar theoretical property between GFT and CFG as in Theorem 1, we find it difficult to expect GFT to solve some potential issues in which CFG has failed. Likewise, if GFT does not work well on some domains, we believe CFG may also struggle.

Official Response to Reviewer oBo4 (Part 1/1)

We thank the reviewer for the insightful review. We are really glad to see the reviewer shares the same belief as us that Guided sampling should eventually be removed from visual modeling. We are also greatly motivated by the high praise given to our work.

Q1: Algorithm 1, line 8: We already have the pseudo-temperature $\beta \in [0,1]$, why do we still need to perform dropout on the condition?

A1: The most direct reason for dropping out condition is that we have detached the gradient from unconditional model ($\mathrm{\mathbf{sg}}(\cdot)$) during training (Algorithm 1 line 12). If we do not randomly dropout the condition, the unconditional model would not be trained at all.

$L_\theta =\|\beta \epsilon\_\theta^s(x\_t|c\_\emptyset,\beta ) + (1-\beta) \mathrm{\mathbf{sg}}$ \epsilon_\theta^u (x_t| c=\emptyset , \beta=1) $

• \epsilon|^2. $

The question now becomes why we have to detach the unconditional gradient.

In short, without detaching the unconditional gradient, loss becomes

$L_\theta =\|\beta \epsilon\_\theta^s(x\_t|c,\beta ) + (1-\beta)$ \epsilon_\theta^u (x_t| c=\emptyset , \beta=1) $

• \epsilon|^2. $

Since $\beta \sim \text{U}[0,1]$, the unconditional model can still be trained. However, the loss function now spends 50 % of its energy to optimize the unconditional model, which is way too much because the unconditional part is not what we finally want. This hurts performance in practice. Also, this causes misalignment with CFG training pipeline. We believe GFT should not only ensure soundness but also offer seamless integration and extreme compatibility.

We refer the reviewer to Section 3.2 (lines 194 to 210) in our paper for detailed response.

Q2: When dropping out the condition, wouldn't the two branches $\beta$ and$(1-\beta)$ be redundant?

A2: Yes, they will be redundant. However, when dropping out condition, we are training the unconditional model, the loss is

$L_\theta =\|\beta \epsilon\_\theta^u(x\_t|\beta ) + (1-\beta) \mathrm{\mathbf{sg}}$ \epsilon_\theta^u (x_t| \beta) $

• \epsilon|^2. $

We have proved in Appendix B (line 737-748) that this objective has exactly the same solution as classic unconditional loss: $L_\theta =\| \epsilon\_\theta^u(x\_t|\beta ) - \epsilon\|^2.$

Therefore, $\mathbf{sg}$ does not affect model convergence.

We could have removed the $1-\beta$ branch when dropping out condition, but again, this causes the unconditional model part to be trained too much and also breaks compatibility with CFG design. We considered and tried many versions of this loss equation, but decided the first loss form is the most elegant one.

Q3: I am very curious about what impact removing the stop gradient would have on performance.

A3: We re-ran the DiT-XL/2 experiments following the settings in the paper, with and without stop-gradient. The results show that using stop-gradient slightly improves the performance.

Q4: I am also curious about the effect that the size of the MLP model following the $\beta$ parameter has on the results.

A4: We ablated how the $\beta$ MLP encoder size affects training performance on DiT-XL/2. Overall, we find GFT to be insensitive to the MLP encoder size:

Q5: If it's a T2I task, how should the negative prompt and β be implemented?

A5:

1. Replace the unconditional mask with a negative prompt $c\_n$ randomly sampled from a pool of negative candidates.
2. We assume randomly masking out "condition (prompt)" might not be necessary anymore. Because these negative prompts should already appear in the dataset several times. We are not sure, this one can be tested out.

$L_\theta =\|\beta \epsilon\_\theta^s(x\_t|c,\beta ) + (1-\beta) \mathrm{\mathbf{sg}}$ \epsilon_\theta^s (x_t| c_n , \beta=1) $

• \epsilon|^2. $

Official Response to Reviewer 5kGx (Part 1/2)

We thank the reviewer for the very detailed comments! We summarize concerns into four categories:

Computational/Memory efficiency

Q1: The reformulation results in two different models active and up to optimization during training, ... Are they sharing the same parameters as in CFG? If not, that makes the advantages of GFT less obvious.

A1: They do share parameters just like CFG, Throughout all our experiments, there is ALWAYS only one model kept in memory. This makes GFT extremely memory-efficient.

As noted in Figure 3, GFT has the same GPU memory usage as CFG training. This distinguishes GFT from all previous distillation approaches, which all require more GPU memory than CFG.

LlamaGen exp: 8H100, bz 256, FSDP

Q2: Did not specify the basis for comparing the training time in a convincing way. ... superficially claimed "less than 5% pretraining computation" and "2x faster in sampling" .... should be more rigorous.

A2: We thank the reviewer for the suggestion. We use VAR and DiT to show how "less than 5% pretraining computation" is calculated in detail.

(8*H100 GPUs)

"2x faster in sampling": our method simply halves the model inference, ==> allows doubling the batch size while keeping the same sampling time.

Method motivation and Evaluation

Q3: The reported experimental results are not consistent..... Figure 7 : GFT and baselines perform similarly..... But Tables 2/3 report a clear, large margin between GFT and CFG.

A3: We feel there is clearly some misunderstanding here. Figure 7 and Tables 2/3 show consistent results, they just focus on different evaluation metrics. Take DiT-XL for instance.

CFG:

GFT:

Two ways to interpret this data:

1. If restricted to guidance-free sampling, ==> can only use $s=1$ for CFG. ==> GFT's FID 1.99 significantly outperforms CFG FID 9.34. ====> Table 2/3.

2. If only focus on the FID-IS trade-off. ===> we can tune CFG $s$. ==> GFT (FID 1.99) slightly outperforms CFG (FID 2.11) , similar FID-IS trade-off. ====> Figure 7

Q4: Figure 2 is not very informative since most related works can demonstrate such evolution under a stronger level of guidance.

A4: Following A3, Figure 2 does NOT mean to prove GFT "outperforms" existing methods like CFG. Instead, it demonstrates GFT can achieve previous work performance without guided sampling. Thus, the reviewer feeling "GFT is similar to related works with a strong level of guidance" is exactly what we want to see.

We thank the reviewer's question and have updated Figure 2's caption to avoid possible misleading [1].

[1] https://anonymous.4open.science/r/Additional-Results-4CDD/updated.pdf

Q5: The claim is somewhat counting intuition. Eq. 6 is essentially the same as Eq. 4... However, the authors claim GFT can bring notable performance gain as shown in Tables 2/3.

A5: We respectfully disagree with the reviewer. We hope A3 and A4 address the reviewer's concern. **

1. Theoretically, GFT and CFG are equivalent. (Eq. 4 -> Eq. 6)
2. guidance-free performance: GFT is significantly better. (Table 2/3)
3. FID-IS trade-off: GFT and CFG perform similarly. GFT is more efficient in sampling. (Figure 7)

Q6: In comparing GFT and other baselines, it is critical to ensure they are on the same level of guidance strength. However, this is missing ...

A6: We respectfully disagree. For GFT and all baselines, we report their best performance by tuning their respective guidance $s$ or temperature $\beta$. This ensures fairness.

If have to align guidance strength, we can easily select a hyperparameter suitable for GFT but not optimal for CFG. For example, for DiT-XL, GFT achieves optimal FID 1.99 with $\beta = 1/1.5 = 0.667$. Under the same level of guidance, CFG FID is 2.14 at $s=1.5$. However, we choose to compare with the CFG optimal FID 2.11 at $s=1.4$.

---

Reminder for Part (2/2)

Due to very detailed questions posted, we borrowed some space from Reviewer 2PrL for Part (2/2) response to answer Q7-Q11. Thank you for understanding!